Image pickup device and control method thereof

Abstract

The present invention discloses an image pickup device and a control method thereof. The image pickup device comprises: an image pickup unit configured to output image data; a control unit configured to control the orientation of the image pickup unit so that the image pickup unit points to a subject; and a state acquisition unit configured to acquire information about the state in which the image pickup device is held; wherein, based on the information about the state in which the image pickup device is held acquired by the state acquisition unit, the control unit changes the change range of the orientation of the image pickup unit.

Description

[0001] This application is a divisional application of an invention patent application with an application date of September 20, 2018, application number 201880072663.0 (international application number PCT / JP2018 / 034818), and invention name “Image pickup device and control method thereof”. Technical Field

[0002] The invention relates to an image pickup device and a control method thereof. Background Art

[0003] In still image or video shooting using an image pickup device such as a camera, a user generally decides a subject to be shot through a viewfinder or the like and adjusts the framing of the image to be shot by checking the shooting conditions by oneself to shoot an image. Such an image pickup device has a function of detecting an error in a user operation and notifying the user of the error, or detecting an external environment and notifying the user that the environment is not suitable for shooting an image when the external environment is not suitable for shooting an image. In addition, there is a mechanism for controlling the camera so that the camera is in a state suitable for shooting an image.

[0004] Such an image pickup device that captures images by user operation includes a life log camera that periodically and continuously captures images without a user's shooting instruction (PCT Japanese Translation Patent Publication No. 2016-536868). The life log camera is used in a state of being attached to the user's body with a strap or the like, and records scenes that the user sees in daily life as video images at set time intervals. The life log camera captures images at set time intervals rather than at expected timing (such as the timing when the user releases the shutter), so that unexpected moments that are not usually captured can be recorded as video images. Summary of the invention

[0005] However, when the life log camera automatically captures images periodically while the user is wearing the life log camera, video images that are not preferred by the user may be acquired, and video images of the moments that the user really wants may not be acquired.

[0006] The present invention has been made in view of the above-mentioned inconvenience, and an object of the present invention is to provide an image pickup apparatus capable of taking a video image preferred by a user without any specific operation of the user.

[0007] The technical feature of the present invention is a control method for an image pickup device, and includes a changing step of changing the processing of the image pickup device based on first data about a captured image captured by a capturing unit. In the changing step, when changing the processing of the image pickup device, a greater weight is assigned to the first data about the captured image based on an instruction from a user than to the first data about the captured image processed automatically. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A is a diagram schematically showing an image pickup device.

[0009] Figure 1B is a view illustrating the pitch direction, yaw direction, and roll direction.

[0010] Figure 2 is a diagram illustrating the configuration of an image pickup device.

[0011] Figure 3 is a diagram illustrating the configuration of an image pickup device and external devices.

[0012] Figure 4 is a diagram showing the configuration of external devices.

[0013] Figure 5 is a diagram showing the configuration of an image pickup device and external devices.

[0014] Figure 6 is a diagram showing the configuration of external devices.

[0015] Figure 7 is a flow chart illustrating a first control unit.

[0016] Figure 8 is a flowchart illustrating a second control unit.

[0017] Fig. 9 is a flowchart illustrating the processing of the shooting mode.

[0018] Fig.10 is a flow chart illustrating automatic edit mode determination.

[0019] Fig.11 is a flow chart illustrating the automatic editing process.

[0020] Fig.12 is a diagram illustrating a neural network.

[0021] Fig.13A : is a view illustrating an example in which area division is performed all around the position of the image pickup device (when both the tilt direction and the pan direction are 22.5 degrees).

[0022] Fig. 13B : is a view illustrating an example in which area division is performed in the entire periphery of the position of the image pickup device (when the inclination angle is greater than or equal to 45 degrees).

[0023] Fig. 13C is a view for illustrating an example in which region division is performed in a photographing angle of view.

[0024] Fig.13D is a view for illustrating an image example of a viewing angle area of ​​an image being imaged.

[0025] Fig.14 is a flow chart illustrating learning mode determination.

[0026] Fig.15 is a flowchart illustrating the learning process.

[0027] Fig.16 is a diagram illustrating display processing according to the present embodiment.

[0028] Fig.17 2 is a diagram showing the arrangement of a lens barrel rotation drive unit.

[0029] Fig.18 It is a view illustrating the target position and actual position of the subject on the image.

[0030] Fig.19 is a flowchart illustrating an operation of detecting a change in shooting direction.

[0031] Fig. 20 is a diagram schematically showing detection of a photographing direction changing operation.

[0032] Fig.21A 1 is a diagram showing an image captured at a timing of time ta.

[0033] Fig. 21B 1 is a view showing an image captured when the user rotates the lens barrel 102 to the right with respect to the fixing portion 103 at the timing of time tb.

[0034] Fig. 21C 17 is a view showing an image when the target subject is placed in the angle of view by rotating the pan axis to the vicinity of a new subject through a shooting direction changing operation by the user in a state where the control output of the compensator 1702 is stopped at time tc.

[0035] Fig.21D 17 is a view showing an image obtained by tracking and imaging a new subject after changing the photographing direction in a state where the control output of the compensator 1702 starts at time t4.

[0036] Fig. 22 2202 is a graph showing time changes in a control output 2201 of panning, a positional deviation 2202, and a moving speed 2203 for a photographing direction changing operation.

[0037] Fig.23 : is a graph showing a change in the output of the acceleration sensor of the device shake detection section 209 when the photographing direction of the image pickup device is changed by the user's operation.

[0038] Fig.24 is a flowchart illustrating a manual voice control photography determination process.

[0039] Fig.25 is a diagram showing a result obtained by applying the sound direction detection value to the histogram processing.

[0040] Fig.26 3 is a diagram illustrating notification to the smart device 301 that a subject is registered.

[0041] Fig. 27 is a flowchart illustrating an automatic shooting process in the case of wake-up based on tap detection.

[0042] Fig.28 is a flow chart illustrating automatic shooting processing in the case of wake-up based on sound detection.

[0043] Fig.29 is a flow chart illustrating the processing of the automatic file deletion mode.

[0044] Fig.30 is a view showing an example of a handheld control accessory.

[0045] Fig.31 50 is a view showing a scene in which the user photographs a selected subject 5011 by pointing the image pickup device to the subject 5011 .

[0046] Fig.32 is a diagram showing an example of the configuration of an accessory shoe 3202 which is mechanically attachable to a camera 3201 different from the image pickup device 101 .

[0047] Fig.33 32 is a flowchart illustrating a process of the image pickup apparatus 101 when the image pickup apparatus 101 and another camera 3201 are connected. DETAILED DESCRIPTION

[0048] [First embodiment]

[0049] <Configuration of Image Pickup Device>

[0050] FIG. 1 is a diagram schematically showing an image pickup apparatus of a first embodiment.

[0051] Figure 1AThe image pickup device 101 shown in the figure includes an operating member (hereinafter referred to as a power button; however, it may also be an operation such as tapping, flicking, and sliding on a touch panel) by which a power switch can be operated, and other components. A lens barrel 102 as a housing containing a shooting lens group and an imaging element for capturing an image is installed in the image pickup device 101. The lens barrel 102 has a rotating mechanism that can drive and rotate the lens barrel 102 relative to the fixing portion 103. The tilting unit 104 is a unit that can make the lens barrel 102 rotate in a certain direction. Figure 1B The pan unit 105 is a motor drive mechanism that can rotate the lens barrel 102 in the pitch direction shown in FIG. The pan unit 105 is a motor drive mechanism that can rotate the lens barrel 102 in the yaw direction. Therefore, the lens barrel 102 can rotate around one or more axes. Figure 1B It is the definition of the axis in the position of the fixed part 103. Both the angular velocity meter 106 and the accelerometer 107 are installed on the fixed part 103 of the image pickup device 101. The vibration of the image pickup device 101 is detected based on the angular velocity meter 106 and the accelerometer 107. The tilt unit and the pan unit are rotationally driven based on the detected vibration angle. Therefore, the movement of the lens barrel 102 as the movable part is corrected, or the tilt of the lens barrel 102 is corrected.

[0052] Figure 2 is a block diagram showing the configuration of the image pickup apparatus of the present embodiment.

[0053] exist Figure 2 In the embodiment, the first control unit 223 is composed of a processor (e.g., CPU, GPU, microprocessor, MPU, etc.) and a memory (e.g., DRAM, SRAM, etc.). These components control the blocks of the image pickup device 101 by performing various processes or controlling data transfer between blocks. The non-volatile memory (EEPROM) 216 is an electrically erasable and recordable memory. Constants, programs, etc. for the operation of the first control unit 223 are stored in the non-volatile memory 216.

[0054] exist Figure 2 In the embodiment of the present invention, the zoom unit 201 includes a zoom lens for zooming. The zoom control section 202 drives and controls the zoom unit 201. The focus unit 203 includes a lens for focus adjustment. The focus control section 204 drives and controls the focus unit 203.

[0055] In the imaging section 206, the imaging element receives the light entering through the lens group, and outputs the information of the charge corresponding to the light amount as analog image data to the image processing section 207. The image processing section 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation to the digital image data output by analog-to-digital conversion, and outputs the processed digital image data. The digital image data output from the image processing section 207 is converted into a recording format such as JPEG format by the image recording section 208, and is sent to the memory 215 or the video output section 217 (described later).

[0056] The lens barrel rotation driving section 205 drives the tilt unit 104 and the pan unit 105 to cause the lens barrel 102 to be driven in the tilt direction and the pan direction.

[0057] In the device shake detection unit 209, for example, an angular velocity meter (gyro sensor) 106 for detecting the three-axis angular velocity of the image pickup device 101 and an accelerometer (acceleration sensor) 107 for detecting the three-axis acceleration of the device are installed. The device shake detection unit 209 calculates the rotation angle of the device, the displacement amount of the device, etc. based on the detected signals.

[0058] The voice input unit 213 obtains a voice signal around the image pickup device 101 from a microphone provided in the image pickup device 101, performs analog-to-digital conversion on the voice signal, and sends the voice signal to the voice processing unit 214. The voice processing unit 214 performs voice-related processing, such as optimizing the processing of the input digital voice signal. The voice signal processed by the voice processing unit 214 is sent to the memory 215 by the first control unit 223. The memory 215 temporarily stores the image signal obtained by the image processing unit 207 and the voice signal obtained by the voice processing unit 214.

[0059] For example, the image processing unit 207 reads out the image signal temporarily stored in the memory 215, encodes the image signal, and generates a compressed image signal. For example, the voice processing unit 214 reads out the voice signal temporarily stored in the memory 215, encodes the voice signal, and generates a compressed voice signal. The first control unit 223 sends these compressed image signals and compressed voice signals to the recording and playback unit 220.

[0060] The recording and playback section 220 records the compressed image signal generated by the image processing section 207, the compressed voice signal generated by the voice processing section 214, and other control data related to shooting, etc., on the recording medium 221. In the case where the voice signal is not compressed and encoded, the first control section 223 sends the voice signal generated by the voice processing section 214 and the compressed image signal generated by the image processing section 207 to the recording and playback section 220, and causes the recording and playback section 220 to record the voice signal and the compressed image signal on the recording medium 221.

[0061] The recording medium 221 may be a recording medium built into the image pickup device 101 or a detachable recording medium. The recording medium 221 can record various data such as a compressed image signal, a compressed voice signal, and a voice signal generated in the image pickup device 101. A medium having a capacity larger than that of the nonvolatile memory 216 is generally used as the recording medium 221. For example, the recording medium 221 may be any type of recording medium such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

[0062] The recording and playback section 220 reads out (plays back) the compressed image signal, compressed voice signal, voice signal, various data and programs recorded on the recording medium 221. The first control section 223 sends the read compressed image signal to the image processing section 207, and sends the compressed voice signal to the voice processing section 214. The image processing section 207 temporarily stores the compressed image signal in the memory 215, decodes the compressed image signal in a predetermined step, and sends the decoded signal to the video output section 217. The voice processing section 214 temporarily stores the compressed voice signal in the memory 215, decodes the compressed voice signal in a predetermined step, and sends the decoded signal to the voice output section 218.

[0063] A plurality of microphones are installed on the image pickup device 101 as a voice input unit 213. The voice processing unit 214 is capable of detecting the direction of the sound in the plane in which the plurality of microphones are installed, and is used for searching or automatic shooting (described later). In addition, the voice processing unit 214 detects a specific voice command. The voice command may be some pre-registered commands, and may also be configured so that the user can register a specific voice in the image pickup device. The voice processing unit 214 also performs sound scene recognition. In the sound scene recognition, the sound scene determination is performed by a network trained by machine learning based on a large amount of voice data in advance. For example, a network for detecting a specific scene, such as "cheer up", "clap" and "speak", is set in the voice processing unit 214. When a specific sound scene or a specific voice command is detected, the voice processing unit 214 is configured to output a detection trigger signal to the first control unit 223 or the second control unit 211.

[0064] The second control section 211 , which is provided separately from the first control section 223 that controls the entire main system of the image pickup apparatus 101 , controls power supplied to the first control section 223 .

[0065] The first power supply section 210 and the second power supply section 212 supply power for operating the first control section 223 and power for operating the second control section 211, respectively. When the power button provided for the image pickup device 101 is pressed, power is first supplied to both the first control section 223 and the second control section 211; however, as described later, the first control section 223 is controlled so that the first power supply section 210 stops supplying power to itself. When the first control section 223 is also not operating, the second control section 211 operates and receives information from the device shaking detection section 209 or the voice processing section 214. The second control section is configured to perform a determination process as to whether to start the first control section 223 based on various input information. When it is determined to be started, the second control section is configured to provide an instruction to supply power to the first power supply section.

[0066] The voice output section 218 outputs a preset voice pattern from a speaker built in the image pickup device 101 , for example, during shooting or the like.

[0067] The LED control section 224 controls the LED provided for the image pickup device 101 in a preset light blinking pattern, for example, during photographing or the like.

[0068] The video output unit 217 is composed of, for example, a video output terminal, and transmits an image signal for displaying a video image on a connected external display, etc. The voice output unit 218 and the video output unit 217 may be a combined terminal, for example, a terminal such as a High Definition Multimedia Interface (HDMI) (registered trademark) terminal.

[0069] The communication section 222 performs communication between the image pickup device 101 and an external device, and, for example, sends or receives data such as a voice signal, an image signal, a compressed voice signal, and a compressed image signal. The communication section 222 also receives control signals related to shooting, such as shooting start and stop commands and pan / tilt or zoom, and drives the image pickup device 101 based on instructions from an external device that communicates bidirectionally with the image pickup device 101. The communication section 222 also sends or receives information between the image pickup device 101 and the external device, such as various learning-related parameters to be processed in the learning processing section 219 (described later). The communication section 222 is, for example, a wireless communication module, such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a wireless USB, and a GPS receiver.

[0070] <Configuration with external communication device>

[0071] Figure 3 is a diagram showing a configuration example of a wireless communication system between an image pickup apparatus 101 and an external apparatus 301. The image pickup apparatus 101 is a digital camera having a photographing function. The external apparatus 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

[0072] The image pickup apparatus 101 and the smart device 301 can communicate, for example, by means of communication 302 based on a wireless LAN conforming to a series of IEEE 802.11 standards and communication 303 having a master-slave relationship like a control station and an auxiliary station, such as Bluetooth Low Energy (hereinafter referred to as "BLE"). Wireless LAN and BLE are both examples of communication methods. Other communication methods can be used in a case where each communication device has two or more communication functions and one of the communication functions that performs communication in a relationship between a control station and an auxiliary station is capable of controlling another communication function. However, without loss of generality, a first communication such as a wireless LAN provides a higher communication speed than a second communication such as BLE, and the second communication is defined as at least any one of a communication having lower power consumption than the first communication and a communication having a shorter communicable distance than the first communication.

[0073] Reference Figure 4 The configuration of the smart device 301 is described.

[0074] The smart device 301 includes not only a wireless LAN control unit 401 for wireless LAN and a BLE control unit 402 for BLE, for example, but also a public line control unit 406 for public wireless communication. The smart device 301 also includes a packet sending and receiving unit 403. The wireless LAN control unit 401 executes a driver program that performs wireless LAN RF control, communication processing, and various controls for communication with a wireless LAN that conforms to a series of IEEE 802.11 standards, and performs protocol processing for communication with the wireless LAN. The BLE control unit 402 executes a driver program that performs BLE RF control, communication processing, and various controls for communication with BLE, and performs protocol processing for communication with BLE. The public line control unit 406 executes a driver program that performs public wireless communication RF control, communication processing, and various controls for communication with public wireless communication, and performs protocol processing related to public wireless communication. Public wireless communication, for example, conforms to the IMT (International Multimedia Telecommunications) standard, the LTE (Long Term Evolution) standard, and the like. The packet transmission and reception unit 403 performs processing for performing at least any one of transmission and reception of packets related to communication with wireless LAN and BLE and public wireless communication. In this example, description will be made based on the assumption that the smart device 301 performs at least one of transmission and reception of packets in communication; however, in addition to packet switching, other communication formats such as circuit switching, etc. may be used.

[0075] The smart device 301 also includes, for example, a control unit 411, a storage unit 404, a GPS receiving unit 405, a display unit 407, an operation unit 408, a voice input and voice processing unit 409, and a power supply unit 410. For example, the control unit 411 controls the entire smart device 301 by running a control program stored in the storage unit 404. The storage unit 404 stores, for example, a control program run by the control unit 411 and various information used for communication, such as parameters. Various operations (described later) are implemented by the control unit 411 running the control program stored in the storage unit 404.

[0076] The power supply unit 410 supplies power to the smart device 301. The display unit 407 has, for example, a function capable of outputting visually recognizable information (such as an LCD and an LED) or a function of outputting sound (such as a speaker), and displays various information. The operation unit 408 is, for example, a button, etc., which receives a user's operation on the smart device 301. The display unit 407 and the operation unit 408 may be composed of a common member such as a touch panel.

[0077] The voice input and voice processing unit 409 may be configured, for example, to obtain a voice uttered by a user through a general-purpose microphone built into the smart device 301 , and obtain an operation command of the user through voice recognition processing.

[0078] The voice command based on the user's voice is acquired via a dedicated application installed in the smart device. Via the wireless LAN-based communication 302, the voice command can be stored as a specific voice command for causing the voice processing section 214 of the image pickup apparatus 101 to recognize the specific voice command.

[0079] The GPS (Global Positioning System) receiving section 405 receives GPS signals transmitted from satellites, analyzes the GPS signals, and estimates the current position (longitude and latitude information) of the smart device 301. Alternatively, for position estimation, the current position of the smart device 301 may be estimated by using WPS (Wi-Fi Positioning System) or the like based on information about surrounding wireless networks. When the acquired current GPS position information is within a preset position range (within a predetermined radius), the movement information is provided to the image pickup device 101 via the BLE control section 402 and is used as a parameter for automatic shooting or automatic editing (described later). When the position change of the GPS position information is greater than or equal to the predetermined range, the movement information is provided to the image pickup device 101 via the BLE control section 402 and is used as a parameter for automatic shooting or automatic editing (described later).

[0080] As described above, the smart device 301 exchanges data with the image pickup apparatus 101 by means of communication using the wireless LAN control unit 401 and the BLE control unit 402. The image pickup apparatus 101 and the smart device 301 send or receive data, for example, a voice signal, an image signal, a compressed voice signal, a compressed image signal, etc. Operation instructions for shooting, etc. of the image pickup apparatus 101, transmission of voice command registration data, or provision of predetermined position detection and location movement based on GPS position information are performed from the smart device. Learning data is also sent or received via a dedicated application installed in the smart device. The external device 301 is not limited to the smart device 301. The external device 301 may be, for example, a device that does not have a display unit 407, an operation unit 408, etc. and is dedicated to voice input. With such a device, a voice emitted from a user is obtained through the above-mentioned microphone, an operation command of the user is obtained through voice recognition processing, and the operation command is provided to the image pickup apparatus 101. In addition to the above, such a device may have a function of reading out news by using voice recognition, a communication function with the cloud, and a speaker. The device may have a voice output function or a dialogue system function for investigation purposes using a search engine.

[0081] <Attachment Configuration>

[0082] Figure 5 1 is a diagram showing a configuration example of an external device 501 that can communicate with the image pickup device 101. The image pickup device 101 is a digital camera having a photographing function. The external device 501 is a wearable device that includes various sensing parts and can communicate with the image pickup device 101 using, for example, a Bluetooth communication module.

[0083] The wearable device 501 is configured to be wearable on, for example, a user's arm, etc. A sensor that detects biological information such as the user's pulse, heart rate, and blood flow at predetermined intervals, an acceleration sensor that can detect the user's motion status, etc. are installed in the wearable device 501.

[0084] Examples of the biological information detection section 502 include a pulse sensor that detects a pulse, a heart rate sensor that detects a heart rate, a blood flow sensor that detects a blood flow, and a sensor that senses a change in potential due to detection of contact between the skin and a conductive polymer. In the present embodiment, description will be made by using a heart rate sensor as the biological information detection section 502. The heart rate sensor detects the heart rate of the user by irradiating infrared light onto the skin using an LED or the like, detecting the infrared light that passes through body tissue using a light receiving sensor, and applying signal processing to the infrared light. The biological information detection section 502 outputs the detected biological information as a signal to the control section 607 (described later).

[0085] The shake detection section 503 that detects the user's motion status is equipped with, for example, an acceleration sensor or a gyro sensor, and can detect motion based on information about acceleration, such as whether the user is moving, whether the user is performing an action by waving his arms, etc.

[0086] The wearable device 501 is also provided with an operation section 505 that receives a user's operation on the wearable device 501, and a display section 504 (such as an LCD and an LED) that outputs visually recognizable information.

[0087] Reference Figure 6 The configuration of the wearable device 501 is described.

[0088] The wearable device 501 includes, for example, a control section 607 , a communication section 601 , a biological information detection section 502 , a shake detection section 503 , a display section 504 , an operation section 505 , a power supply section 606 , and a storage section 608 .

[0089] For example, the control unit 607 controls the entire wearable device 501 by running a control program stored in the storage unit 608. The storage unit 608 stores, for example, a control program run by the control unit 607 and various information such as parameters for communication. For example, various operations (described later) are implemented by the control unit 607 running the control program stored in the storage unit 608.

[0090] The power supply unit 606 supplies power to the wearable device 501. The display unit 504 has, for example, a function capable of outputting visually recognizable information (such as an LCD and an LED) or a function of outputting sound (such as a speaker), and displays various information. The operation unit 505 is, for example, a button, etc., which receives a user's operation on the wearable device 501. The display unit 504 and the operation unit 505 can be composed of a common member such as a touch panel.

[0091] The operation unit may be configured, for example, to obtain a voice uttered by the user through a general-purpose microphone built into the wearable device 501 and obtain an operation command of the user through a voice recognition process.

[0092] Various detected information from the biological information detection section 502 and the shake detection section 503 are processed by the control section 607 , and the detected information is transmitted to the image pickup device 101 by the communication section 601 .

[0093] For example, when a change in the heart rate of the user is detected, the detected information is sent to the image pickup device 101, or when a movement condition (such as walking, running, and stopping) changes, the detected information is sent. Also, for example, when a preset arm swinging motion is detected, the detected information is sent, or when a preset distance movement is detected, the detected information is sent.

[0094] Fig.30 is an example of a handheld control accessory.

[0095] The image pickup device body 101 does not include an operation member such as a shutter button for a shooting instruction, and the image pickup device 101 may be configured to be operated by an operation member provided for attachment.

[0096] like Fig.31 As shown in , the user may want to photograph the selected subject 5011 by pointing the camera at the subject 5011. At this time, if the camera is automatically panned / tilted, such movement interferes with the user's movement of pointing the camera toward the subject that the user wants to photograph. The handheld control accessory 5001 may include a change switch 5005 that is capable of changing the mode between an automatic setting mode that depends on the camera and a mode in which the user can manually operate the camera. In this case, when the change switch 5005 is set to the manual mode of the camera, the camera is panned / tilted for camera stabilization, but a large pan / tilt angle change for subject search is not performed.

[0097] Alternatively, even when the change switch 5005 is not provided, an accessory detection section 5002 capable of detecting whether the accessory 5001 is connected to the image pickup device 101 may be provided. In this case, when the accessory is connected, the camera is panned / tilted for camera stabilization, but a large pan / tilt angle change for subject search may not be performed. The connection of the accessory may be detected using an existing method such as a change in voltage and ID.

[0098] When the manual mode of the camera is set by changing the switch 5005 or the accessory detection unit 5002, a large pan / tilt angle change for subject search is not performed, but the camera is panned / tilted for camera stabilization. Such tilt correction can be performed, that is, the optical axis of the camera is maintained in a certain direction relative to the gravity direction, but this may interfere with the user's operation. Camera stabilization cuts off lower frequencies (for example, a frequency range less than or equal to 0.1 Hz) and corrects only higher frequency components.

[0099] Even when the camera is not panning / tilting for subject search, the user may wish to manually change the pan / tilt direction. Fig.31 As shown in , when the image pickup device 101 is pointed at the subject 5101 and the subject 5101 is photographed, the user can operate the tilt angle of the image pickup device 101 so that the optical axis is pointed above the image pickup device 101, and then shoot. In this case, an operating member 5003 capable of changing the pan / tilt direction can be provided in the handheld control accessory 5001. Such a method can be adopted, in which the operating member 5003 can be freely translated in the XY coordinates, and the image pickup device 101 is panned / tilted according to the operating direction. For example, when the operating member moves upward, the camera is tilted toward the top of the camera; when the operating member moves downward, the camera is tilted toward the bottom of the camera; when the operating member moves to the right or left, the camera will be panned according to the direction.

[0100] A shutter button 5004 may be provided so that the user can shoot at an arbitrary timing. A switch 5006 may be provided so that a shooting mode (eg, still image shooting mode, video shooting mode, panorama shooting mode, time-lapse shooting mode, etc.) can be changed.

[0101] As a method of providing a control instruction from the handheld control accessory 5001 to the image pickup device 101 , a non-contact communication means may be used.

[0102] The control instruction can be issued through connectors that connect electrical signals and are provided to the image pickup device 101 and the handheld control accessory 5001, respectively; however, when a battery is included in the image pickup device 101, the handheld control accessory does not require a connector for the battery. Therefore, if a connector for operations such as release is provided, it is required to add a waterproof function to the connection portion, or some parts are required, resulting in an increase in device size or an increase in cost.

[0103] The contactless communication means may be Bluetooth Low Energy (BLE), may be Near Field Communication (NFC) BLE, or may be other methods.

[0104] The radio wave generator of the handheld control accessory 5001 can be small, have a small power supply capacity, or can be a component that generates a small amount of power using, for example, a button battery or the force of pressing the shutter button 5004.

[0105] In this way, an accessory that is separated from an image pickup device and has an operating member for providing a release instruction to the image pickup device and an operating member for providing an instruction to drive a rotating mechanism of the image pickup device can be constructed to be attachable to the image pickup device. An accessory having an operating member for providing a shooting mode change instruction that can set two or more of a still image mode, a video mode, a panoramic mode, and a time-lapse shooting mode of a shooting unit can be constructed to be attachable to the image pickup device. An operating instruction using an operating member is provided from the accessory to the image pickup device by contactless communication means. The image pickup device can detect information about an accessory connected to the image pickup device and change the control band of the stabilization means based on the information about the accessory. By detecting information about the accessory, it is possible to change whether to perform tilt correction to maintain an angle in a certain direction relative to the direction of gravity. The low-frequency side of the stabilization control band can be cut off based on the detected information about the accessory.

[0106] Fig.32 It is an example of the configuration of the image pickup device 101 which is mechanically mountable on the accessory shoe 3202 of a camera 3201 different from the image pickup device 101 .

[0107] like Fig.32 As shown in , when the image pickup device 101 is mounted on the accessory shoe, the orientation in which the image pickup device 101 is mounted on the camera 3201 and the angular difference between the optical axis direction of the camera 3201 and the optical axis direction of the image pickup device 101 are known. Therefore, the camera 3201 and the image pickup device 101 are easily controlled in cooperation with each other.

[0108] Information transmission between the image pickup device 101 and the camera 3201 may be realized by a mechanism in which an electrical contact is provided at a portion to be connected to an accessory shoe, and information is transmitted between the image pickup device 101 and the camera 3201. The image pickup device 101 and the camera 3201 may have a mechanism for transmitting information via, for example, a USB, etc. Alternatively, wireless communication (BLE, NFC, etc.) may be used, or other methods may be used.

[0109] The cooperation between the image pickup device 101 and the camera 3201 will be described later.

[0110] <Sequence of Image Pickup Operation>

[0111] Figure 7 : is a flowchart illustrating an example of the operation handled by the first control section 223 of the image pickup apparatus 101 in the present embodiment.

[0112] When the user operates the power button of the image pickup apparatus 101 , the first power supply section 210 is caused to supply power from the power supply section to the first control section 223 and blocks of the image pickup apparatus 101 .

[0113] Similarly, in the second control unit 211, the second power supply unit 212 supplies power from the power supply unit to the second control unit. Figure 8 The details of the operation of the second control unit are described with reference to a flowchart (described later).

[0114] When power is supplied, Figure 7 The process starts. In step 701, the start condition is loaded. In this embodiment, the start condition is as follows.

[0115] (1) Press the power button manually to turn on the power.

[0116] (2) Turning on the power based on an instruction from an external device (eg, 301 ) through external communication (eg, BLE communication).

[0117] (3) The power is turned on from the subprocessor (second control unit 211).

[0118] In the case of (3), the power is turned on from the subprocessor, the start condition calculated in the subprocessor is loaded, and will be referred to later. Figure 8 Describe the details.

[0119] The start condition loaded here is used as a parameter element during object search or automatic shooting, which will be described later. When the loading of the start condition is completed, the process proceeds to step 702.

[0120] In step 702, information is loaded from various sensors. The sensor from which the loading is performed here may be a sensor that detects vibration (such as a gyro sensor and an acceleration sensor of the device shake detection unit 209). The rotational position of the tilt unit 104 and the pan unit 105 may be loaded. The voice level detected by the voice processing unit 214, a trigger for detecting a specific voice recognition, or a detected sound direction may be loaded.

[0121] Although not shown in Figures 1 to Figure 6 As shown in the figure, sensors that detect environmental information also obtain information.

[0122] For example, there are a temperature sensor that detects the temperature of the area around the image pickup device 101 at predetermined time intervals and an atmospheric pressure sensor that detects changes in atmospheric pressure around the image pickup device 101. An illuminance sensor that detects brightness around the image pickup device 101, a humidity sensor that detects humidity around the image pickup device 101, a UV sensor that detects the amount of ultraviolet rays around the image pickup device 101, and the like may be provided. In addition to the detected temperature information, atmospheric pressure information, brightness information, humidity information, and UV information, temperature changes, atmospheric pressure changes, brightness changes, humidity changes, ultraviolet changes, and the like obtained by calculating the rate of change at predetermined time intervals from the detected pieces of information are used for determination of automatic shooting, and the like (described later).

[0123] When loading of information from various sensors is performed in step 702 , the process proceeds to step 703 .

[0124] In step 703, it is detected whether there is a communication instruction from an external device, and when there is a communication instruction, communication with the external device is performed.

[0125] For example, remote operations or data (such as voice signals, image signals, compressed voice signals, and compressed image signals) are sent or received via wireless LAN or BLE from the smart device 301. The following items are loaded from the smart device 301: whether there is an operation instruction for shooting, etc. of the image pickup device 101, sending of voice command registration data, notification of predetermined position detection or location movement notification based on GPS position information, or instructions for sending or receiving learning data.

[0126] In addition, for example, when there is updated information about the user's movement, information about the movement of the arm, or biological information such as heart rate, the information is loaded from the wearable device 501. Various sensors that detect the above-mentioned environmental information may be installed on the image pickup device 101, or may be installed on the smart device 301 or the wearable device 501. In the latter case, the environmental information is also loaded via BLE. When the communication and loading of information from the external device are completed in step 703, the process proceeds to step S704.

[0127] At step 704, the mode to be set is determined. The mode to be set in step 704 is determined and selected from among the following modes.

[0128] (1) Automatic shooting mode

[0129] [Mode determination conditions]

[0130] When it is determined that automatic shooting is to be performed based on detected information (image, sound, time, vibration, location, body change, environmental change) set through learning (described later), the time elapsed since the mode was moved to the automatic shooting mode, previous shooting information, etc., the mode is set to the automatic shooting mode.

[0131] [Processing in mode]

[0132] In the process of the automatic shooting mode (step 710), based on the detected information (image, sound, time, vibration, location, body change, environmental change), a search for a subject is automatically performed by panning / tilting or zooming. When it is determined that it is the timing to perform shooting preferred by the user, a process of determining a shooting method from various shooting methods (such as single shooting of a still image, continuous shooting of a still image, video shooting, panoramic shooting, and time-lapse shooting) is performed, and then shooting is automatically performed.

[0133] (2) Automatic editing mode

[0134] [Mode determination conditions]

[0135] When it is determined that automatic editing is to be performed based on the time that has elapsed since the last automatic editing was performed and the previous captured image information, the mode is set to the automatic editing mode.

[0136] [Processing in mode]

[0137] In the processing of the automatic editing mode (step 712), the processing of selecting a still image or a moving image is performed based on learning, and the automatic editing processing of creating a highlight video of one moving image as a summary is performed based on learning by using image effects, the duration of the edited moving image, etc.

[0138] (3) Image transmission mode

[0139] [Mode determination conditions]

[0140] In the case where the mode is set to the automatic image transfer mode based on an instruction via a dedicated application installed in the smart device, when it is determined that image transfer is to be performed based on the time elapsed since the last image transfer was performed and previous shooting information, the mode is set to the automatic image transfer mode.

[0141] [Processing in mode]

[0142] In the process of the automatic image transfer mode (step 714), the image pickup apparatus 101 automatically extracts images that can match the user's preference, and causes the smart device 301 to automatically extract the user's preferred images and transfer the images. The extraction of the user's preferred images is performed based on the scores assigned to the images and determined based on the user's preference.

[0143] (4) Learning Mode

[0144] [Mode determination conditions]

[0145] The mode is set to the automatic learning mode when it is determined that automatic learning is to be performed based on the time elapsed since the last learning process was performed, information associated with images that can be used for learning, the amount of learning data, etc. Alternatively, when there is also an instruction to set learning data via communication from the smart device 301, the mode is set to this mode.

[0146] [Processing in mode]

[0147] In the process of the automatic learning mode (step 716), the image pickup apparatus 101 performs learning suitable for user preferences. Learning suitable for user preferences is performed by using a neural network based on information about operations performed on the smart device 301, learning information provided from the smart device 301, and the like. The information about operations performed on the smart device 301 includes, for example, image acquisition information from the image pickup apparatus, information on manual editing instructions provided via a dedicated application, and information on determination values ​​input by the user for an image in the image pickup apparatus.

[0148] At the same time, learning related to detection such as personal identification registration, voice registration, sound scene registration, general object recognition registration, learning of the above conditions for low power consumption mode, etc. is performed.

[0149] (5) Automatic file deletion mode

[0150] [Mode determination conditions]

[0151] When it is determined that automatic file deletion is to be performed based on the time that has elapsed since the last time automatic file deletion was performed and the remaining capacity of the nonvolatile memory 216 on which images are recorded, the mode is set to the automatic file deletion mode.

[0152] [Processing in mode]

[0153] In the processing of the automatic file deletion mode (step 718), files to be automatically deleted are specified (three selections) based on tag information, shooting date and time, etc. of each image among images in the nonvolatile memory 216, and the files are deleted.

[0154] Details of the processing of the automatic shooting mode, the processing of the automatic editing mode, and the processing of the learning mode will be described later.

[0155] In step 705, it is determined whether the mode set in step 704 is a low power consumption mode. In the low power consumption mode determination, when the determination conditions of any of the "automatic shooting mode", "automatic editing mode", "automatic image transfer mode", "learning mode" and "automatic file deletion mode" (described later) are not satisfied, it is determined that the mode is set to the low power consumption mode. When the determination processing is performed, the processing proceeds to step 705.

[0156] When it is determined in step 705 that the low power consumption mode condition is satisfied, the process proceeds to step 706 .

[0157] In step 706, various parameters about the start-up factors (parameters for shake detection determination, parameters for sound detection, and elapsed time detection parameters) that are to undergo determination in the sub-processor (second control unit 211) are provided to the sub-processor. The values ​​of the various parameters vary with the results of learning in the learning process (described later). When the processing of step 706 is completed, the processing proceeds to step 707, the power supply of the main processor (first control unit 223) is turned off, and the processing ends.

[0158] On the other hand, when it is determined in step 705 that the mode is not the low power consumption mode, the process proceeds to step 709, and it is determined whether the mode to be set is the automatic shooting mode. When the mode is the automatic shooting mode, the process proceeds to step 710, and the process of the automatic shooting mode is performed. When the process is completed, the process returns to step 702, and the process is repeated. When it is determined in step 709 that the mode is not the automatic shooting mode, the process proceeds to step 711.

[0159] In step 711, it is determined whether the mode to be set is the automatic editing mode. When the mode is the automatic editing mode, the process proceeds to step 712, and the automatic editing mode is executed. When the process is completed, the process returns to step 702, and the process is repeated. When it is determined in step 711 that the mode is not the automatic editing mode, the process proceeds to step 713.

[0160] In step 713, it is determined whether the mode to be set is the automatic image transmission mode. When the mode is the automatic image transmission mode, the process proceeds to step 714, and the process of the automatic image transmission mode is performed. When the process is completed, the process returns to step 702, and the process is repeated. When it is determined in step 713 that the mode is not the automatic image transmission mode, the process proceeds to step 715.

[0161] In step 715, it is determined whether the mode to be set is the learning mode. When the mode is the learning mode, the process proceeds to step 716, and the process of the learning mode is performed. When the process is completed, the process returns to step 702, and the process is repeated. When it is determined in step 715 that the mode is not the learning mode, the process proceeds to step 717.

[0162] In step 717, it is determined whether the mode to be set is the automatic file deletion mode. When the mode is the automatic file deletion mode, the process proceeds to step 718, and the process of the automatic file deletion mode is performed. When the process is completed, the process returns to step 702, and the process is repeated. When it is determined in step 717 that the mode to be set is not the automatic file deletion mode, the process returns to step 702, and the process is repeated.

[0163] Figure 8 : is a flowchart illustrating an example of the operation handled by the second control section 211 of the image pickup apparatus 101 in the present embodiment.

[0164] When the user operates the power button of the image pickup device 101 as in the case where power is supplied from the power supply section to the first control section 223 through the first power supply section 210, power is supplied from the power supply section to the second control section 211 by the second power supply section 212 also in the second control section 211. When power is supplied, the sub-processor (second control section 211) starts, and Figure 8 The processing starts.

[0165] In step 801, it is determined whether a predetermined period of time as a sampling time interval has passed. For example, when the predetermined period of time is set to 10 milliseconds, the process proceeds to step 802 at intervals of 10 milliseconds. When it is determined that the predetermined period of time has not passed, the sub-processor returns to step 801 without performing any processing, and waits until the predetermined period of time has passed.

[0166] At step 802, learning information is loaded. The learning information is Figure 7 The information transmitted when the information is transmitted to the secondary processor in step 706, and for example, the following information is loaded.

[0167] (1) Determining conditions for specific shake detection

[0168] (2) Determining conditions for specific sound detection

[0169] (3) Determination conditions for elapsed time determination

[0170] When the learning information is loaded in step 802, the process proceeds to step 803, and a shake detection value is acquired. The shake detection value is an output value from a sensor that detects vibration, such as a gyro sensor and an acceleration sensor of the device shake detection section 209.

[0171] When the shake detection value is obtained in step 803, the process proceeds to step 804, and a process of detecting a preset shake state is performed. Here, the determination process is changed based on the learning information loaded in step 802. Some examples will be described.

[0172] (1) Tap detection

[0173] The state (tapping state) in which the user taps the image pickup device 101, for example, with a fingertip, etc., can be detected by the output value of the acceleration sensor provided in the image pickup device 101. By passing the output of the three-axis acceleration sensor to a bandpass filter (BPF) set in a predetermined frequency range with predetermined sampling, the signal range of the acceleration change based on the tapping can be extracted. The tapping is detected based on whether the number of times the acceleration signal passing through the BPF exceeds a predetermined threshold ThreshA in a predetermined time TimeA is a predetermined count CountA. In the case of a double-click, CountA is set to two. In the case of a triple tap, CountA is set to three. TimeA or ThreshA may also vary according to learning information.

[0174] (2) Detection of shaking state

[0175] The shaking state of the image pickup device 101 can be detected based on the output value of the gyro sensor or the acceleration sensor provided in the image pickup device 101. The high-frequency component of the output of the gyro sensor or the acceleration sensor is cut off with the HPF, the low-frequency component is cut off with the LPF, and then the absolute value conversion is performed. Vibration is detected based on whether the number of times the absolute value calculated in the predetermined time TimeB exceeds the predetermined threshold ThreshB is greater than or equal to the predetermined count CountB. For example, it can be determined whether the shaking state is a small shaking state like placing the image pickup device 101 on a table or a large shaking state like the user is walking with the wearable image pickup device 101. When multiple conditions for determining the threshold and multiple conditions for determining the number of counts are provided, a small shaking state for the shaking level can be detected.

[0176] TimeB, ThreshB or CountB may also vary according to the learning information.

[0177] The method of detecting a specific shaking state based on the condition determination of the shaking detection sensor is described above. However, when the data of the shaking detection sensor sampled within a predetermined time is input to the shaking state determination device using a neural network, the trained neural network can detect the specific shaking state registered in advance. In this case, loading the learning information in step 802 will load the weight parameters of the neural network.

[0178] When the specific shaking state detection process is performed in step 804, the process proceeds to step 805, and the preset specific sound detection process is performed. Here, the detection determination process is changed according to the learning information loaded in step 802. Some examples will be described.

[0179] (1) Specific voice command detection

[0180] Detecting a specific voice command. The voice command may be some commands registered in advance, and the user may register a specific voice in the image pickup device.

[0181] (2) Specific sound scene recognition

[0182] Sound scene determination is performed using a network pre-trained by machine learning based on a large amount of speech data. For example, specific scenes such as "cheering", "clapping", and "speaking" are detected. The scenes to be detected change according to the learning.

[0183] (3) Sound level determination

[0184] The detection based on the sound level determination is performed by, for example, a method of adding the time period during which the magnitude of the sound level exceeds the predetermined level within a predetermined time. The predetermined time, the magnitude of the predetermined level, etc. vary according to learning.

[0185] (4) Sound direction determination

[0186] It is possible to detect the direction of a sound in a plane in which a plurality of microphones are installed, and detect the direction of the sound for a sound level of a predetermined magnitude.

[0187] It is determined in step 805 whether the above-described determination processing has been performed in the voice processing section 214 , and whether specific sound detection has been performed under the settings learned in advance.

[0188] When the specific sound detection process is performed in step 805, the process proceeds to step 806. In step 806, it is determined whether the main processor (first control unit 223) is in the off state. When the main processor is in the off state, the process proceeds to step 807, and the preset elapsed time detection process is performed. Here, the detection determination process is changed according to the learning information loaded in step 802. The learning information is in reference to Figure 7 Information transmitted when information is transmitted to the sub-processor (second control unit 211) in step 706 described above. The time elapsed from the time the main processor is switched from the on state to the off state is measured. When the elapsed time is greater than or equal to the parameter TimeC, it is determined that the time has elapsed; and when the elapsed time is shorter than TimeC, it is determined that the time has not elapsed. TimeC is a parameter that changes according to the learning information.

[0189] When the elapsed time detection process is performed in step 807, the process proceeds to step 808, and it is determined whether the low power consumption mode is determined to be released. The following low power consumption mode release condition is used for the determination.

[0190] (1) Determining conditions for specific shake detection

[0191] (2) Determining conditions for specific sound detection

[0192] (3) Determination conditions for elapsed time determination

[0193] Whether the determination condition for specific shaking detection is satisfied may be determined by the specific shaking state detection process in step 804. Whether the determination condition for specific sound detection is satisfied may be determined by the specific sound detection process in step 805. Whether the determination condition for elapsed time detection is satisfied may be determined by the elapsed time detection process in step 807. Therefore, when any one or more conditions are satisfied, it is determined that the low power consumption mode is released.

[0194] When it is determined in step 808 that the release condition is satisfied, the process proceeds to step 809 and the power of the main processor is turned on. In step 810, the condition (shake, sound, time) based on which the release of the low power consumption mode is determined is provided to the main processor. The process returns to step 801 and the process loops.

[0195] When any release condition is not satisfied and it is determined not to release the low power consumption mode in step 808 , the process returns to step 801 , and the process loops.

[0196] When it is determined in step 806 that the main processor is in the on state, the information acquired from step 803 to step 805 is provided to the main processor. The process returns to step 801, and the process loops.

[0197] In this embodiment, even when the main processor is in the on state, the shaking detection or the specific sound detection is performed by the sub-processor, and the detected result is provided to the main processor. However, when the main processor is on, the processing of steps 803 to 805 may not be performed, and the shaking detection or the specific sound detection may be performed in the processing of the main processor ( Figure 7 is detected in step 702).

[0198] As mentioned above, by executing Figure 7 Steps 704 to 707 and Figure 8 The conditions for moving to the low power consumption mode and the conditions for releasing the low power consumption mode are learned based on the user's operation. Therefore, an image pickup operation suitable for use by the user who owns the image pickup device 101 can be performed. The learning method will be described later.

[0199] The method of releasing the low power consumption mode based on shaking detection, sound detection or elapsed time is described in detail above. Alternatively, the low power consumption mode can be released based on environmental information. Environmental information can be determined based on whether the absolute amount or change in temperature, atmospheric pressure, brightness, humidity, ultraviolet rays exceeds a predetermined threshold, and the threshold can be changed according to learning (described later).

[0200] Whether to release the low power consumption mode as a result of determination of information about shake detection, sound detection, or elapsed time, or the absolute value or change of each piece of environmental information can be determined based on a neural network, and in this determination process, the determination conditions can be changed based on learning (described later).

[0201] <Handling of automatic shooting mode>

[0202] Reference Fig. 9Details of the processing of the automatic shooting mode are described. As described above, the first control section 223 of the image pickup apparatus 101 in the present embodiment handles control of the following processing.

[0203] In S901 , the image processing section 207 is caused to perform image processing on the signal captured by the imaging section 206 , and to generate an image for subject recognition.

[0204] Recognize subjects such as people and objects from the generated images.

[0205] When a person is recognized, the face or body of the subject is detected. In the face detection process, a pattern for determining the face of a person is set in advance, and a portion included in the captured image and matching the pattern can be detected as a face image of the person.

[0206] At the same time, a confidence level indicating the possibility of a face being the subject is also calculated. The degree of consistency is calculated based on, for example, the size of the face region in the image, the degree of consistency with the face pattern, and the like.

[0207] Also for object recognition, similarly, objects matching pre-registered patterns can be recognized.

[0208] For example, there is also a method of extracting a feature subject using a method of a histogram of hue, chroma, etc. in a captured image. In this case, with respect to an image of a subject captured within a shooting angle of view, a process of dividing a distribution derived from a histogram of hue, chroma, etc. of the image into a plurality of parts and classifying the captured image of each part is performed.

[0209] For example, histograms of a plurality of color components are created for a captured image, each histogram is divided into a mountain-shaped distribution range, the captured image is classified into regions belonging to a combination of the same intervals, and the image region of the subject is identified.

[0210] When the evaluation value is calculated for each image region of the recognized subject, the image region of the subject having the highest evaluation value may be determined as the main subject region.

[0211] By using the above method, each piece of subject information can be obtained from the shooting information.

[0212] In S902, the image stabilization amount is calculated. Specifically, first, the absolute angle of the image pickup device is calculated based on the information on the angular velocity and acceleration obtained by the device shake detection section 209. Then, the vibration control angle for actuating the tilt unit 104 and the pan unit 105 in the direction to cancel the absolute angle is found and set as the image stabilization amount. In the process of calculating the image stabilization amount here, the calculation method can be changed according to the learning process (described later).

[0213] In S903, the condition of the image pickup device is determined. The vibration / motion condition in which the image pickup device is placed is determined based on, for example, the angle and movement amount detected from angular velocity information, acceleration information, GPS position information, and the like.

[0214] For example, when the image pickup device 101 is mounted on a vehicle and captures an image, subject information such as surrounding scenery varies significantly depending on the distance moved.

[0215] Therefore, it is determined whether the image pickup apparatus 101 is in a “vehicle moving state” in which the image pickup apparatus 101 is mounted on a vehicle or the like and is moving at a high speed, and can be used for automatic subject search to be described later.

[0216] It is also determined whether the angle change is large, and whether the image pickup device 101 is placed in a “still capturing state” with almost no pan angle.

[0217] In the case of the “still capturing state”, it can be considered that the angle of the image pickup device 101 itself has not changed, and thus the subject search for still capturing can be performed.

[0218] When the change in the angle is relatively large, it is determined that the image pickup apparatus 101 is in the “hand-held state”, and thus a subject search for the hand-held state can be performed.

[0219] In S904, a subject search process is executed. The subject search is composed of the following processes.

[0220] (1) Regional division

[0221] The area division will be described with reference to FIG. Fig.13A As shown in , the area division is performed around the entire periphery of the position of the image pickup device (the origin O is defined as the position of the image pickup device). Fig.13A In the example of , area division is performed every 22.5 degrees in each of the tilt direction and the pan direction. Fig.13A When performing region division as shown in , as the angle in the tilt direction moves away from zero degrees, the perimeter in the horizontal direction decreases and the area decreases. Fig. 13B As shown in , when the inclination angle is greater than or equal to 45 degrees, the area range in the horizontal direction is set to be greater than 22.5 degrees. Fig. 13C and Fig.13D An example of dividing the area within the shooting angle of view is shown. Axis 1301 is the direction of the image pickup device 101 at the time of initialization, and the area is divided by setting this direction angle as the reference position. 1302 indicates the angle of view area of ​​the image being captured, and an example of the image at this time is in Fig.13D In the image appearing in the viewing angle, based on the region division, such as Fig.13D The image is divided as in the case of 1303 to 1318.

[0222] (2) Calculation of importance level for each area

[0223] For each area divided as described above, an importance level indicating the search priority order is calculated based on the subject existing in the area and the scene condition of the area. For example, the importance level based on the condition of the subject is calculated based on the number of people existing in the area, the size of each person's face, the face orientation, the possibility of face detection, the facial expression of each person, and the personal authentication result of each person. The importance level based on the condition of the scene is, for example, a general object recognition result, a scene recognition result (blue sky, backlight, night scene, etc.), a sound level and a voice recognition result from the area, and the mobile sensing information in the area. The vibration condition of the image pickup device has been detected when determining the condition of the image pickup device (S903), and the importance level can also be configured to change according to the vibration condition. For example, when it is determined to be a "static shooting state", when a face identification of a specific person is detected, the importance level is determined to be high, so that the subject search is performed by focusing on the subject with a higher priority among the subjects registered with the face identification (for example, the user of the image pickup device). Automatic shooting (described later) is also performed preferentially on the above-mentioned face. Even when the period of time during which the user of the image pickup device wears the image pickup device on the body for shooting is long, many images containing the user can be recorded when the image pickup device is removed and placed on a table, etc. At this time, it is possible to search with the help of panning / tilting, so even when the placement angle of the image pickup device, etc. is not considered, group photos containing user images and containing many faces can be recorded only by roughly placing the image pickup device. Only under the above conditions, unless the area changes, the area with the highest importance level is the same, with the result that the discovered area always remains unchanged. Therefore, the importance level changes according to the previous shooting information. Specifically, the importance level of an area that has been continuously designated as a search area for a predetermined time can be lowered, or the importance level of an area where shooting is performed in S910 (described later) can be lowered for a predetermined time.

[0224] (3) Determination of search target area

[0225] When the importance level of each area is calculated as described above, an area having a high importance level is decided as a search target area. A target pan / tilt search angle required to capture the search target area into the viewing angle is calculated.

[0226] In S905, the camera is panned / tilted. Specifically, the pan / tilt amount is calculated by adding the drive angle at the control sampling based on the image stabilization amount and the target pan / tilt search angle, and the tilt unit 104 and the pan unit 105 are each controlled to be driven by the barrel rotation drive section 205.

[0227] In S906, the zoom unit 201 is controlled, and the camera is zoomed. Specifically, the camera is zoomed according to the condition of the search target subject set in S904. For example, when the search target object is a person's face, a face that is too small on the image cannot be detected because the size is smaller than the minimum detectable size, and may be lost. In this case, the zoom unit 201 is controlled to zoom to the telephoto side so that the size of the face on the image increases. On the other hand, when the face on the image is too large, the subject may fall outside the angle of view depending on the movement of the subject or the image pickup device itself. In this case, the zoom unit 201 is controlled to zoom to the wide-angle side so that the size of the face in the frame is reduced. By performing zoom control in this way, a state suitable for tracking the subject can be maintained.

[0228] In S904 to S906, a method for performing a subject search by panning / tilting or zooming is described. Alternatively, a subject search can be performed by using an image pickup system that shoots all directions at once using multiple wide-angle lenses. In the case of an omnidirectional camera, if image processing such as subject detection is performed on all signals obtained by shooting as input images, a large amount of processing is required. Therefore, a part of the image is cropped, and a process of searching for a subject is performed within the range of the cropped image. As in the case of the above method, the importance level of each area is calculated, the cropping position is changed based on the importance level, and a determination about automatic shooting is performed (described later). Therefore, it is possible to reduce power consumption and perform high-speed subject search by image processing.

[0229] In S907, it is determined whether there is a (manual) shooting instruction of the user. When there is a shooting instruction, the process proceeds to S910. At this time, the (manual) shooting instruction of the user may be pressing the shutter button of the image pickup device 101 or pressing the shutter button of the hand-held control accessory 5001. Alternatively, the (manual) shooting instruction of the user may be tapping the housing of the image pickup device with a finger or the like, voice command input, instructions from an external device, etc. The shooting instruction provided by tapping is a shooting instruction method in which the device shake detection unit 209 senses successive high-frequency accelerations in a short period of time and triggers shooting when the user taps the housing of the image pickup device. The voice command input is a shooting instruction method in which the voice processing unit 214 recognizes the voice and triggers shooting when the user speaks a password for providing instructions for predetermined shooting (for example, "take a photo", etc.). The instruction from the external device is a shooting method using a shutter instruction signal transmitted from, for example, a smart phone connected to the image pickup device via Bluetooth, etc., via a dedicated application as a trigger.

[0230] In S908, automatic shooting determination is performed. In the automatic shooting determination, determination is performed as to whether to perform automatic shooting and determination of the shooting method (determination of which of single shooting of still images, continuous shooting (continuous exposure) of still images, video shooting, panoramic shooting, time-lapse shooting, etc. is performed).

[0231] In order to shoot an object in front of the user's eyes, a scenario can be conceived in which the user shoots while pushing the image pickup device slightly forward while wearing the image pickup device on the user. For example, in such a scenario, it is expected that a still image is usually shot quickly. In order to shoot a scenic area, a scenario can be conceived in which the user holds the image pickup device hanging from the neck and shoots while lifting the image pickup device to the front and above. In such a scenario, for example, even in the case of expected or unexpected panoramic shooting, when the image obtained by panoramic shooting is provided to the user, the user will learn this shooting method, and this experience can be used for future shooting. Therefore, in order to identify these scenes, the scene is identified, for example, by detecting the moving distance of the image pickup device from the holding state to the shooting. In this way, the preferred shooting method can vary depending on the way the image pickup device is set, and thus the shooting method can be changed according to the condition of the image pickup device when it is moved to the shooting. For the user's purpose, in the case of a scene in which the subject is placed slightly upward and the image pickup device is lifted upward to shoot the subject, it is expected to shoot a normal still image quickly. Therefore, in order to perform these identifications, for example, the subject distance is detected. Therefore, the scene can be identified. In this way, since the preferred shooting method can be changed according to the condition of the subject, the shooting method can be changed according to the condition of the image pickup device when moving to shooting and the condition of the subject seen. In order to shoot a subject that is long in the vertical direction in front of the user's eyes, it is recommended that the user hold the image pickup device hanging from the neck and shoot while holding the image pickup device facing upward. Such a scene can be imagined, such as shooting an image of an attic building at a sightseeing place. In such a scene, for example, even in the case where vertical panoramic shooting is expected or not expected, when the image obtained by vertical panoramic shooting is provided to the user, the user learns this shooting method, and this experience can be used for further shooting. Therefore, in this case, the holding angle is detected. Therefore, the scene can be identified. The accuracy of determining whether vertical panorama or horizontal panorama is preferred can be improved by further determining, for example, the distance to the subject and the distance to the subject in the top, bottom, right and left areas as the condition of the subject at this time. In other words, when the distance to the subject is equal to the distance to the top and bottom areas of the subject, it can be determined that vertical panorama shooting is to be performed. For 360-degree shooting, a scenario can be conceived in which the user holds the image pickup device hanging from the neck and shoots while holding the image pickup device facing directly upward. A scenario can be conceived in which, for example, an image showing a state of looking around at a mountain top is shot. At this time, it can be conceived that, for example, the user sets the mode to the 360-degree shooting mode using an external device and provides a shooting instruction.Therefore, for example, in this case, when a user interface for asking whether to move to 360-degree shooting is provided when the user operates the external device, the time and effort for the user's operation can be reduced. In addition, although this operation is repeated several times, the user expects that 360-degree shooting can be performed by simply pressing the shutter button with the external device facing directly upward without operating the external device. Therefore, in this case, for example, when the moving direction of the image pickup device from the gripping state to the shooting is the directly upward direction, 360-degree shooting is performed, which can reduce the time and effort of the user for shooting.

[0232] In this way, by changing the photographing method based on the condition of the image pickup device and the condition of the subject, the user's time and effort during photographing and during checking the photographed images can be reduced.

[0233] In the determination of the expected shooting method based on the condition of the image pickup device and the condition of the subject and the determination method in <Determination of whether to perform automatic shooting>, the shooting method can be determined by determination based on a neural network. In this determination process, the determination condition can be changed on a user-by-user basis through a learning process (described later). In this case, a plurality of images are recorded with a plurality of shooting methods in the initial stage of learning, and in the learning process (described later), the determination condition can be changed according to which shooting method the user prefers.

[0234] In this way, the determination of the shooting method suitable for the user's intention in the case of a manual shooting instruction of the user is described. Like S908b, in the case of no manual shooting instruction, this processing is suitable for automatic shooting determination processing. In other words, when it is determined that the image pickup device is set, the shooting method reflecting the user's intention can be similarly determined by detecting the way in which the image pickup device is set.

[0235] By monitoring the acceleration of the image pickup device, even during automatic shooting, the way the image pickup device is set can be detected, and a shooting method reflecting the user's intention can be determined.

[0236] (1) Determination of whether to execute automatic shooting

[0237] The determination as to whether to perform automatic shooting is performed based on the following two determinations. One determination is to determine that automatic shooting is performed when the importance level based on the importance level of each area obtained in S904 exceeds a predetermined value. The second determination is a determination based on a neural network. As an example of a neural network, Fig.12An example of a network with a multilayer perceptron is shown in . The neural network is used to predict output values ​​based on input values, and when the neural network has previously learned input values ​​and model output values ​​for the inputs, it is able to estimate output values ​​for new input values ​​based on the learned model. The method of learning will be described later. Fig.12 , 1201 arranged in a column and the circles thereafter are neurons of the input layer, 1203 arranged in a column and the circles thereafter are neurons of the middle layer, and 1204 are neurons of the output layer. Arrows like 1202 represent the connection of connected neurons. In the determination based on the neural network, the subject appearing at the current viewing angle or feature amount based on the condition of the scene or image pickup device is given as input to the neurons of the input layer, and a value is output from the output layer by calculation based on the forward propagation rule of the multilayer perceptron. When the output value is greater than or equal to the threshold, it is determined to perform automatic shooting. The features of the subject are the current zoom factor, the general object recognition result at the current viewing angle, the face detection result, the number of faces contained in the current viewing angle, the degree of smile and the degree of eye closure of each face, the face angle, the face identification ID number, the sight angle of the subject character, the scene recognition result, the detection result of a specific composition, etc. The time elapsed since the last shooting, the current time, the GPS location information, the deviation from the last shooting position, the current voice level, the speaker, the clapping or cheering, etc. can be used. Vibration information (acceleration information, condition of the image pickup device), environmental information (temperature, atmospheric pressure, illuminance, humidity, ultraviolet light amount), etc. can be used. In addition, when information is provided from the wearable device 501, the provided information (user's motion information, information about the movement of the arm, biological information such as heart rate, etc.) can be used as features. These features are converted into numerical values ​​within a predetermined range and given to the neurons of the input layer as feature quantities. Therefore, the number of neurons required for the input layer is the same as the number of feature quantities to be used.

[0238] In determination based on a neural network, the output value is changed by changing the combination weight between neurons in a learning process (described later), and the result of the determination can be adapted to the learning result.

[0239] The determination for automatic shooting also depends on Figure 7 The startup condition of the main processor loaded in step 702 of . For example, in the case of startup based on tap detection or startup based on a specific voice command, it is very likely that this is an operation that the user currently wants to shoot. Therefore, the shooting frequency is set to increase it.

[0240] (2) Determination of shooting method

[0241] When determining the shooting method, a determination as to which of still image shooting, video shooting, continuous shooting, panoramic shooting, etc. is performed based on the conditions of the image pickup device and the conditions of the surrounding subjects detected in S901 to S904. For example, when the subject (person) is stationary, still image shooting is performed, and when the subject is moving, video shooting or continuous shooting is performed. When there are multiple subjects around the image pickup device or it is determined to be in a scenic area based on the above-mentioned GPS information, a panoramic shooting process that generates a panoramic image by combining images sequentially shot when panning / tilting can be performed. As in the case of the determination method in <Determination as to whether to perform automatic shooting>, the shooting method can be determined by a determination based on a neural network using various information detected before shooting, and in the determination process, the determination conditions can be changed by a learning process (described later).

[0242] When it is determined in S909 that shooting is to be performed according to the automatic shooting determination of S908 , the process proceeds to S910 ; otherwise, the process proceeds to the end of the processing of the shooting mode.

[0243] In S910, automatic shooting is started. At this time, shooting is started using the shooting method determined in S908. At this time, automatic focus control is performed by the focus control unit 204. Exposure control for setting the subject to an appropriate brightness is performed by using an exposure control unit, a sensor gain control unit, and a shutter control unit (not shown). In addition, after shooting, the image processing unit 207 generates an image by performing various image processing (such as automatic white balance, noise reduction, and gamma correction).

[0244] At the time of shooting, a method may be adopted in which, when a predetermined condition is satisfied, a process of notifying the person to be shot that the image pickup device is shooting and then shooting is performed. The notification method may, for example, use voice from the voice output section 218 or LED flashing light generated by the LED control section 224, or perform a movement that visually guides the line of sight of the subject by panning / tilting. The predetermined condition is, for example, the number of faces in the viewing angle, the degree of smile and the degree of eye closure of each face, the angle of sight or the angle of face of the subject person, the face identification ID number, the number of people with registered personal identification, etc. The predetermined condition is also a general object recognition result during shooting, a scene recognition result, the time elapsed from the last shooting, the shooting time, whether the current position based on GPS information is a scenic spot, the voice level during shooting, whether there is a speaker, whether it is clapping or cheering, etc. The predetermined condition is also vibration information (acceleration information, the condition of the image pickup device), environmental information (temperature, atmospheric pressure, illumination, humidity, ultraviolet light amount), etc. When shooting with notification is performed based on these conditions, an image with a camera-favored eye can be recorded in a very important scene.

[0245] Alternatively, a plurality of predetermined conditions may be provided, and according to these conditions, the voice may be changed, the LED lighting method (such as color and blinking time) may be changed, or the pan / tilt movement method (movement pattern and driving speed) may be changed.

[0246] For such notification before shooting, the method or timing of notification can also be determined by information about the shot image or by determination based on a neural network using various information detected before shooting. In this determination process, the determination conditions can be changed by learning processing (described later).

[0247] In S911, an editing process is performed in which the image generated in S910 is processed or added to a moving image. The image processing is specifically cropping based on a person's face or focus position, rotation of an image, an HDR (High Dynamic Range) effect, a blur effect, a color conversion filter effect, and the like. In the image processing, a plurality of images can be generated based on the image generated in S910 by using a combination of the above-mentioned processes, and the generated images can be saved separately from the images generated in S910. In the moving image processing, a process of adding a captured moving image or a still image to a generated edited moving image while applying special effects of sliding, zooming, and fading can be performed. In the editing in S911, the method of image processing can also be determined by determination based on a neural network by information about the captured image or by using various information detected before shooting, and in this determination process, the determination conditions can be changed by a learning process (described later).

[0248] In S912, a process of generating learning information of the captured image is performed. Here, information used in the learning process (described later) is generated and recorded. Specifically, the information is the scaling factor during the shooting in the currently captured image, the general object recognition result during the shooting, the face detection result, the number of faces contained in the captured image, the degree of smile and eye closure of each face, the face angle, the face identification ID number, the line of sight angle of the subject person, etc. The information is also the scene recognition result, the time elapsed since the last shooting, the shooting time, the GPS location information, the deviation from the last shooting position, the voice level during the shooting, the speaker, the clapping or cheering, etc. The information is also vibration information (acceleration information, the condition of the image pickup device), environmental information (temperature, atmospheric pressure, illumination, humidity, ultraviolet light amount), video shooting time, whether it is based on a manual shooting instruction, etc. In addition, a score is calculated, which is the output of the neural network and is digitized according to the user's image preference.

[0249] These information are generated and recorded in the captured image file as tag information. Alternatively, a plurality of pieces of information on the captured image may be written in the nonvolatile memory 216 or saved in the recording medium 221 in a list format as so-called directory data.

[0250] In S913, the previous shooting information is updated. Specifically, the number of images shot for each area, the number of images shot for each person with a registered personal identification, the number of images of each subject identified by general object recognition, or the number of images shot for each scene in scene recognition described in S908 associated with the currently shot image is increased by one count.

[0251] (Example of manual shooting using voice recognition)

[0252] like Fig. 9 As described in S907 of the present invention, the user's (manual) shooting instruction also includes an instruction based on a voice command input. The voice command input includes a voice command input in the case where the user wants to shoot an image containing the user himself or herself (for example, "take a picture of me", etc.). Then, in the search process using pan / tilt or zoom, the subject who speaks is searched, and an image in which the subject who speaks the voice command is included in the shooting angle of view is shot.

[0253] Reference Fig.24 The flowchart shown in describes the use of voice commands to a user to take a picture of himself or herself. Fig.24 Yes Execution Fig. 9 The process determined within the process of S907.

[0254] In the manual shooting instruction process of S907, it is determined whether shooting is based on voice command input. In S2401, it is determined whether the voice processing unit 214 detects a specific voice command input (e.g., "Take my photo", etc.). When a voice command is not detected, the process proceeds to S2416, and the manual voice control shooting determination process is ended without manual shooting determination. When a voice command is detected in S2401, the process proceeds to S2402.

[0255] In S2402, the direction of the sound from which the voice command is detected is calculated, and the first sound direction, the second sound direction, the third sound direction, or the fourth sound direction in descending order of the confidence of the sound direction is calculated as a candidate. When the accuracy of detecting the sound direction is very high, it is not necessary to calculate multiple candidates or to perform subsequent search processing, etc. However, due to the condition of ambient noise when detecting the voice command, the influence of the object environment such as sound reflection, etc., noise may be included in the detected sound direction, so multiple candidates are calculated.

[0256] Specifically, candidates are calculated based on all detected sound direction values ​​within a predetermined time before the time point when the voice command is detected. The time when a pre-registered voice command is spoken can be predicted to a certain extent (for example, when "take my photo" is a command, the time taken to speak the command is pre-set as a parameter). Based on all detected sound direction values ​​detected within a predetermined time, in the case of Fig.25 In the histogram processing shown, the first peak 2501 is set as the first sound direction, and the second peak 2502 is set as the second sound direction. The first sound direction and the second sound direction are calculated, and the process proceeds to S2405.

[0257] In S2403, it is determined whether pan / tilt retry is set. For pan / tilt retry setting, pan / tilt retry is set later in S2415, and pan / tilt retry determination is not performed at the time point of the manual voice control shooting determination processing that starts this flow. When pan / tilt retry is not set in S2403, the processing proceeds to S2404, and the first sound direction calculated in S2402 is set as the sound direction. When it is determined in S2403 that pan / tilt retry is set, the processing proceeds to S2405, and the second sound direction calculated in S2404 is set as the sound direction. When the processing of S2404 and S2405 is completed, the processing proceeds to S2406.

[0258] In S2406, it is determined whether the difference between the set sound direction and the current pan / tilt angle falls outside the predetermined range, that is, whether the difference between the sound direction and the current center of the angle of view falls outside the predetermined range. When the difference falls outside the predetermined range, the process proceeds to S2407, the camera is panned / tilted to adjust the pan / tilt angle so that the detected sound direction reaches the center of the angle of view, and the process proceeds to S2408. When the difference between the sound direction and the current center of the angle of view falls within the predetermined range in S2406, the sound direction is located near the center within the angle of view, so the process proceeds to S2408 without panning or tilting.

[0259] In S2408, by processing and analyzing the captured image, it is determined whether the main object falls within the current viewing angle. The specific determination method is described as follows.

[0260] (1) Detecting the main subject through convolutional neural network

[0261] Detection of the main subject by a convolutional neural network is known as a general machine learning method for image recognition processing. Through the convolutional neural network, whether there is a detected main subject (the subject who speaks) is obtained, and when the main subject exists, position information about the image is also obtained. Alternatively, main subject determination can be performed by a convolutional neural network for each image of an area of ​​each person cropped based on the results of face detection and body detection, and the main subject can be estimated. This convolutional neural network is prepared as a convolutional neural network that is pre-trained based on an image of a person who speaks a voice command; however, the convolutional neural network can be trained when it is used in the method described later.

[0262] (2) Detecting the main subject through a neural network

[0263] There is a method in which, for each person included in the current viewing angle, main subject determination is performed for each person by using the feature amount of the subject as an input. In this case, not only facial features of the subject person (such as facial expression determination result, degree of eye closure, face angle, face identification ID number, and sight angle) but also gesture determination result, image scene determination result, current sound level, sound scene determination result, etc. can be used as features to be input. This neural network is also a neural network trained based on the subject feature amount and based on the image of the person who speaks the language command; however, the neural network can be trained when used in the method described later.

[0264] (3) Detecting subjects by determining each person’s characteristics

[0265] Since the subject has just spoken a voice toward the image pickup device 101, the possibility that the subject is facing the camera is very high. Therefore, weight coefficients can be assigned to the detection results of the facial identification ID number, facial expression results, facial angles, sight lines, and gesture determination results, and determination can be simply performed. When the facial identification ID has been registered, the possibility that the person is the main subject is high. When the smile degree of the facial expression is high, the possibility that the person is the main subject is high. When the facial angle or sight line direction points to the camera, the possibility that the person is the main subject is high. When a gesture is performed (for example, waving to the camera, etc.), the possibility that the person is the main subject is high. The main subject can be estimated by using one or more pieces of information.

[0266] Whether or not a main subject exists within the current angle of view may be determined by using any one of the methods, or may be determined by using a combination of two or more of (1) to (3).

[0267] After performing image analysis in S2408, the processing proceeds to S2409, and it is determined whether the main subject is found in the processing of S2408. When it is determined in S2409 that the main subject exists, the processing proceeds to S2410. In S2410, composition adjustment is performed by zooming or panning / tilting, and the processing proceeds to S2411. Determination of a composition suitable for photographing an image containing the main subject can be performed by determination using a neural network. The output value is changed by changing the combination weights between neurons in a learning process (described later), and the determined result can be adapted to the learning result.

[0268] In S2411, it is determined that there is a manual shooting instruction, and the process proceeds to S2416. The manual voice control shooting process ends, and the process proceeds to Fig. 9 Then start shooting.

[0269] When it is determined in S2409 that there is no main subject within the current angle of view, the process proceeds to S2412 .

[0270] In S2412, it is determined whether a predetermined time has passed since the pan / tilt in S2407 was completed. When it has been determined in S2406 that the difference between the sound direction and the current center of the angle of view falls within a predetermined range, it is determined based on the time that has passed from the determined time point. Here, when the predetermined time has not yet passed, the processing proceeds to S2413, and a search using zoom is performed. When the subject who speaks the voice command is very small within the angle of view, the size of the face is small and the resolution of the face is also low, so this affects the detection accuracy based on image analysis. Therefore, in this case, the camera zooms in the display to narrow the angle of view, and performs the processing from S2408 again. When the subject who speaks the voice command is too large within the angle of view, the overall image of the person cannot be seen, and it cannot be determined by considering, for example, making gestures. Therefore, the camera zooms out the display to expand the angle of view, and then performs the processing from S2408 again.

[0271] When it is determined in S2412 that a predetermined time has passed since the pan / tilt was completed, it is determined that no subject has spoken a voice command in the set sound direction. Then, in order to search for the next sound direction candidate, the processing proceeds to S2414, and it is determined whether a pan / tilt retry operation has been performed before. When the retry has not been performed, the processing proceeds to S2415, and the pan / tilt retry is set, and then the processing returns to S2403. At this time, since the pan / tilt retry is set, the viewing angle is set to the sound direction as the second sound direction of S2405 through the determination of S2403, and the process of searching for the main subject is performed again. Fig.24In the example of , a method of searching for two candidates (ie, a first sound direction and a second direction) is shown; however, a third or fourth direction may be detected, and the retry may be repeated.

[0272] When it is determined in S2414 that the pan / tilt retry has been performed, the process proceeds to S2416, and the manual voice control shooting determination process is ended without performing a manual shooting determination. At this time, in order to notify the user that shooting is not performed although a voice command is detected, a method of performing a notification process may be adopted. The notification method may, for example, use a voice from the voice output unit 218 or an LED flashing light generated by the LED control unit 224. The notification method may be a movement that visually guides the subject's line of sight by panning / tilting, or may be a method of providing communication and notification to the smart device 301 or the wearable device 501.

[0273] exist Fig.24 , capturing caused by a voice command input from a user in a case where the user wants to capture an image containing the user himself or herself is described. However, the voice command input may be a voice command input in a case where the user wants to register the user himself or herself as a main subject (e.g., "Register me," "Track me," etc.). In this case, in the above-mentioned search process, the subject who spoke the voice is searched, and the subject is registered. When the subject is registered, automatic capturing is performed by focusing on the registered subject thereafter. Capturing can be performed while always keeping the registered subject within the angle of view by panning / tilting or zooming.

[0274] When you input a voice command for subject registration, you can Fig.24 The process of S2411 within the process of is changed to the process of registering the subject to realize subject registration.

[0275] When the subject is also registered, detection and registration processing is performed by panning / tilting or zooming so that the viewing angle allows easy face recognition registration and registration of the detected color of the worn clothes and the like is easily performed.

[0276] When the image pickup device 101 does not have a monitor, it is impossible to check the registered subject. Fig.26 As shown in , the fact that the subject is registered may be notified to the smart device 301 , or image data of the registered subject may be transmitted to allow the user to check.

[0277] When a subject is registered, data providing a notification 2602 that the registration can be seen is sent to the smart device 301 by the communication section 222 through the communication 2601. When the user checks the subject by operating the smart device 301, the image pickup apparatus 101 sends 2603 image data, and causes the smart device to display the subject 2604 registered in the smart device so that the user can check. The display that the user can check may be a display in which an image related to the registered subject 2604 is superimposed at or near (below) the face frame to indicate that the image pickup apparatus 101 has recognized the face. The related image may be displayed during video shooting, or may be displayed during video playback.

[0278] By providing the registered subject so that the user can check it in this way, even if the registered subject is wrong, the user can be prompted to register the subject again. Since it can be determined whether it is correct or incorrect, the registered subject can be learned and changed through the learning process described later.

[0279] In the present embodiment, shooting and subject registration in the direction of sound by voice command input using both pan / tilt and zoom are described; however, shooting and subject registration may be performed by using only pan / tilt, or may be performed by using only zoom.

[0280] When using only pan / tilt, Fig.24 S2412 and S2413 are not executed.

[0281] When using only the zoom, after detecting the direction of the sound, the zoom is set so that the direction of the sound falls within the angle of view, and the main subject is searched with the zoom. In this way, shooting and subject registration are performed.

[0282] <Processing of automatic editing mode (highlight video)>

[0283] Next, the processing of the automatic editing mode (highlight video) in the present embodiment will be described.

[0284] exist Figure 7 When the mode to be set is determined in step 704, it is determined whether to perform automatic editing processing (wonderful video). When it is determined that the automatic editing processing is to be performed, the processing of the automatic editing mode of step 712 is performed.

[0285] The determination condition of the automatic editing mode will be described. Whether to move to the automatic editing mode is determined based on the time elapsed from the last editing process or the label information of each image captured after the time point when the last editing process was performed (such as learning information and a score that digitizes the user's image preference). The determination processing flow regarding whether to move to the automatic editing mode determined in the mode setting determination processing of step 704 is described in detail. Fig.10 Shown in.

[0286] When an instruction to start automatic editing mode determination is issued in the mode setting determination process of step 704, Fig.10 The processing starts. In step 1001, the time TimeD elapsed since the last automatic editing process was performed is obtained, and the processing proceeds to step 1002. In step 1002, learning information, scores, etc. associated with each image captured after the time point when the last editing process was performed are obtained, and the processing proceeds to step 1003. In step 1003, based on the data obtained in step 1002, an evaluation value DB for determining whether to perform automatic editing is calculated. One method of calculating the evaluation value is, for example, extracting features of the image from each piece of image information, and assigning a higher score when there are many types of features. As described in the automatic shooting above, a score determined for user preferences is calculated for each image, and a higher score is assigned to an image with a higher score. As the number of captured images increases, a higher score will be calculated. Therefore, the evaluation value depends on how high the score of the score is, on the number of images, and on the type of feature. The processing proceeds to step 1004. In step 1004, a threshold DA is calculated based on TimeD. For example, the threshold DAa in the case where TimeD is shorter than a predetermined value is set to be greater than the threshold DAb in the case where TimeD is greater than a predetermined value, and the threshold is set to decrease with time. Therefore, even in the case where the captured image data is small, the automatic editing process is performed when the elapsed time is long, resulting in the image pickup device being configured to automatically create a wonderful video according to the usage time.

[0287] When the processing of step 1004 is completed, the processing proceeds to step 1005. When the evaluation value DB is greater than the threshold value DA, the processing proceeds to step 1006. Since the case is that the data to be automatically edited is obtained after the time point when the last automatic editing is performed, or it is determined that the automatic editing is to be performed because a long time has passed, the automatic editing mode is set to "true" and the automatic editing mode determination ends. When the evaluation value DB is less than or equal to the threshold value DA in step 1005, it is determined that there is no data to be automatically edited, so the automatic editing mode determination is set to "false" so that the automatic editing process is not performed, and the automatic editing mode determination process ends.

[0288] Next, the processing in the automatic editing mode (step 712) will be described. The detailed flow of the automatic editing mode is described in Fig.11 Shown in.

[0289] In S1101 , a process of selecting from still images and moving images recorded in the storage medium 221 is performed in the first control section 223 to select an image to be used in editing, and the process proceeds to S1102 .

[0290] The image selection process here is to extract metadata (such as the number of faces, the size of each face, and the color group) in the still image or moving image taken of each image, convert the metadata into an evaluation value, and list images whose evaluation values ​​are greater than or equal to a set threshold. The selection ratio between still images and moving images is determined by learning (described later), and the selection is performed with priority in consideration of the user's settings, the frequency of shooting, and the settings.

[0291] In S1102 , an image effect is applied to the image selected in S1101 by the first control section 223 and the image processing section 207 , and the process proceeds to S1103 .

[0292] The application of image effects here is cropping a person's face or the center of focus position in a still image, image rotation, HDR (high dynamic range) effect, blur effect, special effects of sliding, zooming and fading, color filter effect, etc.

[0293] Also in moving images, color filters are similarly applied.

[0294] In S1103, the image playback time is set in the first control section 223, and the process proceeds to S1104. An appropriate image playback time is set based on learning (described later) to create a highlight video to be described in S1105 by using the image selected in S1101.

[0295] In S1104, music (BGM) to be applied to the highlight video to be described in S1105 is set in the first control section 223, and the process proceeds to S1105. To set the music (BGM), the most appropriate music to be provided to the user is also set based on the result of learning (described later).

[0296] In S1105, a series of highlight video creation is performed using the results of S1101 to S1104 performed in the first control section 223. The created highlight video is saved in the recording medium 221.

[0297] The selection of the above-mentioned images, application of image effects, playback time, and BGM selection can be determined through neural network-based determination using tag information added to each image (information about the captured image or various information detected before shooting). In this determination process, the determination conditions can be changed through a learning process (described later).

[0298] <Processing of Automatic File Deletion Mode>

[0299] Next, the processing of the automatic file deletion mode in the present embodiment will be described.

[0300] In this embodiment, when there is no free capacity in the recording medium, shooting cannot be performed, and there is a concern that shooting cannot be performed in response to the user's intention or that the expected scene cannot be shot in automatic shooting. The image can be deleted by the user's operation; however, this is complicated. Therefore, in the processing to be described below, it is necessary to automatically delete the captured images that meet the conditions. On the other hand, the user may delete images that are needed later, so it is necessary to select and delete appropriate images.

[0301] Will pass Fig.29 The processing is described with reference to an example of the processing flow of the automatic file deletion mode.

[0302] In S2901, the available capacity in the recording medium is checked. In S2902, the target number of images to be deleted is determined based on the available capacity in the recording medium. For example, the target number of images to be deleted is set to increase as the available capacity decreases, and is set to increase as the set shooting frequency increases. The target number of images to be deleted can be changed based on learning (described later). In S2903, a list of captured images saved in the recording medium and sorted in descending order of scores is created, which digitizes the user's image preferences (described later). From S2904, it is determined whether to delete images one by one from the top to the bottom of the sorted list, and the deletion process is performed. In S2905, it is determined whether the expected image on the list meets the deletion condition.

[0303] The deletion condition may be, for example, the fact that the image is not an image manually taken by the user or the image is not an image highly rated by the user. These are images that the user likes or may need later, so it is desirable to exclude them from the deletion condition.

[0304] The fact that the image has been transmitted to an external communication device (such as a smart device) in the automatic transmission mode, the fact that the user has not browsed the image from the external communication device, etc. can be used. When the image has been transmitted, the transmitted image is usable, so the deletion of the image is unlikely to cause disadvantage to the user. The automatically captured image that the user has not browsed even once is not recognized by the user, so even when the image is deleted, it can be inferred that the user will not feel disadvantaged. When the image meets the deletion condition, the process proceeds to S2906, the image is deleted, and the process proceeds to S2907. When the image does not meet the condition, the process proceeds to S2907 without deleting the image. In S2907, it is determined whether the target number of images to be deleted is reached. When the target number is reached, the process of the automatic deletion mode ends. When the target number is not reached, the process returns to S2904, and the process is repeated sequentially on the next image in the list. When there is no image to be processed in the list in S2904, the process ends.

[0305] <Handling of learning mode>

[0306] Next, learning adapted to user preference in the present embodiment will be described.

[0307] In this embodiment, the Fig.12 , and learning adapted to user preferences is performed in the learning processing unit 219 using a machine learning algorithm. The neural network is used to predict output values ​​based on input values. When the neural network has previously learned the historical values ​​of input values ​​and the historical values ​​of output values, the neural network can estimate output values ​​for new input values. By using the neural network, learning adapted to user preferences is performed for the above-mentioned automatic shooting, automatic editing, and subject search.

[0308] Subject registration (face recognition, general object recognition, etc.) as feature data to be input to the neural network is registered, and shooting notification control, low power consumption mode control, and automatic file deletion are changed through learning.

[0309] In this embodiment, elements to be learned through the learning process are as follows.

[0310] (1) Automatic shooting

[0311] The learning for automatic shooting will be described. In automatic shooting, learning for automatically shooting an image matching the user's preference is performed. Fig. 9As described in the process of , processing for generating learning information is performed after shooting (step S912). An image to be learned by a method (described later) is selected, and learning is performed by changing the weight of the neural network based on the learning information included in the image. Learning is performed by changing the neural network that determines the timing of automatic shooting and changing the neural network that determines the shooting method (still image shooting, video shooting, continuous shooting, panoramic shooting, etc.).

[0312] (2) Automatic editing

[0313] The following will describe learning for automatic editing. Fig. 9 Each edit immediately after the shooting in step 911 and Fig.11 The editing of the wonderful video described in the above is performed by learning. The editing immediately after shooting will be described. An image to be learned by a method (described later) is selected, and learning is performed by changing the weights of the neural network based on the learning information included in the image. Various detection information obtained from the information at the time of shooting or immediately before shooting is input to the neural network, and the editing method (cropping, rotation of the image, HDR (high dynamic range) effect, blur effect, color conversion filter effect, etc.) is determined. The editing of the wonderful video will be described. For the wonderful video, learning is performed for automatically creating a video album that matches the user's preferences. An image to be learned by a method (described later) is selected, and learning is performed by changing the weights of the neural network based on the learning information included in the image. Various detection information obtained from the information at the time of shooting or immediately before shooting is input to the neural network, and the application of image effects (cropping, rotation, HDR effect, blur effect, sliding, zooming, fading, color conversion filter effect, BGM, still image-moving image ratio) is determined.

[0314] (3) Subject Search

[0315] The learning for subject search will be described. For subject search, learning for automatically searching for subjects that match user preferences is performed. Fig. 9 As described in the process, in the subject search process (step S904), the importance level of each area is calculated, the camera is panned / tilted or zoomed, and the subject search is performed. Learning is performed based on the captured image and the information detected during the search, and the learning is performed by changing the weight of the neural network. The subject search reflecting the learning is performed by inputting various detection information during the search operation into the neural network, calculating the importance level, and setting the pan / tilt angle based on the importance level. In addition to setting the pan / tilt angle based on the importance level, for example, learning of pan / tilt (speed, acceleration, movement frequency) is also performed.

[0316] (4) Subject Registration

[0317] Learning for subject registration will be described. In subject registration, learning for automatically registering and ranking subjects that match user preferences is performed. For learning, for example, facial identification registration, registration of general object recognition, and registration of gesture, voice recognition, and sound-based scene recognition are performed. For identification registration, identifications of people and objects are registered, and rankings are set based on the number or frequency of image acquisition, the number or frequency of manually captured images, and the frequency of appearance of the searched subject. The registered information will be registered as an input for determination using a neural network.

[0318] (5) Shooting notification control

[0319] The learning for the shooting notification will be described. Fig. 9 As described in S910, when a predetermined condition is satisfied immediately before photographing, the image pickup device notifies the person to be photographed that an image of the person is to be photographed, and then photographs the person. For example, a movement that visually guides the subject's line of sight by panning / tilting, a speaker sound emitted from the voice output unit 218, and an LED flashing light generated by the LED control unit 224 are used. Whether to use the detection information of the subject in learning is determined based on whether the detection information (e.g., degree of smile, line of sight detection, gesture) is obtained immediately after the notification, and learning is performed by changing the weights of the neural network. Alternatively, an image to be learned by a method (described later) is selected, and learning is performed by changing the weights of the neural network based on the learning information included in the image. Information describing how to perform the notification movement immediately before photographing is embedded in the image, and the detection information added to the selected image and the information about the notification movement immediately before photographing are used as training data for learning. Information detected immediately before shooting is input to the neural network, and it is determined whether to provide notification or each notification operation (sound (sound level / sound type / timing), LED light (color, lighting time, flashing interval) and pan / tilt movement (mode of movement, driving speed). For learning of notification operation, a method of selecting which notification to provide from the prepared notification methods (combined operation of sound, LED light and pan / tilt movement) can be used. The following method can be used, in which separate neural networks are provided for notification operations of sound, LED light and pan / tilt movement, respectively, and these operations are learned.

[0320] (6) Low power consumption mode control

[0321] As reference Figure 7 and Figure 8 As described above, control for starting or stopping the power supply to the main processor (first control section 223) is performed, and a return condition from the low power consumption mode and a transition condition to enter the low power consumption state are learned.

[0322] First, learning of conditions for releasing the low power consumption mode will be described.

[0323] [Tap detection]

[0324] As described above, the predetermined time TimeA or the predetermined threshold ThreshA is changed by learning. Temporary tap detection is performed in a state where the above-mentioned threshold for tap detection is lowered, and the parameters of TimeA and ThreshA are set so that tapping is easily detected depending on whether the temporary tap detection is determined before the tap detection. When it is determined that it is not a starting factor based on the information detected after the tap detection (there is no target to be photographed as a result of the above-mentioned subject search or automatic shooting determination), the parameters of TimeA and ThreshA are set so that it is difficult to detect the tap. The determination of whether there is a target to be photographed at the time of startup varies depending on the detected information about the subject embedded in each image learned by the learning method (described later).

[0325] [Shaking state detection]

[0326] As described above, the predetermined time TimeB, the predetermined threshold ThreshB or the predetermined count CountB is changed by learning. When the start condition is satisfied based on the shaking state, the start is performed; however, when it is determined that it is not a start factor based on the detected information within a predetermined time period after the start (as a result of the above-mentioned subject search or automatic shooting determination, there is no target to be photographed), learning is performed to make the start difficult by changing the parameters determined by the shaking state. When it is determined that the frequency of shooting in a large shaking state is high, the parameters are set so that it becomes easy to start by determining the shaking state. The determination of whether there is a target to be photographed at the time of startup or the determination of whether the frequency of shooting in a large shaking state is high varies depending on the detected information about the subject embedded in each image learned by the learning method (described later), the shaking information during shooting, etc.

[0327] [Sound Detection]

[0328] Learning can be performed by manually setting a specific sound, a specific sound scene, or a specific sound level that the user wants to detect, for example, via communication with a dedicated application of the external device 301. In addition, a plurality of detected sounds are preset in the voice processing unit, an image to be learned by a method (described later) is selected, and learning is performed based on learning information included in the image (such as sound information before and after shooting). Therefore, a sound determination (a specific voice command or sound scene, such as "cheers" and "claps") to be used as a trigger can be set, and a trigger based on sound detection can be learned.

[0329] [Detection of environmental information]

[0330] Learning can be performed by manually setting the change conditions of the environmental information that the user wants to start the image pickup device through, for example, communication with a dedicated application of the external device 301. For example, the image pickup device can be started depending on the specific conditions of temperature, atmospheric pressure, brightness, humidity, absolute amount of ultraviolet light or change. The determination threshold based on environmental information can be learned. When it is determined that it is not a startup factor based on the information detected after startup (there is no target to be photographed as a result of the above-mentioned subject search or automatic shooting determination), the parameters of the determination threshold are set to make it difficult to detect the startup condition. Alternatively, by learning from the environmental information embedded in the image learned by the learning method (described later), the startup based on environmental information can be learned. For example, when many images taken when the temperature rises are learned, learning is performed to make it easier to drive the image pickup device when the temperature rises. The parameters also vary depending on the remaining battery power. For example, when the battery power is low, it is difficult to move to various determinations, and when the battery power is high, it is easy to move to various determinations. Specifically, for the shaking state detection result or the sound scene detection of the sound detection which is a condition rather than a cause for the user to definitely start the image pickup device, the difficulty of each detection determination varies depending on the battery power.

[0331] Next, learning of conditions for shifting to the power consumption state will be described.

[0332] like Figure 7As shown in , when the determination conditions of any one of "automatic shooting mode", "automatic editing mode", "automatic image transfer mode", "learning mode" and "automatic file deletion mode" are not satisfied in the mode setting determination 704, the mode moves to the low power consumption mode. The determination conditions of each mode are as described above, and the conditions for determining each mode also change through learning. For the automatic shooting mode, as described above, the importance level of each area is determined, and automatic shooting is performed while performing a subject search by panning / tilting. Since the importance level of each area is calculated based on the number or size of subjects (such as people and objects), the importance levels of all areas are low in the absence of surrounding subjects. Therefore, for example, the automatic shooting mode can be released based on a condition as to whether the importance levels of all areas or a value obtained by adding the importance levels of the areas is less than or equal to a predetermined threshold. Therefore, in the case where there are no subjects around and the necessity of shooting is low, the automatic shooting mode can be released and converted to a low power consumption state. At this time, the predetermined threshold can be lowered according to the time elapsed from when the mode moves to the automatic shooting mode. The longer the time that has passed since the mode was shifted to the automatic shooting mode, the easier it is for the mode to be shifted to the low power consumption mode. Moreover, by changing the predetermined threshold value depending on the remaining power of the battery, low power consumption mode control in consideration of battery exhaustion can be performed. For example, the threshold value decreases when the battery power is low, and the threshold value increases when the battery power is high. Here, based on the time that has passed since the mode was shifted to the automatic shooting mode last time and the number of images captured, the parameter (elapsed time threshold value TimeC) of the next low power consumption mode release condition is set in the sub-processor.

[0333] The above thresholds vary by learning. Learning can be performed by manually setting the shooting frequency, the startup frequency, etc. via, for example, communication with a dedicated application of the external device 301. Alternatively, the parameters can be learned by storing the average value or distribution data of each time period from the time when the power button of the image pickup device 101 is turned on to the time when the power button is turned off. In this case, learning is performed so that for users with a short time from power on to power off, the time interval for returning from a low power consumption mode or moving to a low power consumption state is shortened, while for users with a long time from power on to power off, the time interval is extended. Learning is also performed based on information detected during the search. Learning is performed so that when the number of important subjects determined to be set by learning is large, the time interval for returning from a low power consumption mode or moving to a low power consumption state is reduced, and when the number of important subjects is small, the time interval is extended.

[0334] (7) Automatic file deletion

[0335] Learning for automatic file deletion will be described. For automatic file deletion, the capacity of available files, the selection of images to be deleted preferentially, and the like are learned. An image to be learned by a method (described later) is selected, and learning can be performed by changing the weight of a neural network based on the learning information included in the image. As described above in automatic shooting, a score determined for user preference is calculated for each image, and images with lower scores are preferentially deleted from the recording medium 221. Learning is performed based not only on the score of the score, but also on the details of the shooting date and time embedded in each image in the recording medium 221 or the selected wonderful video (automatically edited moving image) edited using a method (described later). For example, learning is performed so that when the acquired wonderful video contains many images shot at short time intervals, files with older shooting dates and times are preferentially deleted, and when the acquired wonderful video contains images shot at long time intervals, even files with higher scores with older dates and times are not deleted. Alternatively, the score of each image in the recording medium 221 is sequentially recalculated at predetermined time intervals. At the time of score calculation, shooting date and time information is also input to the neural network, and learning is performed so that when many images are shot at short time intervals, files with older shooting date and time have lower scores. Therefore, such files are preferentially deleted, and learning is performed so that when images shot at long time intervals are included, even if the date and time are old, the score will not be reduced, with the result that learning is performed so that even files with higher scores with older date and time are not deleted. In another example, images to be learned are selected in a method (described later); however, when images with relatively new date and time are often selected intensively, files with older shooting date and time are preferentially deleted. However, learning is performed so that when selected images with relatively old date and time are often selected, even files with higher scores with older date and time are not deleted. In yet another example, when learning is performed so that the shooting frequency increases, files are automatically deleted so that the available area of ​​the files increases; and when learning is performed so that the shooting frequency decreases, files are automatically deleted so that the available area of ​​the files can be small. In yet another example, when learning is performed so that the shooting frequency of moving images increases, files are automatically deleted so that the available area of ​​the files increases; and when learning is performed so that the shooting frequency of still images increases, files are automatically deleted so that the available area of ​​the files decreases.

[0336] (8) Image stabilization

[0337] Learning for image stabilization will be described. Fig. 9Image stabilization is performed by calculating the stabilization amount in S902 and driving pan / tilt based on the stabilization amount in S905. In image stabilization, learning for making corrections adapted to the shaking characteristics of the user is performed. The direction and size of blur can be estimated by, for example, estimating the PSF (point spread function) of the captured image. Fig. 9 When generating learning information in S912 of , the estimated blur direction and size are added to the image as information. Figure 7In the process of the learning mode in step 716 of , the weight of the neural network for stabilization is learned by using the estimated direction and size of blur as output and the information detected during shooting as input. The information detected during shooting is the motion vector information of the image within a predetermined time before shooting, the information about the detected motion of the subject (person or object), the vibration information (gyro output, acceleration output, the condition of the image pickup device), etc. In addition to the above, it is possible to add environmental information (temperature, atmospheric pressure, illumination, humidity), sound information (sound scene determination, specific voice detection, sound level change), time information (time elapsed from startup, time elapsed from the last shooting), position information (GPS position information, change in position movement), etc. as input. By inputting the detected information to the neural network when calculating the stabilization amount in S902, the size of blur at the instant shooting can be estimated, and when the estimated size of blur is large, control such as reducing the shutter speed is possible. When the estimated size of blur is large, the image will be blurred, so a method such as prohibiting shooting can be adopted. Since the pan / tilt angle is limited, no further correction can be performed when the drive end is reached; however, as described above, by estimating the size and direction of blur during shooting, the range of pan / tilt required for stabilization during exposure can be estimated. When there is no margin in the movable range during exposure, large blur can be suppressed by increasing the set cutoff frequency of the filter for calculating the stabilization amount so that the range does not exceed the movable range. When the range may exceed the movable range, the pan / tilt angle is turned in the direction opposite to the direction in which the pan / tilt angle may exceed the movable range immediately before exposure, and then exposure is started. Therefore, blur-free shooting can be performed while ensuring the movable range. Therefore, stabilization can be learned in accordance with the characteristics and usage of the user during shooting, so blur-free images can be shot. In the above-mentioned "determination of shooting method", it is determined whether to perform panning shooting, and the subject blur correction can be performed by estimating the pan / tilt drive speed based on the information detected until before shooting. Here, panning shooting is shooting in which the moving subject is not blurred and the static background flows. In this case, by inputting the detected information into the neural network, the drive speed during still image shooting is estimated. For learning, the direction and size of the blur in the block where the main subject is located are estimated by dividing the image into blocks and estimating the PSF of each block, and learning can be performed based on this information. The background flow can be learned based on the background flow of the selected image by a learning method (described later). In this case, the size of the blur in the block where the main subject is not within the selected image is estimated, and the user preference can be learned based on this information. By setting the shutter speed during shooting based on the learned preferred background flow, shooting that can provide a panning effect adapted to the user preference can be automatically performed.

[0338] (9) Automatic image transmission

[0339] Learning for automatic image transmission will be described. For automatic image transmission, learning is performed on the process of selecting an image to be transmitted preferentially from the images recorded on the recording medium 221, the transmission frequency, etc. An image to be learned by a method (described later) is selected, and learning can be performed by changing the weight of a neural network based on the learning information included in the image. As described above, as described in automatic shooting, a score determined for user preference is calculated for each image, and images with higher scores are preferentially transmitted. Learning information associated with previously transmitted images is also used for image transmission determination. When selecting an image to be learned in a method (described later), which of the learning information (feature quantities) included in the image is set to be given importance, and when many images previously transmitted include similar feature quantities, images with higher scores and including other feature quantities are set to be transmitted. The image transmission frequency also varies according to the condition of the image pickup device. The image transmission frequency varies depending on the remaining battery power. For example, the image transmission frequency is set so that images are more difficult to transmit when the battery power is low, and images are more easily transmitted when the battery power is high. Specifically, for example, the image transmission frequency can also be changed by adopting the following configuration: the time elapsed since the last image was automatically transmitted is multiplied by the highest score among the images captured during the elapsed time, and the image is configured to be transmitted when the multiplied value exceeds a threshold value, and the threshold value varies depending on the battery power. In another example, the frequency of automatic image transmission is changed according to the shooting frequency set by the image pickup device 101. When learning is performed so that the shooting frequency increases, the frequency of automatic image transmission is also set to increase. When learning is performed so that the shooting frequency decreases, the frequency of automatic image transmission is also set to decrease. At this time, the image transmission frequency corresponding to the set shooting frequency can be changed by changing the threshold value according to the shooting frequency. In another example, the frequency of automatic image transmission is also changed according to the available capacity in the file (recording medium 221). When the available capacity in the file is large, the frequency of automatic image transmission is set to decrease. When the available capacity in the file is small, the frequency of automatic image transmission is set to increase. By changing the threshold value depending on the available capacity in the file at this time, the image transmission frequency corresponding to the available capacity in the file can be changed.

[0340] Next, the learning method will be described.

[0341] The learning methods include "learning in the image pickup device" and "learning in cooperation with the communication device".

[0342] A method of learning in the image pickup device will be described below.

[0343] (1) Learning using information detected when the user issues a shooting command

[0344] As in Fig. 9 As described in steps S907 to S913 of the embodiment, in the present embodiment, the image pickup device 101 is capable of performing two types of shooting, namely, manual shooting and automatic shooting. When a shooting instruction based on manual operation is issued in step S907 (performed based on the three determinations described above), information that the shot image is a manually shot image is added in step S912. When it is determined in step S909 that the automatic shooting is turned on and an image is shot, information that the shot image is an automatically shot image is added in step S912.

[0345] When manual shooting is performed here, there is a high possibility that shooting is performed based on a user-preferred subject, a user-preferred scene, and a user-preferred position and time interval. Therefore, learning is performed based on learning information of gesture data and captured images obtained during manual shooting.

[0346] Learning is also performed on the extraction of feature amounts in captured images, registration of personal identification, registration of facial expressions of each person, and registration of combinations of people based on information detected during manual shooting. Learning is also performed to change the degree of importance of nearby people and objects based on facial expressions registered for each subject individual, for example, by using information detected during subject search. When "the user manually turns to pan / tilt" (refer to later Figures 17 to 22 When an instruction to change the angle of view is issued, it is also learned that the subject exists within the angle of view after the rotation. This is also a part of learning based on the information of the detected manual operation.

[0347] (2) Learning using information detected during subject search

[0348] During the subject search operation, it is determined which persons, objects, and scenes the subject registered with the personal identification appears in the captured image simultaneously with, and the ratio of the time that the subject, object, or scene appears simultaneously therewith within the field of view is calculated.

[0349] For example, when the ratio of the time when person A of the subject registered with the personal ID and person B of the subject registered with the personal ID appear at the same time is higher than a predetermined threshold, the importance can be determined to be high. Therefore, when person A and person B are included in the angle of view, various detected information is saved as learning data so that the score for automatic shooting determination increases, and learning is performed in the process 716 of the learning mode.

[0350] In another example, when the ratio of the time when person A, a subject registered with a personal identification, and a subject "cat" determined by general object recognition appear at the same time is higher than a predetermined threshold, it can be determined that the importance is high. Therefore, when person A and "cat" are included in the field of view, various detected information is saved as learning data so that the score for automatic shooting determination increases. Then, learning is performed in the process 716 of the learning mode.

[0351] In this way, when the score for automatic shooting determination is increased in a case where the appearance frequency of the searched subject is high, the importance of people and objects near the subject registered with the personal identification can also be changed to increase.

[0352] When the degree of smile of person A who is a subject registered with a personal ID is detected, or when "joy", "surprise", etc. are detected by detecting facial expressions, a process of learning a subject that appears simultaneously and is therefore important is performed. Since the possibility that a subject that appears simultaneously when "anger", "stiff face", etc. are detected from facial expressions is low, a process such as not learning is performed.

[0353] Next, learning performed in cooperation with an external communication device in the present embodiment will be described.

[0354] In order to cooperate with the external communication device in this embodiment, there are the following methods.

[0355] (3) Learning by acquiring images using an external communication device

[0356] like Figure 3 As described in the foregoing, the image pickup device 101 and the external device 301 have communication means for communication 302, 303. The image is mainly sent or received via the communication 302. The image in the image pickup device 101 can be transferred and acquired to the external device 301 via a dedicated application in the external device 301. Thumbnail images of the image data stored in the image pickup device 101 can be browsed via a dedicated application in the external device 301. Therefore, the user can select his or her favorite image from the thumbnail images, check the image, and acquire the image to the external device 301 by operating the image acquisition instruction.

[0357] At this time, since the user selects an image, issues a transmission instruction, and obtains an image, there is a high possibility that the obtained image is an image preferred by the user. Therefore, it is determined that the obtained image is an image to be learned, and various learning of the user's preferences can be performed by performing learning based on the learning information of the obtained image.

[0358] An operation example will be described. Fig.16An example of browsing images in the image pickup device 101 via a dedicated application of the external device 301 as a smart device is shown in . Thumbnail images (1604 to 1609) of the image data stored in the image pickup device are displayed on the display unit 407, and the user can select an image that the user likes and obtain the image. At this time, a display method changing unit (1601, 1602, 1603) for changing the display method is provided. When unit 1601 is pressed, the display order is changed to a date and time priority display mode, and the images are displayed on the display unit 407 in the order of the shooting date and time of the images in the image pickup device 101 (for example, the images are displayed so that image 1604 has a new date and time, and image 1609 has an old date and time). When unit 1602 is pressed, the mode is changed to a recommended image priority display mode. Based on Fig. 9 The images are displayed on the display section 407 in descending order of the scores of the images in the image pickup device 101 (for example, the images are displayed so that the image 1604 has a higher score and the image 1609 has a lower score) based on the scores determined based on the user preference for each image calculated in step S912. When section 1603 is pressed, a subject person or object can be specified, and then a specific subject person or object is specified, and only the specific subject can be displayed.

[0359] The display method change sections 1601 to 1603 can be set to the on state at the same time. For example, when all are set to the on state, images are displayed so that only the specified subject is displayed, and images with newer shooting dates and times are given higher priority, and images with higher scores are given higher priority.

[0360] In this way, since the user preference is also learned from the captured images, only the images preferred by the user can be easily extracted from a large number of captured images with a simple inspection work.

[0361] (4) Learning by inputting a certain value to an image via an external communication device

[0362] As described above, the image pickup device 101 and the external device 301 have a communication means, and the images saved in the image pickup device 101 can be browsed via a dedicated application in the external device 301. Here, a configuration in which the user scores each image can be applied. The user is able to give a high score (e.g., 5 points) to an image that the user considers to be preferred, or a low score (e.g., 1 point) to an image that the user considers to be non-preferred. The image pickup device is configured to learn based on the user's operation. The score of each image is used for re-learning together with the learning information in the image pickup device. Learning is performed so that the output of the neural network is close to the score specified by the user when feature data of the specified image information is input.

[0363] In the present embodiment, a configuration is adopted in which the user inputs a determination value to each captured image via the communication device 301. Alternatively, a configuration may be adopted in which the determination value is directly input to each image by operating the image pickup device 101. In this case, for example, the image pickup device 101 is equipped with a touch panel display, and a mode is set to a mode for displaying captured images by the user pressing a GUI button displayed on the touch panel display screen display section. Then, similar learning can be performed by a method in which the user inputs a determination value to each captured image while checking the image.

[0364] (5) Learning by analyzing images stored in external communication devices

[0365] The external device 301 includes a storage section 404, and images other than the images photographed by the image pickup device 101 are also recorded in the storage section 404. At this time, the user can easily browse the images saved in the external device 301, and can easily upload the images to the shared server through the common line control section 406, so the possibility that many images preferred by the user are included is very high.

[0366] The external device 301 may be configured to be able to perform a learning process equivalent to the learning process of the learning processing section 219 in the image pickup device 101 having the control section 411 on the image stored in the storage section 404 via a dedicated application. In this case, the external device 301 may be configured to perform learning by transmitting the processed learning data to the image pickup device 101. Alternatively, the external device 301 may be configured to perform learning in the image pickup device 101 by sending an image or data that the external device 301 wants the image pickup device 101 to learn.

[0367] Alternatively, the external device 301 may be configured to perform learning by the user selecting an image to be learned from images stored in the storage section 404 via a dedicated application.

[0368] (6) Learning from information uploaded to the SNS server using an external communication device

[0369] A method for learning using information in a social networking service (SNS) which is a service or website that can build a social network focusing on connections between people will be described. There is a technology for inputting tags related to an image from a smart device when uploading an image to the SNS and sending the tags together with the image. There is also a technology for inputting likes and dislikes for images uploaded by other users, and it is also possible to determine whether the images uploaded by other users are preferred photos of the user who owns the external device 301.

[0370] By downloading a dedicated SNS application to the external device 301, the images uploaded by the user himself or herself and the information about the images as described above can be obtained. Alternatively, when the user inputs likes or dislikes to images uploaded by other users, the images and tag information preferred by the user can be obtained. Those images and tag information are analyzed and can be learned and set in the image pickup device 101.

[0371] As described above, the external device 301 may be configured to obtain an image uploaded by a user or an image determined to be favorite by a user, and to be able to perform a learning process equivalent to the learning process of the learning processing section 219 in the image pickup device 101 with the control section 411. Therefore, learning may be performed by transmitting the processed learning data to the image pickup device 101. Alternatively, the external device 301 may be configured to send an image that the external device 301 wishes the image pickup device 101 to learn, and cause the image pickup device 101 to perform learning.

[0372] Furthermore, learning is performed so that Figure 7 The automatic editing mode of processing 712 in the color conversion filter effect or Fig. 9 The color conversion filter effect of the editor S911 varies based on the image filter provided in the SNS using tag information.

[0373] Alternatively, information about a subject that is a user preference is estimated based on the subject information set in the tag information, and learning is performed by registering the subject as a subject to be detected and inputting it into the neural network. The subject information can be estimated to be, for example, information about an object subject (such as a dog and a cat), information about a scene (such as a beach), information about a facial expression (such as a smile), etc.

[0374] The currently popular image information in the world may be configured to be estimated from the statistical value of tag information (image filter information or subject information) on the SNS, and to be learned and set in the image pickup apparatus 101 .

[0375] (7) Learning by changing parameters using an external communication device

[0376] As described above, the image pickup device 101 and the external device 301 have communication means, and the learning parameters currently set in the image pickup device 101 can be transmitted to the external device 301 and saved in the storage section 404 of the external device 301. For example, it is conceivable to use the weight of the neural network, the selection of the subject to be input to the neural network, etc. as the learning parameters. Alternatively, the learning parameters set in the dedicated server can be acquired via the common line control section 406 by a dedicated application in the external device 301, and can be set as the learning parameters in the image pickup device 101. Therefore, the learning parameters can be returned by saving the parameters in the external device 301 at a certain point in time and setting the parameters in the image pickup device 101, or the learning parameters of another user can be acquired via the dedicated server and set to his or her own image pickup device 101.

[0377] Through the dedicated application of the external device 301, it is possible to register a voice command registered by the user, a logo registration or a gesture, or to register an important place. For this information, the shooting trigger ( Fig. 9 ) is processed as input data determined by automatic shooting.

[0378] The shooting frequency, startup interval, still image-moving image ratio, (one or more) preferred images, etc. can be set, or the startup interval described in <Low Power Consumption Mode Control>, the still image-moving image ratio described in <Automatic Editing>, etc. can be set.

[0379] (8) Learning from manually edited information about images using external communication devices

[0380] The dedicated application of the external device 301 may have a function that allows manual editing by the user's operation, and the details of the editing work may be fed back to the learning. For example, an image may be edited by applying an image effect, and a neural network for automatic editing may be trained to determine the application of the manually edited image effect in learning image information. It is conceivable that the image effect is, for example, cropping, rotating, sliding, zooming, fading, color conversion filter effect, time, still image-moving image ratio, or BGM.

[0381] Here, in the case where it is learned that the user himself or herself has performed shooting (1) or image selection ((3) to (8)), because the user intentionally performed the operation, the learning is likely to be reliable learning in combination with the user's preference. However, (2) learning using information detected during subject search is not learning based on intentional user operation, and therefore there is a possibility that the learning includes learning that the user does not expect. For example, during the search, another person, object, or scene that appears at the same time as the subject registered with the personal identification is learned from the image information; however, subjects that often appear at the same time are not necessarily user preferences. Therefore, learning in the case where the user intentionally shoots or selects an image ((1), (3) to (8)) is performed preferentially rather than the case where learning is not based on intentional user operation (2).

[0382] Learning data is various data (image information, vibration information, environmental information, sound information, location information, etc.) recorded as tag information during shooting or searching, and when incorporated into learning, various data are saved in a list form. The number of data in the group used for learning is a number set to a fixed value. The data group used for learning is divided into two areas, namely, an area of ​​learning data intentionally made by the user and an area of ​​learning data unintentionally made by the user. The ratio of the number of data in the area is set to a ratio that makes the number of areas of learning data intentionally made by the user larger. When a new instruction for combined learning is issued, the learning data is deleted from the learning data associated with each area, and new learning data is added. For example, when two pieces of learning data intentionally made by the user are to be added, the two data are deleted from the area of ​​learning data intentionally made by the user, the new two data are added, and learning is performed again.

[0383] With the above configuration, learning ((1), (3) to (8)) performed in the case where the user intentionally captures or selects an image is preferentially performed rather than the case (2) where the learning is not based on an intentional user operation.

[0384] Alternatively, among various data used for learning, the date and time when each piece of learning data is generated are managed, and a weighting coefficient La corresponding to the time elapsed from the date and time when the learning data is generated is calculated. The weighting coefficient La is updated so as to decrease as the elapsed time increases. In addition, a weighting coefficient Lb based on whether the learning data is learning data intentionally produced by the user or learning data unintentionally produced by the user is also managed in association with each learning data. The weighting coefficient Lb for learning data intentionally produced by the user is set to be greater than the weighting coefficient Lb for learning data unintentionally produced by the user. Among multiple pieces of learning data intentionally produced by the user, the weighting coefficient Lb can be changed depending on which of the learning (1), (3) to (8) is studied.

[0385] When new learning data is added, the learning data with the smallest value obtained by multiplying the weighting coefficients La and Lb in the current learning data group is preferentially deleted, and then the additional data is inserted, and machine learning is performed based on the updated learning data group.

[0386] With the above configuration, learning ((1), (3) to (8)) is preferentially performed in the case where the user intentionally captures or selects an image rather than the case (2) where the learning is not based on an intentional user operation.

[0387] For Figures 1 to Figure 6 In the case of a configuration in which a screen is not provided and it is difficult to set the priority on the image pickup device and the priority setting is performed on the menu on the external device, the time and effort of the user is required. However, when the priority mark is automatically set not based on the operation for setting the priority mark but based on the operation for shooting processing or the operation for editing processing, the time and effort of the user can be reduced. When the priority is automatically evaluated by using the captured image, for example, when evaluating images captured at predetermined time intervals, similar images may be captured, but those are not necessarily the main subjects, and thus the user's intention may not be reflected. In contrast, in the present embodiment, the priority mark is set based on the processing intentionally performed by the user, and thus it is likely that the user's intention is fully reflected.

[0388] Next, the learning process sequence will be described.

[0389] exist Figure 7 When the mode to be set is determined in step 704, it is determined whether to perform the learning process. When the learning process is performed, it is determined that the mode is the learning mode, and the process of the learning mode of step 716 is performed.

[0390] The determination conditions of the learning mode will be described. Whether to move to the learning mode is determined based on the time elapsed from the time when the last learning process was performed, the amount of information available for learning, whether there is a learning process instruction via the communication device, etc. Fig.14 4 shows a flowchart of the determination process regarding whether to shift to the learning mode in the mode setting determination process of step 704 .

[0391] When an instruction to start learning mode determination is issued in the mode setting determination process of step 704, Fig.14The processing starts. In step 1401, it is determined whether there is a registration instruction from the external device 301. The registration here is a determination as to whether there is the above-mentioned registration instruction for learning. For example, there is <Learning using image information obtained through a communication device> or <Learning by determining a value by inputting an image via an external communication device>. There is also <Learning by analyzing an image stored in a communication device> and the like. When there is a registration instruction from an external device in step 1401, the processing proceeds to step S1410, and the processing of step 716 is set to be performed by setting the learning mode determination to "true". When there is no registration instruction from an external device in step S1401, the processing proceeds to step 1402. In step 1402, it is determined whether there is a learning instruction from an external device. The learning instruction here is a determination as to whether there is an instruction to set learning parameters, such as <Learning by changing image pickup device parameters with a communication device>. When there is a learning instruction from an external device in step 1402, the process proceeds to step S1410, and the process of step 716 is set to be performed by setting the learning mode determination to "true", after which the learning mode determination process ends. When there is no learning instruction from an external device in step S1402, the process proceeds to step 1403.

[0392] In step 1403, it is determined whether a predetermined learning condition is met. A learning condition based on a predetermined time can be used. For example, learning is performed at 24:00 every day. Therefore, learning is performed periodically, so the novelty of the learned result can be kept constant. As another example, learning can be performed under the condition that an instruction for power off is issued by pressing the power button of the image pickup device 101. At this time, after the learning process is completed, the power is turned off. The learning process generally requires a long processing time. When the learning process is performed at a time when it is estimated that the user does not use the image pickup device 101 for shooting for a period of time such as during a power off, the learning process can be performed without interfering with the user's use. When the predetermined learning condition is met, the process proceeds to step S1410. When the condition is not met, the process proceeds to step 1404. In step 1404, it is determined whether there is a possibility of shooting. As described above, the learning process takes time, so it is recommended to avoid performing the learning process at a time when shooting is possible. Therefore, for example, based on the condition that no instruction for manual shooting is issued for the last set time period or longer, or based on the importance level of the area in the automatic shooting mode being lower than or equal to a predetermined level, it is determined that the possibility of performing shooting for a period of time is small. When it is determined that the shooting possibility is low, the processing proceeds to step 1405. Otherwise, the processing proceeds to step 1411, and the learning mode determination is set to "false". In step 1405, the time TimeN elapsed since the last learning process (recalculation of the weight of the neural network) is performed, and then the processing proceeds to step S1406. In step 1406, the number DN of new data to be learned (the number of images to be learned specified during the time TimeN elapsed since the last learning process was performed) is obtained, and the processing proceeds to step 1407. In step 1407, the threshold DT is calculated according to TimeN. For example, the threshold DTa for the case where TimeN is shorter than a predetermined value is set to be larger than the threshold DTb for the case where TimeN is larger than a predetermined value, and the threshold is set to decrease with time. Therefore, even when the amount of learning data is small, learning is performed again when the elapsed time is long, thereby making it easy for the image pickup apparatus to learn and change according to the use time.

[0393] When the threshold value DT is calculated in step 1407, the process proceeds to step 1408, and it is determined whether the number of data DN to be learned is greater than the threshold value DT. When DN is greater than the threshold value DT, the process proceeds to step 1409, and DN is set to zero. Then, the process proceeds to step 1410, and the learning mode determination is set to "true" to set so that the process of step 716 is executed, after which the learning mode determination process ends.

[0394] When DN is less than or equal to the threshold value DT in step 1408, the process proceeds to 1411. Since there is neither a registration instruction nor a learning instruction from the external device, and the amount of learning data is also less than or equal to the predetermined value, the learning mode determination is set to "false" for setting so that the process of step 716 will not be performed, after which the learning mode determination process ends.

[0395] Next, the processing in the learning mode processing (step 716) will be described. The detailed flow of the processing in the learning mode is described in Fig.15 Shown in.

[0396] When Figure 7 When it is determined in step 715 that the mode is the learning mode and the process proceeds to step 716, Fig.15 In step 1501, it is determined whether there is a registration instruction from the external device 301. When there is a registration instruction from the external device in step 1501, the process proceeds to step 1502. In step 1502, various types of registration processing are performed.

[0397] Various types of registration are registration of features to be input to the neural network, and include, for example, registration of face recognition, registration of general object recognition, registration of sound information, registration of location information, and the like.

[0398] When the registration process is completed, the process proceeds to step 1503 , and the elements to be input to the neural network are changed based on the information registered in step 1502 .

[0399] When the processing of step 1503 is completed, the processing proceeds to step 1507.

[0400] When there is no registration instruction from the external device 301 in step 1501, the processing proceeds to step 1504, and it is determined whether there is a learning instruction from the external device 301. When there is a learning instruction from the external device, the processing proceeds to step 1505, the learning parameters transmitted from the external device are set to each determination device (weights of the neural network, etc.), and the processing proceeds to step 1507.

[0401] When there is no learning instruction from the external device in step 1504, learning (recalculation of the weight of the neural network) is performed in step 1506. The condition for moving to the process of step 1506 is such a condition as referring to Fig.14As described, the number DN of data to be learned exceeds the threshold value, and re-learning of each determination device is allowed. Re-learning is performed by using methods such as the error back propagation method and the gradient descent method, recalculating the weight of the neural network, and changing the parameters of each determination device. When the learning parameters are set, the process proceeds to step 1507.

[0402] In step 1507, the scores are assigned to the images in the file again. In the present embodiment, scores are assigned to all captured images stored in the file (recording medium 221) based on the learned results, and automatic editing or automatic file deletion is performed according to the assigned scores. Therefore, when relearning from an external device or setting learning parameters is performed, the scores of the images that have been captured also need to be updated. Therefore, in step 1507, recalculation is performed for assigning new scores to the captured images stored in the file. When this processing is completed, the processing of the learning mode is completed.

[0403] In this embodiment, a method is described for suggesting a video image preferred by a user by extracting scenes that the user seems to like, learning the features of the scenes, and reflecting the learned features in operations such as automatic shooting and automatic editing; however, the present invention is not limited to this application. For example, the learned features can be intentionally used in an application to provide a video image different from the video image preferred by the user. An example of the implemented method is as follows.

[0404] (1) Using a neural network that has learned preferences

[0405] For learning, learning of user preferences is performed as described above. Then, in S908 of "Automatic Shooting", when the output value of the neural network is a value indicating a different value from those of the user preference as training data, automatic shooting is performed. For example, when an image that the user likes is set as a training image and learning is performed so that a higher value is output when the image has features similar to those of the training image, on the contrary, automatic shooting is performed under the condition that the output value is smaller than a predetermined value or more. Similarly, also in the subject search processing or the automatic editing processing, processing is performed in which the output value of the neural network is a value indicating a different value from those of the user preference as training data.

[0406] (2) Methods using a neural network that has learned situations different from the preference

[0407] In this method, at the time point of the learning process, learning is performed using a situation different from the situation preferred by the user as training data. For example, a manually shot image is a scene that the user likes to shoot, and a learning method using a manually shot image as training data is described above. In contrast, in the present embodiment, on the contrary, a manually shot image is not used as training data, and a scene that has not been manually shot for a predetermined time or longer is added as training data. Alternatively, when scenes having features similar to those of manually shot images are included in the training data, those scenes may be deleted from the training data. Alternatively, an image whose features are different from those of an image shot by an external communication device may be added to the training data, or an image whose features are similar to those of a shot image may be deleted from the training data. With this configuration, data different from the user preference data is collected in the training data, and, as a result of learning, the neural network is able to recognize a situation different from the user preference situation. In automatic shooting, by performing shooting according to the output value of the neural network, a scene different from the user preference scene can be shot. Moreover, in automatic editing, editing of an image different from the user preference image can be similarly suggested.

[0408] As described above, by intentionally suggesting a video image different from the video image preferred by the user, shooting is performed in a scene that the user is reluctant to shoot manually, thereby obtaining an effect of reducing missed photo opportunities. Moreover, by suggesting the effect of shooting or editing in a scene that the user himself or herself is not aware of, it can be expected that the user will discover something or expand the preferred effect.

[0409] By combining the above means, it is easy to adjust the degree of adaptation to user preferences, just like suggesting a situation slightly similar to the user's preference situation but partially different. The degree of adaptation to user preferences can be changed according to the mode setting, the status of various sensors, and the status of detected information.

[0410] In the present embodiment, the description is made based on a configuration in which learning is performed within the image pickup device 101; however, a configuration in which learning processing is provided on the external device 301, data required for learning is transmitted to the external device 301, and learning is performed only on the external device can also achieve the same learning effect. In this case, as described above in <Learning by changing parameters with a communication device>, a configuration in which learning is performed by setting parameters learned on the external device (such as weights of a neural network) to the image pickup device 101 via communication can be adopted.

[0411] A configuration may be adopted in which a learning process is provided in each of the image pickup device 101 and the external device 301. For example, a configuration may be adopted in which a process 716 of a learning mode is executed in the image pickup device 101 and the timing of learning is performed by incorporating learning parameters, and the learning information in the external device 301 is transmitted to the image pickup device 101.

[0412] In the above automatic shooting mode processing ( Fig. 9 ), it is determined whether there is a user (manual) shooting instruction, and when there is a shooting instruction, the process proceeds to S910. Here, the user's (manual) shooting instruction can be issued by manually rotating the orientation of the image pickup device (hereinafter referred to as the shooting direction). Fig.17 205 is a block diagram showing the configuration of the barrel rotation drive section 205 . Fig.17 Components 1701 to 1707 are related to the driving of the pan axis. Fig.17 Components 1708 to 1714 relate to the driving and control of the tilt axis. The basic configurations related to the driving of the pan axis and the driving of the tilt axis are the same, and therefore, only the configuration related to the driving of the pan axis will be described, and the description of the configuration related to the driving of the tilt axis will be omitted. Reference numeral 1701 indicates an image position-pan position conversion section for calculating a target position when driving the pan axis 1706 based on a difference between a target position and a current position of a subject on an image. Fig.18 1801 indicates a certain instantaneous image obtained by the image processing section 207 during the subject search of the image pickup device. Reference numeral 1802 indicates the current position (x1, y1) of the subject. Reference numeral 1803 indicates the target position (x0, y0) of the subject. When calculating the pan and tilt target position based on the difference between the target position 1803 and the current position 1802 of the subject on the image, the following formula is used.

[0413] kp(f)×(x1-x0) (1)

[0414] kt(f)×(y1-y0) (2)

[0415] kp(f) is a conversion coefficient for calculating a target pan position based on a difference between a target position and a current position of a subject on an image, and varies according to the focal length f of the image pickup device. kt(f) is a conversion coefficient for calculating a target tilt position based on a difference between a target position and a current position of a subject on an image, and varies according to the focal length f of the image pickup device.

[0416] Fig.17Reference numeral 1702 in the figure indicates a compensator. The compensator 1702 calculates a control output by performing a PID control calculation so as to eliminate the difference between the current pan position and the target pan position calculated by the image position-pan position conversion section 1701. Reference numeral 1703 indicates a shooting direction change operation detection section. The shooting direction change operation detection section 1703 detects a shooting direction change operation based on a difference (hereinafter referred to as a position deviation) between the target pan position and the current pan position, a control output, and a pan movement speed. When the shooting direction change operation detection section 1703 detects a change in the shooting direction, the shooting direction change operation detection section 1703 stops panning by stopping the control output. On the other hand, when the shooting direction change operation detection section 1703 does not detect a change in the shooting direction, the shooting direction change operation detection section 1703 drives and controls panning according to the control output calculated by the compensator 1702. Reference numeral 1704 indicates a driver for generating a drive signal corresponding to the control output calculated by the compensator 1702. Reference numeral 1705 is an ultrasonic motor (USM), which is an actuator for driving a pan axis 1706. Reference numeral 1707 denotes a moving speed detection section for calculating a pan moving speed from a temporal change in a pan position. The moving speed detection section 1707 calculates a pan moving speed from a change in a pan position at each control sampling. Fig.19 is a flowchart showing a flow of detecting a photographing direction changing operation performed by a user's operation and updating learning information by setting a photographing area after the photographing direction changing operation as an important area.

[0417] In S1901, it is determined whether there is a shooting direction change operation performed by the user on the image pickup device. When detecting the shooting direction change operation performed by the user, when the control output and the position deviation (described later) meet the predetermined conditions, the shooting direction change operation detection unit 1703 determines that there is a change in the shooting direction. When the shooting direction change operation is detected in S1901, the processing proceeds to S1902, and the position control operation is stopped. When subject tracking or searching is performed, the subject tracking or searching is interrupted, and then the position control operation is stopped. On the other hand, when the shooting direction change operation is not detected in S1901, the shooting direction change operation continues to be detected. After the position control is stopped in S1902, the processing proceeds to S1903, and it is determined whether the shooting direction change operation performed by the user is ended.

[0418] When it is determined that the shooting direction change operation is ended, the shooting direction change operation detection unit 1703 determines whether the shooting direction change operation is continued or ended based on the panning movement speed. When it is determined that the shooting direction change operation is ended, the processing proceeds to S1904, and the shooting area information after the shooting direction change operation is ended is stored. By comparing the angle of view determined according to the position of the image pickup device, the panning position, the tilt position and the focal length with each divided area, the area to be stored is the closest area. When it is determined in S1903 that the shooting direction change operation is being performed, the end of the shooting direction change operation continues to be detected. In S1905, the learning information is updated so that the area stored in S1904 is more important than the other divided areas. In S1906, subject tracking and position control are enabled, and then the processing proceeds to S1901, and the detection of the shooting direction change operation is resumed. At this time, in order to notify the user that the image pickup device 101 is tracking, a special image (image effect) different from the image of the above-mentioned face identification is displayed on or around the tracking target image. As an example of a user performing a shooting direction changing operation, an example will be described in which the user performs a shooting direction changing operation by rotating the lens barrel 102 by hand while shooting flowers with the image pickup device 101 so that the optical axis of the image pickup device 101 is directed toward a specific person outside the angle of view. Fig. 20 1 is a schematic diagram for illustrating an example in which the lens barrel 102 is turned toward a person 2003 by the user's hand while an image of a flower 2001 is captured by the image pickup device 101, and then learning information is updated so that a region where the person 2003 is located is set as an important region. Fig. 20 Reference numeral 2002 in FIG. 1 indicates the optical axis of the image pickup device 101 that is photographing the flower 2001. Reference numeral 2004 indicates the optical axis after the user changes the photographing direction by hand. Reference numeral 2005 indicates the rotation direction of the lens barrel 102 when the user changes the photographing direction. Fig. 22 An operation of changing the photographing direction toward the person 2003 by the user's operation when photographing the flower 2001 and then updating the learning information so that the area where the person 2003 is located is set as the important area is described. Fig.21A , Fig. 21B , Fig. 21C and Fig.21D It is a view showing instantaneous images captured in a period during which the shooting direction is changed toward the specific person 2003 by changing the shooting direction when shooting flowers and the learning information is updated. Fig. 22 It is a graph showing the temporal changes in the panning control output 2201, the position deviation 2202, and the moving speed 2203 during the period when the user changes the shooting direction toward a specific person 2003 when shooting flowers and the learning information is updated so that the area of ​​the changed angle of view is set as the important area. Fig. 22The ta, tb, tc, and td in the figure are respectively Fig.21A , Fig. 21B , Fig. 21C and Fig.21D The time of the image shown in . Fig. 22 ThC in ThDiff is a threshold value of the control output, which is used to determine that the user has manually rotated the lens barrel 102. ThDiff is a threshold value of the position deviation, which is used to determine that the user has manually rotated the lens barrel 102. When the state in which the control output is greater than or equal to ThC and the position deviation is greater than or equal to ThDiff continues for a predetermined time ( Fig. 22 When the shooting direction is changed by the user (t2-t1 in FIG. 1 ), the control output of the compensator 1702 is stopped. ThV is a threshold value of the moving speed of the pan axis, which is used to determine that the user has stopped operating the shooting direction. CMax is the maximum value of the control output of the compensator 1702. In position control, the pan axis is driven and controlled by changing the control output within the range of -CMax to CMax so that the subject is located at the target position on the image. Fig. 22 t1 in the figure indicates the time when the control output 2201 is greater than or equal to ThC and the position deviation is greater than or equal to ThDiff after the user starts the shooting direction operation. t2 indicates the time when the time period during which the control output 2201 is greater than or equal to ThC and the position deviation 2202 is greater than or equal to ThDiff has reached the shooting direction change determination time (t2-t1). t3 indicates the time when the pan axis movement speed becomes ThV or less for the first time after time t2. t4 indicates the time when the time elapsed after the movement speed becomes less than ThV at time t3 becomes the shooting direction change end determination time (t4-t3).

[0419] Fig.21A This is a time-lapse image taken at time ta when the flower 2001 was photographed. Fig.21A Reference numeral 2101 in the figure indicates a subject frame representing a subject as a tracking, searching or shooting target. Reference numeral 2102 indicates a target point, which is a target position on the image at the center of the subject frame 2101. The point where the two lines indicated by 2102 intersect each other is the target position of the subject on the image. During a normal shooting operation (not in a state of a shooting direction changing operation), position alignment is performed by driving and controlling the pan axis or the tilt axis so that the center of the subject frame 2101 coincides with the center of the target point 2102. Fig. 21B is Fig.21A An image captured when the user rotates the lens barrel 102 to the right relative to the fixing portion 103 at a timing of time tb in the state of FIG. Fig. 21BThe solid arrow in indicates the panning direction of the position control. The hollow arrow indicates the rotation direction of the lens barrel 102 by the user's shooting direction change operation. When the control output 2201 and the position deviation 2202 are seen at time tb, although the control output is the maximum value CMax, the position deviation 2202 tends to increase. Based on this, it is determined that the user is intentionally turning the pan axis. In the present embodiment, after a predetermined time (t2-t1) elapsed from the state of sensing the user turning the lens barrel 102 by hand to the control output of the compensator 1702 stopping, a determination is performed as to whether the shooting direction has changed. This is a measure to determine whether there is a shooting direction operation when the user accidentally touches the lens barrel although the user has not performed a direction change operation or under the influence of a load fluctuation of a rotating axis or a tilting axis driven for searching. In order to quickly perform a shooting direction change determination after the user starts a shooting direction change operation, the time for determination can be shortened or omitted.

[0420] Fig. 21C This is a diagram when a subject is brought into the field of view by rotating the pan axis to the vicinity of a new subject through a shooting direction change operation by the user in a state where the control output of the compensator 1702 stops at time tc. In this way, the user needs to continue the shooting direction change operation until the subject as the new shooting target enters the field of view. When the image whose shooting direction is changing cannot be directly checked as in the case of the image pickup device 101 described in the present embodiment, the user confirms that the subject as the shooting target has entered the field of view by operating while checking the changing image with a smart device. As another means for the user to know that the subject as the shooting target has entered the field of view, when a new subject enters the field of view during the shooting direction change, the user can be notified by lighting the LED by the LED control unit 224 or outputting a voice by the voice output unit 218.

[0421] Fig.21D It is an image during tracking or photographing a new subject after changing the shooting direction under the state of the control output of the timing start compensator 1702 at time t4. Time t4 is a time period after the panning movement speed 2203 becomes less than or equal to ThV at time t3, which is longer than or equal to the shooting direction change operation end determination time (t4-t3). When it is determined at time t4 that the user's shooting direction change operation ends, the shooting area at time point t4 is set as the user's preference area, and is set to have a higher importance than other areas, and then the learning information is updated. Moreover, the subject existing in this area can be subjected to one or more operations of tracking, photographing, and identification registration as an important subject. For example, as Fig.21DAs shown in , when person 2003 exists within the field of view at the timing when the end of the user's shooting direction change operation is detected, person 2003 is subjected to any one or more operations of tracking, shooting, and identification registration as an important subject. The learning information update process can be performed only when there is a learning instruction from the user and it is not automatically executed. The learning information can be updated only when there is a learning instruction from the user. For example, after the image pickup device notifies the user that the subject has entered the field of view, the learning information is updated only when the user inputs a specific voice command for a pre-registered learning instruction.

[0422] In this embodiment, an example of detecting the start and end of the user's shooting direction change operation on the image pickup device based on the control output of the compensator, the position deviation, and the moving speed of the drive shaft is described; however, as long as the user's shooting direction operation can be detected, the user's shooting direction operation can be detected by other methods. For example, it is possible to detect whether the user has made a change in the shooting direction based on the time change of the signal from the gyro sensor or the acceleration sensor of the device shake detection unit 209. Fig.23 The change in the output of the acceleration sensor of the device shake detection unit 209 when the shooting direction of the image pickup device is changed by the user's operation is shown. Reference numeral 2301 indicates the time change of acceleration. ThA1 is the threshold value of acceleration used when determining that the user has started the shooting direction change operation. ThA2 is the threshold value of acceleration used when determining that the user has stopped the shooting direction change operation. The start and stop of the shooting direction change operation can be detected by comparing the acceleration with these thresholds. At this time, in order to prevent the shooting direction change operation from being erroneously detected, the time change pattern of acceleration during the shooting direction change operation can be learned in advance, and when the similarity between the detected time change of acceleration and the learned time change pattern is greater than or equal to a predetermined value, it can be determined that the shooting direction has changed. Similarly, the presence or absence of a shooting direction operation can be detected based on the change in the motion vector of the image captured by the image pickup device.

[0423] In the above description, a process is described in which a shooting area that enters the viewing angle after a shooting direction change operation is learned as an important area. However, without limitation to this, a process may be adopted in which, when there is a zoom change or a change in the shooting area by an operation of a user on an external device, the shooting area after the change operation is learned as an important area.

[0424] <Change processing according to low power consumption mode release condition>

[0425] refer to Fig. 9The basic processing sequence of the shooting mode in this embodiment is described; however, if processing is performed according to this sequence at any time, it takes time to find the subject and perform automatic shooting. In this case, there is a concern that the camera opportunity is missed or a subject different from the user's intention is photographed. Specifically, when the low power consumption mode is released (hereinafter referred to as wake-up), the optimal processing sequence changes depending on the condition based on which the release is performed. Here, an example of a wake-up condition and a processing sequence suitable for the condition will be described.

[0426] (1) Wake-up based on tap detection

[0427] As described above, wake-up based on tap detection is possible. In this case, it is assumed that the owner of the image pickup device 101 has issued a wake-up instruction intended to perform shooting. Therefore, such a process is preferred: in which the owner is found as a result of searching around, and automatic shooting is immediately performed to allow the owner to be photographed.

[0428] Fig. 27 The processing of the shooting mode in this case is shown.

[0429] Steps S2701 to S2703 and Fig. 9 The processing during normal time described in is the same, so its description is omitted.

[0430] In step S2704, unlike normal processing, the search is performed while panning / tilting the camera so that all angles of view are covered.

[0431] In step S2705, it is determined whether the specific recognized person is in the viewing angle. At this time, it is desired that the owner's face is pre-registered as the owner's recognized face, and the owner is searched as the specific recognized person. When the owner is found within the viewing angle, the process proceeds to S2706.

[0432] In step S2706, the camera is panned / tilted or zoomed so that the owner is included in the angle of view, and the process proceeds to a shooting start operation of step S2712.

[0433] Steps S2707 to S2715 are related to Fig. 9 S905 to S913 are similar processes, so their description is omitted.

[0434] Through such processing, it is possible to shoot immediately in response to the user's intention.

[0435] (2) Wake-up based on sound detection

[0436] As described above, wake-up based on sound detection and voice command recognition is possible. In the case of sound detection, there is a high probability that a person of interest exists in the direction of the sound. In the case of voice command recognition, it is assumed that the person who uttered the voice command has the intention of having the camera take a picture of himself. Therefore, a process is preferred in which a person in the direction of the detected voice is found and automatic shooting is immediately performed.

[0437] Fig.28 The processing of the shooting mode in this case is shown.

[0438] Steps S2801 to S2803 and Fig. 9 The processing during normal time described in is the same, so its description is omitted.

[0439] In step S2804, unlike normal processing, the camera is panned / tilted so that the direction in which the sound direction is detected is included in the angle of view.

[0440] In step S2805, it is determined whether there is a person in the viewing angle in the direction of the sound. When there is a person, the person is regarded as the source of the sound or voice command, and the process proceeds to S2806 for photographing the person.

[0441] In step S2806, the camera is panned / tilted or zoomed so that the person is included in the angle of view, and the process proceeds to a shooting start operation of step S2812.

[0442] Steps S2807 to S2815 are similar to Fig. 9 The processing of S905 to S913 is omitted, and thus its description is omitted.

[0443] With such processing, an effect can be expected that shooting can be performed without missing an opportunity to shoot a highly interesting moment such as cheering. Also, shooting that immediately responds to the intention of the person who spoke the voice command is possible.

[0444] (3) Awakening under other conditions

[0445] When waking up based on other conditions (e.g. Figure 8 ), according to the elapsed time described in Fig. 9 With this configuration, automatic shooting is performed only when an important subject is required, thereby reducing power consumption and consumption of available capacity of the storage device.

[0446] According to the above-described embodiment, the following functions are possible.

[0447] (1) Start

[0448] The search and shooting process after activation changes according to the activation conditions.

[0449] In this way, depending on what starts the image pickup device, the processing after startup (automatic shooting determination, search processing, or sleep determination processing) is changed. Therefore, this can solve the following problems: when it takes time to execute a unified startup sequence every time, a shooting opportunity is missed or a subject different from the user's intention is shot.

[0450] [Example 1] When awakened by voice, the image pickup device faces the direction of the voice and starts searching and photographing determination.

[0451] [Example 2] When awakened by tapping, the image pickup device searches for the owner (recognized face).

[0452] (2) Sleep

[0453] A subject scene determination means is provided, and the image pickup device decides to enter automatic sleep according to the scene determination result. The sleep time is adjusted according to the determination result. A means for determining the internal condition of the image pickup device is provided, and the image pickup device enters automatic sleep according to the internal condition determination means.

[0454] In this way, the image pickup device enters automatic sleep according to the subject or scene. The sleep time is also adjusted. The image pickup device enters automatic sleep according to the internal conditions of the processing on the image pickup device. Therefore, the problem that the power saving effect is weak and there is a fear of missing a photo opportunity in the case of sleep based only on the elapsed time or no operation can be solved.

[0455] [Example 1] When there is no subject, the image pickup device moves to power saving.

[0456] [Example 2] When there is a lack of change in the scene, the image pickup device is put to sleep for a long time.

[0457] [Example 3] When the mode does not correspond to any of the automatic shooting mode, the learning mode, the editing mode, and the transfer mode, the image pickup device enters the sleep state.

[0458] [Example 4] Battery level

[0459] (3) Automatic image transmission

[0460] The images are automatically transferred or the image transfer frequency is automatically determined based on at least one of the following conditions, ie, the elapsed time, the evaluation value of the captured image, the battery level, and the card capacity.

[0461] In this way, images are automatically transmitted according to conditions (at each lapse of a predetermined time when a high-value image is captured). The image transmission frequency is automatically determined according to conditions (when the battery power is low, image transmission becomes difficult; when the shooting frequency is set to a higher frequency, the transmission frequency also increases; when the available capacity of the storage medium is small, the transmission frequency increases). Therefore, it is possible to solve the problem that there is a waiting time for the transmission process when image transmission is performed according to the user's instruction or the capacity of the external device becomes tight for the specified number of image transmissions or transmission frequency depending on the user.

[0462] (4) Learning

[0463] The image pickup device automatically enters the learning mode according to at least one of the following conditions, namely, elapsed time, accumulation degree of training data, determination result of the current scene or object, scheduled time, possibility of future shooting, and time of power off.

[0464] In this way, the image pickup device automatically enters the learning mode according to the condition (when automatic shooting is not performed for a certain time, such as when new training data has been accumulated to be more than or equal to a predetermined amount, when the time elapsed since the last learning is long, or when there is no distinguishable subject around). Therefore, the following problem can be solved: unless the condition for the shooting mode to enter the learning mode is appropriately set, there will be a problem of waiting time for learning processing or wasting power.

[0465] (5) Automatic deletion of images

[0466] Perform automatic deletion based on conditions. Set the target number of images to be deleted based on shooting frequency and available capacity. Make <images manually shot by the user>, <images highly rated by the user>, and <images with high importance scores calculated by the image pickup device> difficult to delete. Make <images transferred to an external device> and <images that the user has not seen even once> easy to delete. When shooting a wonderful video at short intervals, old files can be deleted first. When shooting a wonderful video at long intervals, old but high-scoring files can be set not to be deleted. When performing learning to increase the frequency of video shooting, more images than usual can be automatically deleted.

[0467] Therefore, when there is no free capacity and it is troublesome for the user to manually delete one by one, the problem that automatic shooting cannot be performed can be solved.

[0468] (6) Automatic editing

[0469] The editing process is automatically performed based on at least one of the conditions, ie, the degree of accumulation of captured images, the time elapsed from the last editing, the evaluation value of each captured image, and the time situation.

[0470] Therefore, it is possible to solve the problem that, when a story photo moving image is created according to an instruction of a user, there is a time to wait for the creation process and the usability is poor.

[0471] In the above <Attachment Configuration>, refer to Fig.32 An accessory for connecting to another camera 3201 is described. An example of a case where the image pickup apparatus 101 and another camera 3201 perform photographing in cooperation with each other will be described.

[0472] A method of simultaneously photographing by releasing one camera between cameras to match the release timing of another camera is known as a related art.

[0473] In the present embodiment, photographing is performed in cooperation between cameras; however, before the release button 3203 of the camera 3201 is pressed, the image pickup device 101 starts photographing before the camera 3201 performs photographing by predicting the pressing of the release.

[0474] The image pickup device 101 performs automatic photography in a method similar to the method of automatic photography described. At this time, learning for predicting the timing at which the camera 3201 performs photography is performed. When cooperative photography is performed, automatic photography determination is performed using this network.

[0475] The operation of the image pickup apparatus 101 in a configuration in which a user operates the camera 3201 to perform photographing will be described. Fig.33 A flowchart of the image pickup apparatus 101 is shown.

[0476] Here, an example in which the camera 3201 captures a still image and the image pickup device 101 captures a moving image will be described.

[0477] When the shooting mode process starts, first it is determined in S3301 whether the mode is the camera cooperation mode. When the mode is the cooperation mode, the process proceeds to S3303; and when the mode is not the cooperation mode, the process proceeds to S3302.

[0478] The cooperation mode may be determined by whether the camera 3201 and the image pickup apparatus 101 are connected by wire or wirelessly, or may be set with the smart device 301 .

[0479] In S3202, the mode is not the camera cooperation mode, so the reference Fig. 9 The described processing, the shooting mode processing ends, and the processing waits for the next calculation cycle. In S3303, information is loaded from the camera 3201. Information related to the pressing of the release switch of the camera 3201, power-on state information, information related to the subject from the image, etc. is provided as information to the image pickup device 101, and the processing proceeds to S3304.

[0480] In S3304, it is determined whether the image pickup device 101 is performing shooting. When the image pickup device 101 is not performing shooting, the processing proceeds to S3305; and when the image pickup device 101 is performing shooting, the processing proceeds to S3306. In S3305, it is determined whether the camera 3201 has started shooting. When the camera 3201 has started shooting, the processing proceeds to S3310, starts shooting of the image pickup device 101, ends the shooting mode processing, and the processing waits for the next calculation cycle. When the camera 3201 has not started shooting in S3305, the processing proceeds to S3307 and performs automatic shooting determination processing. Can be used with reference Fig.12 The automatic shooting determination processing is realized by a method similar to the described method. At this time, the determination may be performed by using information from the camera 3201 and information from the image pickup device 101 as feature amount inputs, or may be performed by using only information from either one of them.

[0481] When the automatic shooting determination processing is completed, the processing proceeds to S3308, and it is determined whether to start shooting in the automatic shooting determination processing. When it is determined to start automatic shooting, the processing proceeds to S3309, and the image pickup device 101 starts automatic shooting. When the start of automatic shooting is not determined, shooting is not performed, the shooting mode processing ends, and the processing waits for the next calculation cycle.

[0482] When it is determined in S3304 that shooting is being performed after shooting is started in S3310 or S3309, shooting end determination processing is performed in S3306. Then, when it is determined that shooting is ended, the process proceeds to S3311, and the image pickup device 1010 ends shooting. When shooting is not determined to be ended, the shooting mode process is ended while shooting is performed as it is, and the process waits for the next calculation cycle.

[0483] Can be used for reference Fig.12 The automatic shooting determination processing is realized by a method similar to the described method. At this time, the determination may be performed by using information from the camera 3201 and information from the image pickup device 101 as feature amount inputs, or may be performed by using only information from either one of them.

[0484] In this configuration, the image pickup device 101 is configured to perform automatic shooting. Alternatively, the image pickup device 101 may continuously shoot moving images, attach tags to important time frames, and record the tags in a final moving image file.

[0485] Collaborative automatic shooting timing can be learned by using shooting results.

[0486] For example, when the image pickup device 101 does not perform automatic shooting or when the camera 3201 starts shooting, the learning data at this time is Fig.12 The input feature quantity is saved as incorrect data.

[0487] When the image pickup device 101 is performing automatic shooting or when the camera 3201 starts shooting, the learning data at this time Fig.12 The input feature quantity is saved as correct data.

[0488] When the image pickup device 101 is performing automatic shooting or when the camera 3201 does not start shooting after a predetermined time has passed, the learning data at this time Fig.12 The input feature quantity is stored as incorrect data.

[0489] When the learning data accumulation is greater than or equal to a predetermined amount, the learning data, and Fig.12 The weights of the neural network change.

[0490] An example is described in which the camera 3201 captures a still image and the image pickup apparatus 101 captures a moving image; however, the capturing method is not limited thereto. The following modes can be manually selected using the smart device 301 or the like.

[0491] The image pickup device 101 can automatically select the following modes. When automatically selected, it is also automatically determined in which mode to perform shooting.

[0492] For example, while the camera 3201 captures a still image, the image pickup device 101 captures a moving image.

[0493] Alternatively, for example, when the camera 3201 captures a still image, the image pickup device 101 captures a still image.

[0494] Alternatively, while the camera 3201 captures a moving image, the image pickup device 101 captures a still image.

[0495] Alternatively, when the camera 3201 captures a moving image, the image pickup device 101 captures the moving image.

[0496] The orientation and viewing angle of the optical axis direction of the camera 3201 and the image pickup device 101 may be manually selected or automatically selected.

[0497] For example, the optical axis directions of the camera 3201 and the image pickup device 101 are oriented in the same direction.

[0498] Alternatively, for example, the optical axis directions of the camera 3201 and the image pickup device 101 are oriented in different directions.

[0499] The viewing angles of the camera 3201 and the image pickup device 101 are the same.

[0500] Alternatively, the viewing angles of the camera 3201 and the image pickup device 101 are different.

[0501] Even when the image pickup device 101 captures a still image, the image pickup device 101 can predict the timing before starting the capture, and can automatically capture not only a single image but also several images during the automatic capture period.

[0502] In the present embodiment, an example in which the image pickup device 101 is connected to the accessory shoe 3202 of the camera 3201 and used is described; however, the configuration is not limited thereto. For example, the image pickup device 101 may be mounted on another member of the camera 3201 (e.g., a tripod thread, etc.), or may be used without being directly mounted on the camera 3201 (e.g., worn on the user in a wearable manner and providing information through wireless communication).

[0503] In the present embodiment, an example is described in which the image pickup device 101 captures an image by predicting in advance that the camera 3201 captures the image. Alternatively, the camera 3201 itself may predict the capture in advance. In this case, when it is determined in advance by prediction that the capture is to be performed, the camera 3201 may output an instruction to start the capture to the image pickup device 101, so that the camera cooperative capture with the prediction in advance may be performed.

[0504] The notification of information between the camera 3201 and the image pickup device 101 may be configured to be provided only at the release timing. The detected information of both the camera 3201 and the image pickup device 101 may be used to determine the start of shooting. Alternatively, only the detected information of the image pickup device 101 may be used to determine the start of shooting.

[0505] <Learning with Camera 3201>

[0506] (1) Transmitting information from the camera 3201 to the image pickup device 101

[0507] For example, the main subject is extracted from the image captured by the camera 3201 through the user's operation.

[0508] Then, the subject information is provided and set to the image pickup device 101. Thereafter, the image pickup device 101 determines whether the subject is important based on the number of captured images of the subject, registers the subject, and performs automatic capturing, tracking, and the like.

[0509] (2) Subject registration is performed using information acquired in the image pickup device 101 at the time of release start.

[0510] For example, the timing at which the camera 3201 performs photographing is provided to the image pickup device 101 through the user's operation. Then, an important subject is set from the image in the image pickup device 101 at the photographing timing. Thereafter, the image pickup device 101 determines whether the subject is important based on the number of photographed images of the subject, registers the subject, and performs automatic photographing, tracking, and the like.

[0511] <Notification of Information from Image Pickup Device 101 to Camera 3201>

[0512] An example in which, in the case where photographing is performed in cooperation between the image pickup device 101 and the camera 3201 , another camera 3201 is assisted with information from the image pickup device 101 will be described.

[0513] (1) Notification of subject information

[0514] The subject information detected by the image pickup device 101 (for example, a face registered by an individual, a subject such as a dog or a cat, an aesthetic determination result determined as the owner's preference, or a subject preferred by the user) is provided to the camera 3201. Then, the position where the subject is located in the real-time image of the camera 3201 and what subject exists outside the image (for example, there is a vehicle on the right side of the screen) is provided, and whether there is a subject preferred by the user is provided.

[0515] (2) Release Notice

[0516] The image pickup device 101 may be configured to provide a shooting instruction to the camera 3201 .

[0517] Using the method described in the processing of the automatic shooting mode, the shooting timing is determined, and the automatic shooting instruction is provided to the camera 3201.

[0518] It may be determined whether a specific object enters the screen of the camera 3201, and continuous shooting or video shooting may be performed at the timing when the specific object enters the screen.

[0519] According to the present embodiment, it is possible to provide an image pickup apparatus capable of taking a video image preferred by a user without any special operation by the user.

[0520] The embodiments of the present invention are not limited to the above-described embodiments. Various changes or modifications may be made without departing from the spirit and scope of the present invention. Therefore, the appended claims show the scope of the present invention.

[0521] This application claims the benefit of Japanese Patent Application No. 2017-188938 filed on September 28, 2017, Japanese Patent Application No. 2017-254231 filed on December 28, 2017, and Japanese Patent Application No. 2018-053078 filed on March 20, 2018, which are all incorporated herein by reference in their entirety.

Claims

1. An image pickup device, the image pickup device being operable in an automatic shooting mode, the image pickup device comprising: an image pickup unit configured to output image data; a subject detection unit configured to detect a specific subject from the image data output by the image pickup unit; a control unit configured to control the orientation of the image pickup unit in an automatic shooting mode so that the image pickup unit is directed toward the detected specific subject and to perform a shooting process for recording image data output from the image pickup unit based on information about the detected specific subject; as well as a state acquisition unit configured to acquire information about a state in which the image pickup device is held; wherein, based on the information on the state in which the image pickup device is held acquired by the state acquisition unit, the control unit changes the change range of the orientation of the image pickup unit, and The state in which the image pickup device is held includes at least one of a vehicle moving state, a stationary capturing state, a hand-held state, a state hanging from the neck, and a wearable state.

2. An image pickup device, comprising: an image pickup unit configured to output image data; a subject detection unit configured to detect a specific subject from the image data output by the image pickup unit; as well as a control unit configured to control the orientation of the image pickup unit in an automatic shooting mode so that the image pickup unit is directed toward the detected specific subject and to perform a shooting process for recording image data output from the image pickup unit based on information about the detected specific subject; Wherein, the control unit changes the change range of the orientation of the image pickup unit according to whether the image pickup device is in a stationary capturing state or a handheld state.

3. The image pickup device according to claim 1 or 2, wherein: The control unit performs control so that the image pickup unit rotates in at least one of a pan direction and a tilt direction.

4. The image pickup device according to claim 1 or 2, wherein: The image data output by the image pickup unit is input to a neural network to perform learning of the change range.

5. The image pickup device according to claim 1, wherein: The information about the state in which the image pickup device is held is accessory information about an accessory connected to the image pickup device.

6. The image pickup device according to claim 1 or 2, further comprising: The subject detection unit is configured to detect a specific subject from the image data output by the image pickup unit.

7. The image pickup device according to claim 6, wherein: Detection information of a specific subject detected by the subject detection unit is input to a neural network to perform learning of the change range.

8. The image pickup device according to claim 7, further comprising: The automatic capturing unit is configured to perform a capturing process of recording the image data output by the image pickup unit based on the detection information of the specific subject detected by the subject detecting unit.

9. The image pickup device according to claim 8, in, The specific subject is a human face, and The automatic capturing unit performs the capturing process according to at least one of the frequency of appearance of the specific subject and the facial expression of the person.

10. The image pickup device according to claim 9, in, The specific subject is an object, and Wherein, the automatic capturing unit performs the capturing process according to object recognition.

11. The image pickup device according to claim 9, wherein: The automatic capturing unit performs the capturing process by using parameters generated based on machine learning.

12. A method for controlling an image pickup device, the image pickup device being operable in an automatic shooting mode, the image pickup device comprising an image pickup unit outputting image data, the method comprising: a detection step of detecting a specific subject from the image data output by the image pickup unit; a control step of, in an automatic shooting mode, controlling the orientation of the image pickup unit so that the image pickup unit is directed toward the detected specific subject and performing a shooting process for recording image data output from the image pickup unit based on information about the detected specific subject; as well as a state acquisition step of acquiring information about a state in which the image pickup device is held; wherein, in the control step, based on the information on the state in which the image pickup device is held acquired in the state acquisition step, the range of change of the orientation of the image pickup unit is changed, and The state in which the image pickup device is held includes at least one of a vehicle moving state, a stationary capturing state, a hand-held state, a state hanging from the neck, and a wearable state.

13. A method for controlling an image pickup device including an image pickup unit that outputs image data, the method comprising: a detection step of detecting a specific subject from the image data output by the image pickup unit; as well as a control step of, in an automatic shooting mode, controlling the orientation of the image pickup unit so that the image pickup unit is directed toward the detected specific subject and performing a shooting process for recording image data output from the image pickup unit based on information about the detected specific subject; Wherein, in the control step, the changing range of the orientation of the image pickup unit is changed according to whether the image pickup device is in a stationary capturing state or a handheld state.

Images