Control device, imaging apparatus, control method, program, and imaging system

Abstract

To achieve automatic photographing for easily recording an image in accordance with a user's demand.SOLUTION: A control device is for controlling the operation of an imaging apparatus that acquires an image provided to a user by means of automatic photographing by changing a composition. The control device has first acquisition means for acquiring demand information indicating a demand tendency of the user with respect to the composition of a subject, setting means for setting the frequency of adopting for photographing each of a plurality of predetermined compositions on the basis of the demand information acquired by the first acquiring means, and determination means for determining the composition in which the subject is to be photographed by the imaging apparatus on the basis of the frequency of adopting each of the plurality of compositions set by the setting means.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a control device, an imaging device, a control method, a program, and an imaging system, and more particularly to a system that detects a subject and automatically captures an image. [Background technology]

[0002] In surveillance systems and the like, imaging devices are employed that detect a predetermined subject, and record images by capturing images (hereinafter sometimes referred to as automatic capture) without the user having to input a capture instruction. In recent years, the development of services that use such imaging devices to enable users to obtain images captured automatically at events such as athletic meets and recitals has also been considered.

[0003] Since automatic photography does not require the user to input a photography instruction, there is a possibility that an image that the user prefers may not be recorded. For this reason, the imaging device of Patent Document 1 performs automatic photography on the condition that a smiling face of the subject is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-124793 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in an image obtained by automatic shooting using the imaging device of Patent Document 1, while the facial expression of the subject reflects the user's preferences, there is a possibility that the user's preferences may not be reflected in the composition of the entire image.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a control device, an imaging device, a control method, a program, and an imaging system that realize automatic shooting that makes it easy to record images that meet the user's needs. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the control device of the present invention is a control device that controls the operation of an imaging device that acquires images to be provided to a user by automatic shooting with different compositions, and is characterized by having a first acquisition means that acquires demand information indicating user demand trends regarding the composition of a subject, a setting means that sets the frequency of adoption in shooting for each of a plurality of predetermined types of compositions based on the demand information acquired by the first acquisition means, and a determination means that determines a composition for the imaging device to photograph a subject based on the frequency of adoption of each of the plurality of types of compositions set by the setting means. Effect of the Invention

[0008] With this configuration, the present invention makes it possible to realize automatic photography that makes it easy to record images that meet the needs of the user. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a configuration of an imaging system according to an embodiment and a modification of the present invention. [Diagram 2] FIG. 1 is a diagram for explaining an imaging device 100 according to an embodiment and a modification of the present invention; [Diagram 3] FIG. 1 is a block diagram illustrating a functional configuration of an image capture device 100 according to an embodiment and a modification of the present invention. [Figure 4] 1 is a flowchart illustrating a process related to a search operation of an automatic shooting sequence according to the first embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram for explaining a search operation according to an embodiment and a modification of the present invention; [Figure 6] FIG. 11 is a diagram for explaining a change in the adoption probability of each composition based on demand information according to the first embodiment of the present invention; [Figure 7] 1 is a flowchart illustrating a process related to shooting determination in an automatic shooting sequence according to a first embodiment of the present invention. [Figure 8]FIG. 11 is a diagram for explaining a change in the adoption probability of each composition based on demand information according to the second embodiment of the present invention; [Figure 9] 11 is a flowchart illustrating a process related to a search operation of an automatic shooting sequence according to a second embodiment of the present invention. [Figure 10] 11 is a flowchart illustrating a process related to shooting determination in an automatic shooting sequence according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In the embodiment described below, an example of the present invention is applied to an image capture device capable of capturing an image to be provided to a user while changing the composition (automatic capture), as an example of a control device. However, the present invention can be applied to any device capable of controlling the composition of an image capture device capable of automatic capture according to the demand tendency of a user.

[0012] In addition, in this specification, "automatic photography" does not refer to photography triggered by the receipt of an operation input related to a photography instruction from a user, but rather photography triggered by a specific device recognizing a scene and determining that it is a situation in which photography should be performed. Here, the "scene" refers to the environment captured by the imaging device, and in a narrow sense, may refer to a space that can be captured in the angle of view of the imaging device, including panning and tilting operations. The scene can be recognized using a captured image or an output from an external sensor, and automatic photography is performed, for example, when the recognition result satisfies a predetermined photography condition.

[0013] Configuration of the imaging system FIG. 1 is a diagram showing a schematic configuration of an imaging system according to an embodiment of the present invention.

[0014] The imaging system of this embodiment provides a service of selling to users images (hereinafter referred to as recorded images) obtained by automatically capturing images of subjects in a given scene using an imaging device 100 arranged in the scene. As will be described in detail later, the imaging device 100 is configured to be able to autonomously perform image capture related to image recording without relying on operation input by a user. The recorded images may be either still images or moving images, but in this embodiment, the recorded images will be described as still images to facilitate understanding of the invention.

[0015] 1, a recorded image obtained by automatic shooting by the imaging device 100 is transmitted to a data server 130 on a network 170 via a communication terminal 110 connected to the same local area network (LAN) 120. The LAN 120 may be a network configured with a wired or wireless connection, and realizes mutual information communication between the imaging device 100 and the communication terminal 110. The LAN 120 is also connected to the network 170 via a router (not shown). The network 170 is a wide-area network such as the Internet, and enables information communication between devices outside the LAN 120 and the communication terminal 110.

[0016] The data server 130 is a server that accumulates recorded images. In the example of Fig. 1, only one imaging device 100 is shown, but the number of imaging devices 100 that capture recorded images is not limited to one, and may include multiple imaging devices. In this case, the data server 130 collects and accumulates the recorded images captured by each imaging device 100.

[0017] The providing server 140 provides a service (hereinafter referred to as a photography sales service) for selling the recorded images stored in the data server 130 to the user. When using the photography sales service, each user accesses a predetermined site (service providing site) related to the providing server 140 via a browsing application on the client terminal 160, which may be an electronic device such as a PC or a smartphone. In the service providing site of this embodiment, the user can view images related to the user among the recorded images stored in the data server 130 by logging in to the photography sales service using the user's unique identification information, and can select and purchase a desired recorded image from among them. Here, the recorded image related to the user corresponds to, for example, an image that has the user himself / herself, or the user's family, acquaintances, related parties, etc. as a subject, or an image having attributes specified by the user such as an event in which the user participated. The recorded image purchased by the user may be provided to the user in a manner such as downloading high-resolution data, or printing on a predetermined paper and mailing it.

[0018] The collection server 150 manages demand information that analyzes the demand trends of users based on information (purchase information) of recorded images purchased by each user of the photography and sales service. In the imaging system of this embodiment, the demand information is managed for each user, that is, the demand information for one user is described as being configured based on the purchase information of the user. The demand information is information that is referred to when the imaging device 100 determines the composition for automatic photography, and is information obtained by analyzing the tendency (demand trend) of the composition that the user desires to photograph from the composition of the subject related to the recorded image purchased by the user. When the user desires automatic photography by the imaging device 100, the demand information based on the user's past purchase information is transmitted to the imaging device 100, and the operation of the imaging device 100 is controlled so as to increase the frequency of automatic photography with the user's preferred composition. As a result, a recorded image that is likely to lead to a user's purchase is automatically photographed, so that from the user's perspective, it becomes easier to obtain a recorded image in which the subject appears in a desired state, and from the perspective of the service provider, an increase in sales can be expected.

[0019] Although details will be described later, the collection server 150 of this embodiment identifies the occupancy rate (area relative to the entire recorded image) of the face area of ​​a person (family member, etc.) related to the user in the recorded image purchased by the user, and analyzes the user's demand tendency for each occupancy rate. The analysis is performed by machine learning, and the collection server 150 obtains the learning result of the user's demand tendency by deriving the occupancy rate of the subject's face area in the purchased recorded image and learning the proportion of each occupancy rate in the entire recorded image purchased.

[0020] For simplicity, in this embodiment, the devices constituting the imaging system are described as being in the form of Fig. 1, but the implementation of the present invention is not limited to this. The functions (image recording, sales, learning, etc.) performed by each device such as a server do not need to be performed by one device, and some functions may be performed by multiple devices working together, or multiple functions may be performed by one device. In addition, some functions performed by a device such as a server may be performed by the imaging device 100.

[0021] Configuration of the Imaging Device 100 Next, the configuration and functional configuration of the imaging device 100 of this embodiment will be described in detail with reference to FIGS.

[0022] FIG. 2(a) shows a schematic external configuration of the imaging device 100. As shown in the figure, in the imaging device 100, a lens barrel 201, which is a housing including a group of photographing lenses for capturing images and an imaging element, is attached to a fixed part 202 via a rotation mechanism. In the example shown in the figure, a fixed three-dimensional coordinate system (XYZ Cartesian coordinate system) is defined for the fixed part 202. As shown in FIG. 2(b), the rotation mechanism includes a tilt rotation part 203 that rotates the lens barrel 201 in the pitch direction around the X axis, and a pan rotation part 204 that rotates the lens barrel 201 in the yaw direction around the Y axis. That is, in the imaging device 100 of this embodiment, the lens barrel 201 is configured to be able to change the imaging direction by a two-axis rotation mechanism including the tilt rotation part 203 and the pan rotation part 204.

[0023] The fixed section 202 of the imaging device 100 is provided with an angular velocity meter 205 and an accelerometer 206 that detect vibrations and movement occurring in the imaging device 100 and the lens barrel 201. When a shake detection section 309 described below detects vibrations based on the outputs from the angular velocity meter 205 and the accelerometer 206, driving control is performed on at least one of the tilt rotation section 203 and the pan rotation section 204 so as to perform shake correction and tilt correction.

[0024] FIG. 3 is a block diagram illustrating an example of the functional configuration of the imaging device 100. As shown in FIG.

[0025] The first control unit 323 includes a processor (e.g., a CPU, a GPU, a microprocessor, an MPU, etc.) and a memory (e.g., a DRAM, an SRAM, etc.). These execute various processes to control the operation of each block of the imaging device 100 and control data transfer between each block. The non-volatile memory (Flash ROM) 316 is an electrically erasable and recordable memory, and stores constants, programs, etc. for the operation of the first control unit 323.

[0026] The lens barrel 201 includes a zoom unit 301 including a zoom lens that changes magnification, and a focus unit 303 including a lens that adjusts focus. The zoom unit 301 is driven and controlled by a zoom control unit 302, and the focus unit 303 is driven and controlled by a focus control unit 304.

[0027] The imaging unit 306 includes an imaging element, receives light incident through a lens group, and outputs digital image data obtained by analog-to-digital (A / D) conversion of charge information according to the amount of light to an image processing unit 307. The image processing unit 307 applies various types of image processing to the input image data. When image data is input from the imaging unit 306, the image processing unit 307 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the image data, and outputs the digital image data after the processing. The digital image data output from the image processing unit 307 is converted into a recording format such as JPEG format by an image encoding unit 308, and is transmitted to a memory 315 or an output unit 317 described later.

[0028] The rotation drive unit 305 drives the tilt rotation unit 203 and the pan rotation unit 204 to rotate the lens barrel 201 in the pitch direction and the yaw direction. The rotation drive of the lens barrel 201 by the rotation drive unit 305 is used not only to change the imaging direction, but also for shake correction to cancel vibrations occurring in the imaging device 100. As described above, the shake detection unit 309 includes the angular velocity meter 205 that detects the angular velocity of the imaging device 100 in three axial directions, and the accelerometer 206 that detects the acceleration of the device in three axial directions. In this case, the rotation angle, shift amount, etc. of the imaging device 100 are derived based on the signal detected by the shake detection unit 309, and are used for drive control of the rotation drive unit 305.

[0029] The audio input unit 313 acquires audio around the imaging device 100 as an audio signal from a microphone (not shown) provided in the imaging device 100, performs A / D conversion, and transmits the audio to the audio processing unit 314. The audio processing unit 314 performs audio-related processing such as optimization processing of the input digital audio data. The audio data processed by the audio processing unit 314 is then transmitted to the memory 315 by the first control unit 323. The memory 315 temporarily stores the image data and audio data obtained by the image processing unit 307 and the audio processing unit 314.

[0030] When recording is performed during shooting, the image processing unit 307 and the audio processing unit 314 read out image data and audio data temporarily stored in the memory 315, and perform processing such as encoding the image signal and encoding the audio data to generate compressed image data and compressed audio data. The first control unit 323 transmits this compressed image data and compressed audio data to the recording and playback unit 320.

[0031] The recording and reproducing unit 320 records the compressed image data, compressed audio data, and other control data related to shooting generated by the image processing unit 307 and audio processing unit 314 on the recording medium 321. When the audio data is not compressed and encoded, the first control unit 323 transmits the audio data generated by the audio processing unit 314 and the compressed image data generated by the image processing unit 307 to the recording and reproducing unit 320 and causes them to be recorded on the recording medium 321.

[0032] The recording medium 321 may be a storage device built into the imaging device 100, or a storage device configured to be detachable from the imaging device 100. The recording medium 321 can record various data such as compressed image data and compressed audio data generated by the imaging device 100, and a storage device with a larger capacity than the nonvolatile memory 316 is generally used. The recording medium 321 may include any type of storage device, such as a hard disk, optical disk, magneto-optical disk, CD-R, DVD-R, magnetic tape, nonvolatile semiconductor memory, flash memory, etc.

[0033] The recording and reproducing unit 320 reads (reproduces) various data such as compressed image data and compressed audio data recorded on the recording medium 321. Then, the first control unit 323 transmits the read compressed image data and compressed audio data to the image processing unit 307 and the audio processing unit 314. The image processing unit 307 and the audio processing unit 314 temporarily store the compressed image data and compressed audio data in the memory 315, decode them in a predetermined procedure, and transmit the decoded signals to the output unit 317 and the audio output unit 318.

[0034] The voice input unit 313 includes a plurality of microphones (not shown), and the voice processing unit 314 can detect the direction of a sound on a plane on which the plurality of microphones are installed, and is used for searching and automatic shooting, which will be described later. Furthermore, the voice processing unit 314 may be configured to be capable of detecting a specific voice command. The voice command may be configured to include a specific voice registered by the user in addition to a command registered in advance. Furthermore, the voice processing unit 314 may also perform sound scene recognition. In sound scene recognition, a network trained using machine learning based on a large amount of voice data prepared in advance determines which scene (sound scene) is being shot. For example, a network for detecting specific scenes such as "cheering," "clapping," and "uttering a voice" may be set in the voice processing unit 314, and the determination may be possible. In this case, the voice processing unit 314 may be configured to output a detection trigger signal to the first control unit 323 or the second control unit 311 when it detects a specific sound scene or a specific voice command.

[0035] The second control unit 311 is a control device provided separately from the first control unit 323, and controls the power supply to the first control unit 323. The imaging device 100 of this embodiment is provided with two types of control devices, the first control unit 323 and the second control unit 311, so that it can be started not only by operation input to an operation member such as a power switch, but also by the above-mentioned voice command or the detection result of the shaking detection unit 309. More specifically, when only limited functions such as voice detection by the voice processing unit 314 and shaking detection by the shaking detection unit 309 are executed, only the second control unit 311 operates to control these functions and determines whether the start-up condition of the imaging device 100 is satisfied. Then, when the second control unit 311 determines that the start-up condition is satisfied, it causes the first power supply unit 310 to supply power to the first control unit 323 so that the operation control of all blocks of the imaging device 100, including shooting, is performed. The first control unit 323 starts control according to the determination of the second control unit 311, and performs various operations according to the start-up cause. Therefore, the second power supply unit 312 is a power supply circuit configured to be always ON when power is supplied from a battery connected to the imaging device 100 or when external power is supplied via an adapter or USB, and supplies power to the second control unit 311.

[0036] The LED control unit 324 controls the LEDs provided in the imaging device 100 so as to light and blink in a predetermined pattern, for example, during shooting. The audio output unit 318 outputs a predetermined audio pattern from a speaker built into the imaging device 100, for example, during shooting. The output unit 317 is, for example, a video output terminal, and outputs an image signal to display a video on a connected external display or the like. The audio output unit 318 and the output unit 317 may be a single combined terminal, for example, a terminal such as an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal.

[0037] The communication unit 322 is a communication interface with an external device that the imaging device 100 has, such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, etc. The communication unit 322 transmits and receives, for example, audio data and image data to and from the external device. In this embodiment, the imaging device 100 communicates with the communication terminal 110 via the communication unit 322, and transmits recorded images and receives demand information related to the learning results of the collection server 150. The communication unit 322 may also receive control signals related to imaging, such as commands to start and end imaging, and pan / tilt and zoom drive, from an external device. In this case, the received control signals are transmitted to the first control unit 323 to control the drive of the imaging device 100. In this embodiment, the transmission of recorded images obtained by imaging in the imaging device 100 to the data server 130 is described as being performed via the communication terminal 110, but the implementation of the present invention is not limited to this. The transmission of recorded images to the data server 130 may be performed directly by the imaging device 100 communicating with the data server 130.

[0038] The recognition unit 325 detects the face of a person present in a scene, and performs face recognition processing to recognize which person the detected face belongs to, particularly whether or not the face belongs to the person to be photographed. In a photography sales service such as this embodiment, the face of the person to be photographed (subject) differs depending on which user is using the service. For this reason, when the imaging device 100 receives a request for automatic photography from a user, a facial image of the subject is acquired prior to automatic photography. The facial image of the subject may be, for example, received from the terminal used by the user who requested automatic photography, acquired from the collection server 150 or the data server 130 in association with the user, or acquired by the imaging device 100 through preliminary photography. Therefore, the recognition unit 325 performs face detection processing on the captured images intermittently acquired during the startup of the imaging device 100, and when a face is detected, determines whether or not the face matches the facial image of the person to be photographed through face recognition processing. The facial recognition process may use, for example, a 2D recognition method that recognizes the positions of the eyes, nose, mouth, etc. on the face and compares them with a database, or a 3D recognition method that uses an infrared sensor or a dot projector for recognition.

[0039] The setting unit 326 sets the photographing conditions for automatic photographing. The imaging device 100 of this embodiment performs automatic photographing without receiving an operation input related to a photographing instruction from a user, so that the photographing conditions are set to indicate under which conditions photographing is performed. The photographing conditions may be determined in any manner, but in the embodiment described below, they are set for an evaluation value that evaluates the degree to which a scene (subject) is suitable for photographing (suitability for photographing). The evaluation value related to the suitability may be obtained by evaluating a captured image for evaluation criteria such as a face detection situation, a face recognition result, a degree to which the subject has closed his / her eyes, the facial expression of the subject, the direction of the face, and the size of the subject area. The evaluation value is derived, for example, every time a captured image is obtained, and a decision is made to photograph when the evaluation value exceeds a predetermined photographing threshold. That is, the higher the evaluation value, the more suitable the scene captured by the imaging device 100 is for photographing. Therefore, the photographing conditions may determine a method for deriving the photographing threshold and the photographing threshold.

[0040] Overview of the photography sales service An outline of a photography sales service provided by using the imaging system of this embodiment will be described below. The photography sales service is roughly divided into a photography stage in which the imaging device 100 automatically photographs a specific subject (person) upon receiving a photography request from a user, and a sales stage in which the user selects and purchases a desired image from images recorded by automatic photography. The photography request may be accepted, for example, by the user operating an operation input member provided in the communication terminal 110, or may be accepted in response to the communication terminal 110 receiving corresponding information from the communication terminal used by the user. Alternatively, the photography request may be accepted via an operation input member (not shown) provided in the imaging device 100. When the communication terminal 110 accepts a photography request, it transmits information indicating that to the imaging device 100 via the LAN 120. At this time, in order to make a photography request, login to the mail order service is required, and the user is identified by the login.

[0041] The photographing stage corresponds to a period during which an event to be photographed, such as an athletic meet or a play, is held. The imaging device 100 is installed in advance in a venue or the like by, for example, an event organizer, and takes images of the venue during the event, and records recorded images when set photographing conditions are met. The imaging device 100 may be installed in a manner fixed (mounted) on a tripod head or a pedestal, or may be installed in a movable manner on a car or the like. The subject to be photographed is a person associated with the user who made the photographing request, and in an event such as an athletic meet, may be a child (children) of the user who participates in the competition in the athletic meet. The recorded images are uploaded and stored in the data server 130 via the communication terminal 110, either during the event or at a predetermined timing after the event.

[0042] On the other hand, the sales stage corresponds to the period when, for example, uploading of the recorded images recorded in the shooting stage to the data server 130 is completed and a viewing web page (hereinafter referred to as the sales page) corresponding to the event is installed on the providing server 140. More specifically, the sales stage corresponds to the period when the user is granted access authority to the sales page in the service providing site. The user can access the service providing site from a browsing application (browser) in the client terminal 160, log in by entering a user ID and password, and view the sales page to which the user is granted access authority. On the sales page, the user can check the contents of each recorded image, select whether or not to purchase it, and carry out the payment procedure for purchasing the selected recorded image.

[0043] In the imaging system of this embodiment, demand information indicating the demand tendency of the user is updated from time to time based on the purchase information, and the frequency of automatic photography is controlled to be different based on the demand information at the photography stage when the photography sales service is used thereafter. In other words, in the photography sales service, not only automatic photography and image sales that are completed at one event are performed, but the user's demand is learned based on the purchase information of the user related to one event and reflected in the photography control in the automatic photography of other events. As a result, at the photography stage, the photography frequency is controlled to be high when the subject is in a state (composition) that is in high demand from the user, and the number of recorded images taken in such a state can be increased. Also, at the sales stage, the user, who is the user, can select an image to purchase from a larger number of recorded images that match his / her preferences, and the service provider can expect to increase profits.

[0044] Therefore, to provide the photography and sales service of this embodiment, the following sequence of processing is executed in the imaging system. Note that, as an example of the present invention, the following describes a case where, for an event where the imaging system is used, when one user requests to photograph his / her child, imaging device 100 automatically photographs a scene including the child in response to the request.

[0045] User Registration Sequence In the photography and sales service of this embodiment, as described above, a person related to the user who made the photography request is selected as the subject in the photography stage. In other words, the subject for the user who will use the service needs to be set before the photography stage begins. In addition, in order to control the photography frequency of automatic photography based on past purchase information, it is necessary to identify which user the user is, and obtain the user's past purchase information or information on demand trends learned based on the past purchase information before the photography stage begins.

[0046] Therefore, it is assumed that each user has registered as a user before using the photography and sales service. In the imaging system of this embodiment, processing related to user registration is performed in the providing server 140. User registration may be performed, for example, on a service providing site related to the providing server 140, and each user performs user registration by accessing the service providing site using a browser on the client terminal 160. When a user registers as a user, information related to the user (user information) is registered and managed in a database held by the providing server 140, for example. The user information managed in the database may include identification information such as a user ID that uniquely identifies the user, as well as information on a face image of a person who is a subject related to the user.

[0047] Demand Information Generation Sequence In addition, in order to reflect the demand trend of images purchased by the user in the automatic shooting of the imaging device 100, the purchase information of each user is also shared with the collection server 150. The collection server 150 manages information for each user who uses the shooting and sales service in a database (not shown) in the same way as the provision server 140. The information managed for each user in the database of the collection server 150 includes the user ID of the user and information (demand information) of the learning result obtained by machine learning based on the purchase information of the user. The demand information is generated and sequentially updated in the collection server 150 based on the purchase information of the user. In the collection server 150 of this embodiment, the occupancy rate of the subject area in the recorded image purchased by the user is analyzed and learned as the occupancy rate that is in demand for the user. In this embodiment, the face area of ​​the subject associated with the user is identified as the subject area. Here, the occupancy rate of the face area of ​​the subject in the recorded image is obtained by deriving the area of ​​the face area as, for example, the number of pixels and dividing it by the total number of pixels of the recorded image. In this embodiment, the occupancy rate is learned by rounding off the 1st digit and classifying it into values ​​in 10% units. Therefore, the demand information generated in the collection server 150 is a learning result obtained by learning, for each user, the type of preferred occupancy rate and the degree to which the user desires recorded images of each type of occupancy rate (purchase frequency) based on the purchase information. When purchase information indicating a recorded image newly purchased by the corresponding user is acquired, the demand information is updated to reflect the user's demand tendency in combination with the previous learning result.

[0048] In order to make the demand information reflect the user's preferences appropriately, the demand information in the automatic photography described below may be used on the condition that purchase information of a predetermined number or more of recorded images is acquired for the user. In other words, if the number of purchased recorded images is small (the number of samples is small), even if the demand information is generated based on these purchase information, it may not appropriately indicate the user's demand. Therefore, the demand information generated for the user may be used on the condition that purchase information of a number of recorded images or more determined to show the demand trend is learned. At this time, the demand information itself may not be generated until the condition is met, that is, the demand information for the user may not be generated until the purchase information of a predetermined number or more of recorded images is purchased.

[0049] Automatic shooting sequence Also, in the shooting stage, the imaging device 100 recognizes the subject to be shot (a person related to the user who made the shooting request, hereinafter referred to as the target subject) included in the scene and performs automatic shooting without accepting operation input related to shooting instructions from the user. The imaging device 100 is installed so that it can shoot the scene, but the target subject is not always captured within the angle of view. For this reason, the automatic shooting sequence includes a search operation that finds the target subject and controls it to capture it in the angle of view, and a shooting decision operation that adjusts the composition of the target subject captured in the angle of view, evaluates the state, and decides whether to perform automatic shooting.

[0050] (Processing related to search operation) First, the search operation of the imaging device 100 will be outlined with reference to the flowchart of Fig. 4. The process corresponding to this flowchart can be realized by the first control unit 323 reading out a corresponding processing program stored in, for example, the non-volatile memory 316, and executing the program by expanding it in the built-in memory. This process will be described as being started, for example, when it is determined that the target subject is not captured in the angle of view, or when the position of the target subject captured in the angle of view has changed.

[0051] In S401, under the control of the first control unit 323, the recognition unit 325 acquires the distribution status of a predetermined object in an image captured with the current imaging settings. More specifically, the image processing unit 307 first applies a predetermined image processing to the captured image acquired by the imaging unit 306 to generate an image for subject detection (hereinafter referred to as a detection image). The recognition unit 325 then executes a detection process for detecting an image of a predetermined object in the detection image to acquire the distribution status. In this embodiment, since the target subject is a person, the recognition unit 325 detects an image of a person as the predetermined object in the detection process.

[0052] The detection of a person's image may be performed by detecting a face or a human body from the detection image. For example, in face detection processing, a pattern for identifying a person's face is used, and an area showing the pattern in the detection image is detected as a face area of ​​the person. At this time, the recognition unit 325 simultaneously derives a reliability indicating the likelihood that the detected face area is a face. The reliability is derived based on, for example, the size of the face area in the detection image, the degree of agreement with the face pattern, etc.

[0053] In S402, the first control unit 323 derives the shake (vibration) occurring in the imaging device 100 as an absolute angle based on the information on the angular velocity and acceleration acquired by the shake detection unit 309. Then, the first control unit 323 derives, as an image shake correction amount, a rotation angle for performing image shake correction by driving the tilt rotation unit 203 and the pan rotation unit 204 in an angular direction that cancels the derived absolute angle.

[0054] In S403, the first control unit 323 determines the installation state of the imaging device 100. More specifically, the first control unit 323 determines the installation state based on the shooting direction and movement amount of the imaging device 100 obtained based on the angular velocity information and acceleration information acquired by the shaking detection unit 309, or the position information output by a GPS sensor (not shown). The installation state indicates the current vibration state and movement state of the imaging device 100. For example, when the imaging device 100 is moving faster than a predetermined speed, the first control unit 323 determines that the imaging device 100 is in a "vehicle moving state" since it is estimated that the imaging device 100 is attached to a moving body such as a car. Furthermore, for example, when the change amount of the imaging direction is smaller than a reference value, the first control unit 323 determines that the imaging device 100 is in a "stationary shooting state" with almost no shaking. Furthermore, for example, when the change amount of the imaging direction is larger than a reference value, the first control unit 323 determines that the imaging device 100 is in a "handheld state" since it is estimated that a person is holding the imaging device 100. Since the changes in the images acquired sequentially may differ depending on the installation state of the imaging device 100, the information on the installation state obtained in this step is referenced in determining the search target area described below.

[0055] In S404, the first control unit 323 next determines the imaging direction in which to search for the presence of a target subject. The imaging direction is determined by including the steps of "area division," "deriving the importance level for each area," and "determining the search target area." Each step will be described in more detail below.

[0056] (1) Area division The process of dividing the areas will be described with reference to FIG. 5(a). In the example of FIG. 5(a), the imaging device 100 is placed at the origin O, and the entire periphery centered on the imaging device 100 is divided into areas. In the example of FIG. 5(a), the tilt direction and the pan direction are each divided in units of 22.5 degrees. On the other hand, in the embodiment of division as shown in FIG. 5(a), as the tilt angle moves away from 0 degrees, the horizontal circumferential distance becomes shorter, and the divided areas become narrower. For this reason, in the region where the tilt angle is 45 degrees or more as shown in FIG. 5(b), the width of the divided areas in the horizontal direction (pan direction) is set to be larger than 22.5 degrees.

[0057] 5(c) and (d) show examples of divided areas within the imaging angle of view. In FIG. 5(c), axis 501 indicates the orientation of imaging device 100 at the time of initialization, and area division is performed with this orientation as the reference position. Area 502 indicates the area captured in the angle of view captured at the time of initialization, and an example of a captured image captured in this case is shown in FIG. 5(d). As shown in the figure, areas 503 to 518 in the captured image each correspond to a different divided area.

[0058] (2) Deriving the importance level for each area Next, the first control unit 323 derives an importance level indicating a priority order for searching for each area divided as described above, depending on the state of the subject (person) present in the area. The importance level based on the state of the subject is derived based on, for example, the number of people present in the area, the size of the person's face, the direction of the face, the certainty of face detection, the person's facial expression, the person's individual recognition result, etc. When deriving the importance level for each area, the importance level may be further adjusted based on the installation state of the imaging device 100 determined in S403.

[0059] In this embodiment, since the target subject is a person, the importance level of each area is derived based on information about the person, but the importance level may be derived based on other information. The importance level may be derived according to the situation of the scene, for example, and may be derived using the recognition results of objects in the scene, the scene discrimination results (blue sky, backlight, evening scene, etc.), the sound level or voice recognition results from the direction of the area, motion detection information within the area, etc.

[0060] In addition, since the above conditions alone may result in the area with the highest importance level remaining the same unless there is a change in each area, and as a result the area may not change to a search target area, the importance level may be adjusted according to past shooting information. Specifically, the first control unit 323 may lower the importance level of an area that has been designated as a search target area for a predetermined period of time, or may lower the importance level of an area in the direction in which shooting was performed within the predetermined period of time.

[0061] (3) Determining the search area The first control unit 323 determines the search target area based on the importance level derived for each area. For example, the first control unit 323 may determine the area with the highest importance level obtained in the step (2) as the search target area.

[0062] Once the search target area has been determined in this manner, the first control unit 323 determines the imaging direction of the imaging device 100 that will capture the area in its angle of view, and derives the pan and tilt angles (hereinafter referred to as the search target angles) required to point the lens barrel 201 in the imaging direction.

[0063] In S405, the first control unit 323 executes pan / tilt driving related to the change of the imaging direction. More specifically, the first control unit 323 derives the pan / tilt driving amount by adding the image shake correction amount at the control sampling frequency derived in S402 and the driving angle based on the search target angle derived in S404. Then, the first control unit 323 causes the rotation driving unit 305 to perform driving control of the tilt rotation unit 203 and the pan rotation unit 204 with the derived pan / tilt driving amount.

[0064] In S406, the first control unit 323 judges whether or not the target subject is present in the scene. This judgment is made by the recognition unit 325 performing face recognition processing using the captured image with respect to the imaging direction changed in S405. More specifically, the recognition unit 325 detects a face area in the captured image in the same manner as in S401, and executes face recognition processing on each of the detected face areas to judge whether or not it is the face area of ​​the target subject. The face recognition of the target subject is performed by acquiring a face image of the target subject associated with the user who made the shooting request from the provision server 140 and using the face image. If it is determined as a result of the face recognition processing that the face area of ​​the target subject is included in the captured image, the first control unit 323 judges that the target subject is present in the scene and ends the processing related to this search operation. If the first control unit 323 judges that the target subject is not present in the scene, the process returns to S401 and the search operation is repeated.

[0065] (Processing related to shooting judgment) When it is determined in the process related to the search operation that the target subject exists in the scene, the first control unit 323 executes a process related to shooting determination. More specifically, the first control unit 323 determines a composition for shooting the target subject and adjusts shooting parameters, and then determines whether or not to shoot an image for recording based on a captured image (determination image) acquired by the imaging unit 306.

[0066] In the imaging device 100 of this embodiment, the occupancy rate of the face area of ​​the target subject in an image recorded when automatic imaging is performed (hereinafter referred to as the recording occupancy rate) is determined as a composition for photographing the target subject. In other words, the composition when the imaging device 100 of this embodiment performs automatic imaging is determined based on the extent to which the face of the target subject occupies the angle of view. For example, when the first control unit 323 determines the recording occupancy rate, the first control unit 323 derives a zoom magnification to be applied to the zoom unit 301 based on the current size of the face area of ​​the target subject (in the captured image used for face recognition in S406) and the determined recording occupancy rate. Then, the zoom control unit 302 controls the zoom unit 301 based on the zoom magnification, and a judgment image showing the determined composition (recording occupancy rate) is acquired by the imaging unit 306.

[0067] Incidentally, the photography and sales service sells recorded images autonomously photographed by the imaging device 100 to users, and at the stage where the user's taste (demand tendency) cannot be specified, it is necessary to photograph with various compositions in order to understand what composition the user prefers. For this reason, in the default setting, that is, when demand information about the user who made the photography request is not acquired, the first control unit 323 randomly determines the recording occupancy rate and controls so that the automatic photography composition becomes diverse. In other words, when demand information is not acquired and automatic photography is performed with the default setting, the recording occupancy rate for the composition is adopted completely randomly by lottery processing or the like, for example, one of the occupancy rates 10% to 100% (10 types) set in 10% increments. In this embodiment, in order to make the invention easier to understand, it is assumed that the collection server 150 does not configure demand information related to the user until the number of recorded images purchased by the user exceeds a predetermined number. In other words, the fact that the imaging device 100 acquires demand information means that the learning result of the demand tendency of the composition based on the user's purchase history is accumulated. Conversely, if the imaging device 100 has not acquired demand information, this means that there is no purchase history of the user, or the number of purchases is small, and the demand trend for the composition of the user is not known.

[0068] However, the recorded images obtained by automatic shooting with a randomly determined recording occupancy rate may include compositions that do not match the user's preferences, and as a result, even if sold, the images are not purchased by the user, and it is difficult to generate revenue from the shooting and sales service. For this reason, in the imaging device 100 of this embodiment, when the user's demand tendency can be identified, the recording occupancy rate is determined by reflecting the demand tendency, thereby increasing the number of recorded images with compositions that match the user's preferences. More specifically, when demand information related to the user who made the shooting request is acquired, the first control unit 323 controls the adoption probability of the occupancy rate that is determined in the demand information as being in demand by the user to be higher than the default case. At this time, the adoption probability of each occupancy rate defined in the demand information is increased according to the level of the user's demand (degree of preference) for that occupancy rate.

[0069] For example, assume that the result of learning the demand tendency (occupancy rate of the face area of ​​the target subject) of a user for four recording images purchased by the user is as shown in FIG. 6(a). In the example of FIG. 6(a), the demand information includes the learning result that, for the four recording images purchased by the user, two images have an occupancy rate of 40%, one image has an occupancy rate of 20%, and one image has an occupancy rate of 10%. In this case, the first control unit 323 sets four types of recording occupancy rates, namely, "occupancy rate 40%", "occupancy rate 20%", and "occupancy rate 10%" that are in demand in the demand information, and "random occupancy rate" related to the default, as candidates for the recording occupancy rate, and adopts one of them as the recording occupancy rate. That is, when the first control unit 323 acquires the demand information, it includes "occupancy rate 40%", "occupancy rate 20%", and "occupancy rate 10%" that are actually in demand by the user as candidates for the recording occupancy rate, in addition to "random occupancy rate". For this reason, the probability of these occupancy rates (40%, 20%, 10%) being adopted is higher than when using the lottery process with only the default setting of "random occupancy rate."

[0070] Specifically, the probability of each candidate being adopted changes depending on whether or not demand information has been acquired, as shown in FIG. 6(b). FIG. 6(b) shows the ratio (adoption probability) of each candidate for the recording occupancy rate when demand information has not been acquired, and the ratio of each candidate for the recording occupancy rate when demand information has been acquired. The former is the default setting, and the only candidate for the recording occupancy rate is "random occupancy rate." The first control unit 323 performs a lottery process to adopt one of 10 types of occupancy rates ranging from 10% to 100% each time as a composition for capturing the target subject. The first control unit 323 then performs control to change the zoom magnification so that the recording occupancy rate becomes the result of the lottery.

[0071] On the other hand, the latter shows a case where the demand information of FIG. 6(a) is acquired, and the candidates for the recording occupancy rate include "40% occupancy rate", "20% occupancy rate", and "10% occupancy rate" in addition to "random occupancy rate". That is, the first control unit 323 first performs a lottery process to adopt one of "random occupancy rate", "40% occupancy rate", "20% occupancy rate", and "10% occupancy rate" as a composition for capturing the target subject. Then, if the lottery result is any of "40% occupancy rate", "20% occupancy rate", and "10% occupancy rate", the first control unit 323 controls to change the zoom magnification so that the recording occupancy rate of the lottery result is obtained. On the other hand, if the lottery result is "random occupancy rate", the first control unit 323 further performs a lottery process to adopt one of 10 types of occupancy rates from 10% to 100%, as in the case of the default, and controls to change the zoom magnification so that the recording occupancy rate of the lottery result is obtained.

[0072] In the example of FIG. 6(b), when demand information is acquired, the adoption probability of "random occupancy rate" is set to 60%, and the remaining 40% is assigned to the adoption probability of the occupancy rate determined as being in demand in the demand information. Specifically, when the demand tendency of the user is analyzed as in FIG. 6(a), half of the remaining 40%, i.e., 20% of the total, is assigned to "40% occupancy rate", and another half of that, i.e., 10% of the total, is assigned to "20% occupancy rate" and "10% occupancy rate". In this way, in the imaging device 100 of this embodiment, when demand information is acquired for a user who has made a shooting request, the adoption probability is set according to the level of demand so that a specific recording occupancy rate in demand is likely to be adopted. Then, a lottery process is performed based on the adoption probability of each candidate that has been set, and the recording occupancy rate for the judgment image used for shooting judgment is determined.

[0073] In this way, it is possible to increase the frequency of photographing with a composition that matches the user's preference while retaining the aspect of photographing a target subject with various compositions. In other words, when performing automatic photographing in response to a photographing request from a user with a purchase history, the imaging device 100 of this embodiment can increase the frequency of adopting a composition related to a recording occupancy rate that matches the user's preference estimated from the purchase history.

[0074] In FIG. 6(b), for the sake of illustration, the adoption probability of "random occupancy rate" when demand information is acquired is set to 60%, but the implementation of the present invention is not limited to this. The adoption probability of 60% set for the random occupancy rate is merely an example and may be changed to another value. It goes without saying that this may be changed according to the size of the sampling number learned when composing the demand information (the number of purchases by the user). "Random occupancy rate" does not necessarily need to be included as a candidate for composition determination, and for example, when a specific condition is met, only occupancy rates determined to be in demand in the demand information may be candidates.

[0075] Hereinafter, the operation of the shooting judgment including the determination of the recording occupancy rate and the change of the zoom magnification will be described in detail with reference to the flowchart of FIG. 7. The process corresponding to this flowchart can be realized by the first control unit 323 reading out the corresponding processing program stored in, for example, the non-volatile memory 316, and expanding it to the built-in memory and executing it. This process will be described as being started when, for example, the execution of the process related to the search operation determines that the target subject exists in the scene and an image for judgment is captured. Furthermore, it is assumed that information on the face image of the target subject related to the user who made the shooting request has been acquired prior to the execution of this process. In addition, if the collection server 150 holds demand information related to the user, it is assumed that the demand information has been acquired. Furthermore, since this process is executed on the condition that the target subject exists in the scene (imaging range) in S406 of the process related to the above-mentioned search operation, it is assumed that the captured image output by the imaging unit 306 includes at least the face area of ​​the target subject. In this embodiment, in order to simplify the understanding of the invention, the shooting performed by the imaging device 100 will be described as still image shooting. However, the present invention is not limited to this embodiment, and may be used to determine whether or not to perform different types of photography, such as movie photography or continuous shooting.

[0076] In S701, the first control unit 323 determines the recording occupancy rate based on the demand information. As described in detail including the example of Fig. 6, the processing of this step differs depending on whether or not the demand information indicating the demand tendency of the composition related to the user (the occupancy rate of the face area of ​​the target subject in the image) has been acquired. First, the first control unit 323 identifies candidates for the recording occupancy rate, and determines the recording occupancy rate based on one occupancy rate adopted by performing a lottery process on the candidates.

[0077] In S702, the first control unit 323 executes zoom driving related to a change in zoom magnification. More specifically, the first control unit 323 determines a zoom magnification (zoom parameter) to be applied to the zoom unit 301 based on the occupancy rate of the face region of the target subject in the captured image output by the imaging unit 306 and the occupancy rate for recording determined in S701. Then, the first control unit 323 transmits the determined zoom magnification to the zoom control unit 302, and causes the zoom unit 301 to perform zoom driving. In the captured image output by the imaging unit 306 after the zoom driving in this step, the occupancy rate of the face region of the target subject is the occupancy rate for recording determined in S701, and in the subsequent steps, a determination is made as to whether or not to perform shooting based on the captured image. Hereinafter, the captured image is referred to as a determination image.

[0078] In this embodiment, in order to facilitate understanding of the invention, the occupancy rate of the face area of ​​the target subject in the judgment image is described as the recording occupancy rate, but the implementation of the present invention is not limited to this. The zoom magnification that can be set in the zoom unit 301 is limited according to its driving resolution, and the state of the person who is the target subject also changes sequentially, so the occupancy rate of the face area of ​​the target subject in the judgment image may not necessarily be the recording occupancy rate. Therefore, a predetermined allowable error may be provided for the zoom magnification determined in relation to the acquisition of the judgment image and the occupancy rate of the face area of ​​the target subject in the actually obtained judgment image. In other words, the recording occupancy rate determined in S701 does not strictly determine the occupancy rate of the face area of ​​the target subject in the judgment image or the recorded image obtained by shooting, but determines a target value in the process of acquiring these images.

[0079] In S703, the first control unit 323 derives an evaluation value of the judgment image. In deriving the evaluation value, the face detection result or face recognition result related to the judgment image, and information such as the facial expression of the target subject may be referenced.

[0080] In S704, the first control unit 323 judges whether the evaluation value of the judgment image exceeds the shooting threshold. The shooting threshold is a value determined in advance by the setting unit 326 for the state of the target subject to be photographed. If the first control unit 323 judges that the evaluation value of the judgment image exceeds the shooting threshold, the process proceeds to S705, and if it judges that the evaluation value does not exceed the shooting threshold, the process is completed without photographing.

[0081] In S705, the first control unit 323 judges whether a predetermined time has passed since the most recent photographing. Here, the predetermined time is a time threshold set by the setting unit 326 to adjust the photographing interval by the imaging device 100. The evaluation value of the judgment image may change from moment to moment, but the value may be similar between photographed images captured close to each other, such as images continuously output from the imaging unit 306. Therefore, once the evaluation value of the judgment image exceeds the photographing threshold, the evaluation value of the subsequently acquired photographed image may also exceed the photographing threshold. For this reason, if automatic photographing is performed simply on the condition that the evaluation value exceeds the photographing threshold, similar images may be recorded multiple times. Therefore, in the imaging device 100 of this embodiment, in order to prevent excessive photographing frequency, once photographing is performed, it is controlled so that photographing is not performed until a predetermined time has passed, even if the evaluation value exceeds the photographing threshold. If the first control unit 323 determines that the predetermined time has elapsed since the most recent image capturing, it shifts the process to S706, and if it determines that the predetermined time has not elapsed, it completes this process without capturing any more images.

[0082] In S706, the first control unit 323 causes the imaging unit 306 to perform an imaging operation, executes imaging processing for recording the obtained image as a record image, and completes this processing. The record image may be recorded in the imaging device 100, such as the non-volatile memory 316 or the recording medium 321, or may be recorded in the data server 130 by transmitting the record image via the communication unit 322.

[0083] As described above, the control device of this embodiment can realize automatic shooting that makes it easy to record images that meet the user's demand. More specifically, the control device acquires demand information that indicates the user's demand tendency related to the occupancy rate of the face area, which is configured based on the occupancy rate of the face area of ​​the target subject in the recording image purchased by the user. Then, the control device determines candidates for a plurality of types of recording occupancy rates based on the demand information, and adopts one of them by lottery processing or the like for use in shooting. Here, the control device sets the adoption frequency of each candidate so that the adoption frequency of the candidate determined to be in high demand by users in the demand information increases, thereby making it easy to record images with the occupancy rate preferred by the user. As a result, in a service that sells recording images recorded by automatic shooting, users can easily obtain images in a suitable state that matches their preferences. Also, from the perspective of the service provider, the possibility that the recording image will be purchased by the user increases, and an increase in profits is expected.

[0084] In this embodiment, the target subject is described as a person related to the user, such as the user who made the shooting request, or the user's family or acquaintances, but it will be easily understood that the implementation of the present invention is not limited to this. The present invention does not require that the target subject be a person, and any type of object, such as an animal such as a dog or a cat, or other object, can be used as the subject. In this case, for example, the user's demand tendency may be analyzed based on the occupancy rate of the area (subject area) related to the target subject in the purchased recorded image. Therefore, the learning of the user's demand tendency is not limited to the occupancy rate of the person's face area in the recorded image, and the area from which the occupancy rate is derived may be adaptively changed depending on the type of object selected as the target subject. In other words, the subject area is not limited to the person's face area, but may be a specific part or the whole body of the human body, or a part or the whole of the target object.

[0085] In addition, in the present embodiment, the detection of the subject is described as being performed by a method of identifying an area that matches a predetermined image pattern, but the implementation of the present invention is not limited to this. The subject detection may be performed by other methods, such as a method of extracting a characteristic subject using a histogram of hue, saturation, etc. in a captured image. In the method of extracting a characteristic subject, a distribution derived from a histogram of the hue, saturation, etc. of the image of the subject captured within the shooting angle of view is divided into a plurality of sections, and a process of classifying the captured image for each section is performed. For example, a histogram of a plurality of color components is created for the captured image, and the image is divided into areas that belong to the same combination of sections, and the area in which the image of the subject appears is identified. In this case, the recognition unit 325 can derive an evaluation value for each area of ​​the identified subject, and identify the area with the highest evaluation value as the main subject area.

[0086] In the present embodiment, the process related to the search operation of the automatic shooting sequence is described as executing the process related to shooting judgment on the condition that the target subject exists in the scene, but the implementation of the present invention is not limited to this. Automatic shooting by the imaging device 100 does not need to be limited to a specific target subject, and may be, for example, a composition determination according to a demand trend for any detected person. In this case, the first control unit 323 may, for example, recognize a person included in the judgment image, identify a user associated with the person, obtain demand information related to the user, and perform shooting judgment.

[0087] [Embodiment 2] In the above-described embodiment, the aspect of determining the occupancy rate of the face area of ​​the target subject in the image recorded when automatic shooting is performed as the composition for shooting the target subject has been described, but the implementation of the present invention is not limited to this. The composition is not limited to being specified by the occupancy rate of the area related to the target subject in such an angle of view, and may be specified using other items. In this embodiment, the aspect of specifying the composition for shooting the target subject as the position of the face area of ​​the target subject in the image recorded when automatic shooting is performed (hereinafter referred to as the face position for recording) will be described. Note that the imaging system of this embodiment may be similar to the system exemplified in the first embodiment, but the automatic shooting sequence in the imaging device 100 and the demand information learned and managed in the collection server 150 are different.

[0088] In the collection server 150 of this embodiment, the position of the face area of ​​the target subject in the recorded image purchased by the user is analyzed and learned as a face position that is in demand for the user. In this embodiment, the analysis of the face position of the target subject in the recorded image is performed including classification into which of nine areas (A to I) the face position is located, which are defined by dividing the image into three horizontally and three vertically, as shown in FIG. 8(a). The area in which the face position is located may be specified based on, for example, the position of the center of gravity of the face area of ​​the target subject included in the recorded image. Therefore, the demand information generated in the collection server 150 of this embodiment is configured as a learning result obtained by learning, for each user, the type of preferred face position and the degree to which the user desires recorded images with each type of face position (purchase frequency) based on purchase information.

[0089] Automatic shooting sequence Hereinafter, a description will be given of the automatic shooting sequence performed in the imaging device 100 of this embodiment. Note that, in each process of the automatic shooting sequence of this embodiment, steps that perform the same processes as in the first embodiment are given the same reference numbers and their description will be omitted, and the following will focus on the description of the steps that perform processes unique to this embodiment.

[0090] (Processing related to search operation) First, the search operation of the imaging device 100 will be outlined with reference to the flowchart of Fig. 9. The process corresponding to this flowchart can be realized by the first control unit 323 reading out a corresponding processing program stored in, for example, the non-volatile memory 316, and executing the program by expanding it in the built-in memory. This process will be described as being started, for example, when it is determined that the target subject is not captured in the angle of view, or when the position of the target subject captured in the angle of view has changed.

[0091] When the pan-tilt driving for changing the imaging direction is performed in S405, the first control unit 323 performs the zoom driving for changing the zoom magnification in S901. More specifically, the first control unit 323 determines the zoom magnification to be applied to the zoom unit 301 according to the size of the face area distributed in the search target area determined in S404. In this embodiment, in order to accurately identify the face area corresponding to the target subject and determine the composition and shoot, it is necessary to capture the face area included in the search target area at a size that allows face recognition. In other words, if the size of the face area in the judgment image that is the target of shooting judgment is too small, it is not possible to determine whether the person related to the face area is the target subject. Therefore, when the face area included in the search target area is smaller than the minimum size to which face recognition processing can be applied, the first control unit 323 changes the zoom magnification to the telephoto side and performs zoom driving so that the judgment image can be obtained with the size of the face area increased. On the other hand, if the size of the face area included in the search target area is too large, there is a possibility that the face area will not fit in the angle of view of the judgment image if the subject or the imaging device 100 moves by the time the judgment image is captured. For this reason, when the face area included in the search target area exceeds a predetermined size, the first control unit 323 changes the zoom magnification to the wide-angle side and performs zoom driving on the judgment image with the size of the face area reduced. In the imaging system of this embodiment, the face position is classified into areas such as those shown in FIG. 8(a) to learn the demand tendency of the user, so the size of the face area of ​​the target subject after zoom driving may be adjusted to a size that matches the size of the area. In this way, the success rate of the face recognition process in the judgment image can be increased, and appropriate angle of view control can be realized even when, for example, a specific subject is tracked and photographed. After performing such zoom driving, the first control unit 323 moves the process to S406.

[0092] (Processing related to shooting judgment) Next, when it is determined in the process related to the search operation that the target subject exists in the scene, the first control unit 323 executes a process related to shooting determination. More specifically, the first control unit 323 determines a composition for shooting the target subject and adjusts shooting parameters, and then determines whether or not to shoot an image for recording based on a captured image (determination image) acquired by the imaging unit 306.

[0093] As described above, in the imaging device 100 of this embodiment, the face position of the target subject in the image recorded when automatic shooting is performed is determined as the composition for shooting the target subject (recording face position). In other words, the composition when the imaging device 100 of this embodiment performs automatic shooting is determined based on the position at which the face of the target subject is captured in the angle of view. For example, when the first control unit 323 determines the recording face position, the first control unit 323 derives shooting parameters related to changing the shooting direction based on the position of the face area of ​​the target subject in the current captured image (the captured image used for face recognition in S407) and the subject distance and the determined recording face position. That is, the first control unit 323 derives the pan / tilt drive amount to be applied to the tilt rotation unit 203 and the pan rotation unit 204 so that the face area of ​​the target subject can be captured at the recording face position. Then, the rotation drive section 305 controls the tilt rotation section 203 and the pan rotation section 204 based on the pan / tilt drive amount, and an image for determination showing the determined composition (face position for recording) is acquired by the imaging section 306.

[0094] As in the first embodiment, the imaging device 100 uses default settings related to face position when the user's demand tendency has not been identified, and captures images with various compositions to understand what composition the user prefers. For this reason, in the default settings, the first control unit 323 randomly determines the face position for recording, and controls so that the compositions of automatic shooting are diverse. In the example of this embodiment, the demand information is configured by learning the user's demand tendency based on which of nine types of areas the face position in the recorded image is classified into, as shown in FIG. 8(a), so that one of these areas is adopted as the face position for recording by default.

[0095] On the other hand, since a recording image obtained by automatic shooting at a randomly determined face position for recording may include a composition that does not match the user's preference, in the imaging device 100 of this embodiment, if the user's demand tendency can be identified, the recording face position is adopted accordingly. In other words, when the imaging device 100 acquires demand information indicating the user's demand tendency, control is performed so that the adoption probability of a face position determined in the demand information as being in demand by the user is higher than the default case. At this time, the adoption probability of each face position determined in the demand information is increased according to the level of the user's demand (degree of preference) for that face position.

[0096] For example, assume that the result of learning the demand tendency (face position of the target subject) of a user for four recording images purchased by the user is as shown in FIG. 8(b). In the example of FIG. 8(b), the demand information includes the learning result that, for the four recording images purchased by the user, two images are face position E, one image is face position B, and one image is face position F. In this case, the first control unit 323 sets four types of face positions for recording, namely, "face position E", "face position B", and "face position F" for which demand is indicated in the demand information, and "face position random" related to the default, as candidates for face positions for recording, and adopts one of them as the face position for recording. That is, when the first control unit 323 acquires the demand information, it includes "face position E", "face position B", and "face position F" which are actually in demand by the user, as candidates for face positions for recording, in addition to "face position random". For this reason, the probability of adopting these face positions (E, B, F) is higher than the lottery process using only the default setting "face position random".

[0097] Specifically, the adoption probability of each candidate changes as shown in FIG. 8(c) depending on whether or not demand information is acquired. FIG. 8(c) shows the ratio (adoption probability) of each candidate for the face position for recording when demand information is not acquired, and the ratio of each candidate for the face position for recording when demand information is acquired. The former is the default setting, and the candidate for the face position for recording is only "random face position", and the first control unit 323 performs a lottery process to adopt one of nine types of face positions A to I as a composition for capturing the target subject each time. Then, the first control unit 323 performs control to change the pan-tilt angle so that the face position for recording becomes the lottery result.

[0098] On the other hand, the latter shows a case where the demand information of FIG. 8(b) is acquired, and the candidates for the face position for recording include "face position E", "face position B" and "face position F" in addition to "face position random". That is, the first control unit 323 first performs a lottery process to adopt one of "face position random", "face position E", "face position B" and "face position F" as a composition for capturing the target subject. If the lottery result is "face position E", "face position B" and "face position F", the first control unit 323 derives the pan / tilt drive amount so that the face position for recording of the lottery result is obtained, and controls to change the shooting direction. On the other hand, if the lottery result is "face position random", the 323 further performs a lottery process to adopt one of nine types of face positions A to I, as in the default case, and controls to change the pan / tilt angle so that the face position for recording of the lottery result is obtained.

[0099] In the example of FIG. 8(c), when the demand information is acquired, the adoption probability of "random face position" is set to 60%, and the remaining 40% is assigned to the adoption probability of the face position determined to be in demand in the demand information. Specifically, when the demand tendency of the user is analyzed as shown in FIG. 8(b), half of the remaining 40%, i.e., 20% of the total, is assigned to "face position E", and another half of that, i.e., 10% of the total, is assigned to "face position B" and "face position F". In this way, in the imaging device 100 of this embodiment, when the demand information is acquired for the user who made the shooting request, the adoption probability is set according to the level of demand so that a specific recording face position in demand is likely to be adopted. Then, a lottery process is performed based on the adoption probability of each candidate that has been set, and the recording face position related to the judgment image used for shooting judgment is determined.

[0100] In this way, it is possible to increase the frequency of photographing with a composition that matches the user's preference while retaining the aspect of photographing a target subject with various compositions. In other words, when performing automatic photographing in response to a photographing request from a user with a purchase history, the imaging device 100 of this embodiment can increase the frequency of adopting a composition related to a face position for recording that matches the user's preference estimated from the purchase history.

[0101] In FIG. 8(c), for the sake of illustration, the adoption probability of "random face position" when demand information is acquired is set to 60%, but the implementation of the present invention is not limited to this. The adoption probability of 60% set for random face position is merely an example and may be changed to another value. It goes without saying that this may be changed according to the size of the sampling number learned when composing the demand information (the number of purchases by users). "Random face position" does not necessarily need to be included as a candidate for composition determination, and for example, when a specific condition is deemed to be met, only face positions determined to be in demand in the demand information may be set as candidates.

[0102] Hereinafter, the operation of determining whether or not to shoot, including the determination of the recording occupancy rate and the change of the zoom magnification, will be described in detail with reference to the flowchart of FIG. 10. The process corresponding to this flowchart can be realized by the first control unit 323 reading out the corresponding processing program stored in the non-volatile memory 316, for example, and developing and executing it in the built-in memory. This process will be described as being started when it is determined that the target subject exists in the scene by executing a process related to a search operation, for example. As in the first embodiment, it is assumed that information on the face image of the target subject related to the user who made the shooting request has been acquired prior to the execution of this process. In addition, if the collection server 150 holds demand information related to the user, it is assumed that the demand information has been acquired.

[0103] In S1001, the first control unit 323 determines a face position for recording based on demand information. The processing of this step differs depending on whether or not demand information indicating a demand tendency of a composition related to a user has been acquired, as described in detail including the example of Fig. 8. The first control unit 323 first identifies candidates for the face position for recording, and determines the face position for recording based on one face position adopted by performing a lottery process on the candidates.

[0104] In S1002, the first control unit 323 executes pan / tilt driving so as to capture the face area of ​​the target subject at the face position for recording determined in S1001. More specifically, the first control unit 323 determines the pan / tilt driving amount to be applied to the pan rotation unit 204 and the tilt rotation unit 203 based on the face position of the target subject in the captured image output by the imaging unit 306 and the face position for recording determined in S1001. Then, the first control unit 323 transmits the determined pan / tilt driving amount to the rotation drive unit 305, and causes the pan rotation unit 204 and the tilt rotation unit 203 to perform pan / tilt driving. The captured image (image for determination) output by the imaging unit 306 after the pan / tilt driving in this step has the face position of the target subject at the face position for recording determined in S1001. In steps from S703 onwards, it is determined whether or not to perform shooting based on the image for determination.

[0105] As described above, the control device of this embodiment can realize automatic shooting that makes it easy to record images according to the user's demand. More specifically, the control device acquires demand information that indicates the user's demand tendency related to face positions, which is configured based on the face positions of the target subject in the recording image purchased by the user. Then, the control device determines a plurality of types of candidates for recording face positions based on the demand information, and adopts one of them by lottery processing or the like for use in shooting. Here, the control device sets the adoption frequency of each candidate so that the candidate determined to be in high demand by users in the demand information is adopted more frequently, thereby making it easy to record an image with a face position that the user prefers. As a result, in a service that sells recording images recorded by automatic shooting, the user can easily obtain an image in a suitable state that matches his or her preferences. Also, from the viewpoint of the service provider, the possibility that the recording image will be purchased by the user increases, and an increase in profits is expected.

[0106] [Variation 1] In the above-described first and second embodiments, the demand tendency of the occupancy rate of the face area of ​​the target subject in the image and the face position in the image are learned as the demand information, and the drive control related to the composition is performed by adopting the recording occupancy rate or the recording face position from the candidates based on the demand information. However, the present invention is not limited to the above-described embodiments as long as it controls the composition with high demand to be easily adopted for automatic shooting according to the demand tendency of the composition related to the user. In other words, the learning of the user's preferred composition and the determination of the composition related to automatic shooting are not limited to those based on either the occupancy rate of the face area in the image or the face position, and may be based on a combination of these or other parameters that specify the composition.

[0107] In addition to the composition, the user's demand trend may also include image quality parameters such as exposure, ISO sensitivity, white balance, and depth of field based on the analysis results of the brightness and blur of the subject and its surroundings. In this case, the candidates for parameters to be adopted for automatic photography may include image quality parameters in addition to zoom and pan / tilt parameters, with the adoption probability adjusted according to the demand trend.

[0108] [Variation 2] In the above-mentioned embodiment and modified example, the adoption probability of each parameter is set according to the level of demand (demand level) of the composition of the target subject defined in the demand information, but the implementation of the present invention is not limited to this. That is, in the above-mentioned embodiment and modified example, the adoption frequency of each composition of the target subject is controlled to be increased in proportion to the value of the demand level of the composition, but the implementation of the present invention is not limited to this. The present invention only needs to control the adoption frequency of a composition with a high demand level defined in the demand information to be higher than that of a composition with a lower demand level, and the degree of adoption does not need to be set in proportion to the value of the demand level. In this case, the same adoption frequency may be set for two compositions with different demand levels whose demand levels are included in a predetermined range.

[0109] [Variation 3] In the above-mentioned embodiment and modified example, the evaluation value is derived by a common derivation method in all aspects, but for example, when the composition is determined to be in demand in the demand information, the evaluation value may be derived by adding a predetermined value. In this way, it is possible to further increase the frequency of shooting when the judgment image shows a composition in demand by users.

[0110] [Variation 4] In the above-mentioned embodiment and modified example, the collection server 150 performs machine learning based on the purchase information of each user to configure the demand information of the user, but the embodiment of the present invention is not limited to this. That is, the demand information does not need to be configured as a learning result of machine learning, and may be, for example, statistical data based on the purchase information configured in the form of a lookup table or the like.

[0111] [Variation 5] In the above-mentioned embodiment and modified example, the demand information of an individual is configured as the demand tendency of the user based on the purchase information of each user, but the implementation of the present invention is not limited to this. The demand information may be configured based on the purchase information of one or more users classified into any set, such as the class or grade to which the child who is the subject belongs, the school, or the area of ​​the school. For example, at an event held at a certain school, automatic shooting is performed with multiple children as subjects, and then learning is performed based on the recorded images purchased by each guardian, so that the demand information is configured as the demand tendency of all the guardians of the children who attend the school. In this case, at an event held at the school afterwards, the demand information related to all the guardians is transmitted to the imaging device 100 and referred to in the automatic shooting during the event, so that many recorded images with compositions that are desired on average by multiple users (guardians) are recorded. In this case, since the demand information to be referred to in the shooting judgment can be specified on an event-by-event basis, each user does not necessarily need to be identified at the shooting stage.

[0112] [Variation 6] In the above-mentioned embodiment and modified example, the imaging device 100 acquires demand information and performs drive control of the composition based on the demand information in the shooting judgment, but the implementation of the present invention is not limited to this. Operation control related to automatic shooting of the imaging device 100 may be performed, for example, by an external control device (communication terminal 110, etc.) connected to the imaging device 100 so as to be able to communicate with it. In this aspect, the control device adopts a composition based on the demand information, and judges whether or not to cause shooting to be performed based on a judgment image obtained with the adopted composition. Then, when the control device judges that shooting should be performed, the operation of the imaging device 100 is controlled based on the judgment result.

[0113] [Variation 7] In the above-mentioned embodiment and modified example, the recorded image is provided to the user in the form of a sale, but the implementation of the present invention is not limited to this. The recorded image may be provided to the user in any form that does not involve payment of money. In addition, the demand trend of the user may be reflected in the demand information by collecting and analyzing any information based on the user's operation, such as input information of evaluation indicating the user's preference for each image, favorite registration, the operation of the user selecting a preferred image, the number of views, etc.

[0114] The disclosure of this specification includes the following control device, imaging device, control method, program, and imaging system. (Item 1) A control device for controlling an operation of an imaging device for acquiring an image provided to a user by automatic photographing with a different composition, A first acquisition means for acquiring demand information indicating a demand tendency of a user regarding a composition of a subject; A setting means for setting a frequency of use in photography for each of a plurality of predetermined types of compositions based on the demand information acquired by the first acquisition means; a determination unit that determines a composition for photographing a subject by the imaging device based on the frequency of adoption of each of the plurality of types of compositions set by the setting unit; A control device comprising: (Item 2) 2. The control device according to item 1, wherein the setting means sets a higher frequency of adoption of a type of composition that is in higher demand in the demand information. (Item 3) The control device described in item 1 is characterized in that the setting means sets a higher adoption frequency for a type of composition having a first demand level in the demand information than for a type of composition having a second demand level lower in demand than the first demand level. (Item 4) Each of the plurality of types of compositions has a different occupancy rate of a subject area in a captured image, The demand information indicates the level of demand from users for each occupancy rate of a subject area in an image provided to the user. 4. The control device according to any one of items 1 to 3. (Item 5) each of the plurality of types of compositions has a different position of a subject area in a captured image; The demand information indicates the level of demand from users for each position of a subject area in an image provided to the users. 5. The control device according to any one of items 1 to 4. (Item 6) 6. The control device according to item 4 or 5, wherein the subject area in the captured image is a face area of ​​a person. (Item 7) The provision of the image to the user is carried out in the form of a sale of the image, The demand information is information configured based on the composition of the subject of the image purchased by the user. 7. The control device according to any one of items 1 to 6, (Item 8) The control device according to item 7, characterized in that the demand information is a result of machine learning learning of demand trends for subject composition based on images purchased by users. (Item 9) an identification means for identifying a user; a detection means for detecting a subject associated with the user identified by the identification means as a target subject; and The first acquisition means acquires the demand information associated with the user identified by the identification means, The determining means determines the composition to be photographed for the target subject detected by the detecting means. 9. The control device according to any one of items 1 to 8, (Item 10) The control device described in any one of items 1 to 8, characterized in that the demand information is information constructed based on the composition of the subjects in multiple images taken at a specified event and determined to be in high demand based on operations by multiple users. (Item 11) 11. The control device according to any one of items 1 to 10, further comprising a change unit that changes a shooting parameter of the imaging device based on the composition to be photographed determined by the determination unit. (Item 12) A second acquisition means for acquiring a captured image related to the composition to be photographed captured by the imaging device; a control means for controlling whether or not to cause the imaging device to take a photograph based on the captured image acquired by the second acquisition means; 12. The control device according to any one of items 1 to 11, further comprising: (Item 13) An imaging means; A control device according to any one of items 1 to 12, a means for causing the image capturing means to capture an image for recording based on the information on the composition to be photographed determined by the control device; An imaging device comprising: (Item 14) A control method for a control device that controls an operation of an imaging device that acquires an image provided to a user by automatic photographing with a different composition, comprising: A first acquisition step of acquiring demand information indicating a demand tendency of a user regarding a composition of a subject; A setting step of setting a frequency of use in photography for each of a plurality of predetermined types of compositions based on the demand information acquired in the first acquisition step; a determination step of determining a composition for photographing a subject by the imaging device based on the frequency of adoption of each of the plurality of types of compositions set in the setting step; A control method comprising the steps of: (Item 15) A program for causing a computer to function as each of the means of the control device according to any one of items 1 to 12. (Item 16) An imaging system for providing a user with images automatically captured by an imaging device while changing a composition, the imaging system including the imaging device, a control device for controlling an operation of the imaging device, and a collection server for collecting user demand trends based on the images provided to the user, The control device includes: An acquisition means for acquiring demand information indicating a user's demand tendency regarding a composition of a subject; a setting means for setting a frequency of use in photography for each of a plurality of predetermined types of compositions based on the demand information acquired by the acquisition means; a determination unit that determines a composition for photographing a subject by the imaging device based on the frequency of adoption of each of the plurality of types of compositions set by the setting unit; An imaging system comprising:

[0115] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0116] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0117] 100: imaging device, 110: communication terminal, 130: data server, 140: providing server, 150: collecting server, 160: client terminal, 323: first control unit, 316: non-volatile memory, 306: imaging unit, 307: image processing unit, 322: communication unit, 325: recognition unit, 326: setting unit

Claims

1. A control device for controlling the operation of an imaging device that acquires an image sold to a user by automatic shooting with a changed composition, comprising: a first acquisition means for acquiring demand information indicating a user's demand trend regarding the composition of a subject; a setting means for setting the adoption frequency for automatic shooting for each of a plurality of predetermined types of compositions based on the demand information acquired by the first acquisition means; a determination means for determining a composition for causing the imaging device to shoot a subject based on the adoption frequency of each of the plurality of types of compositions set by the setting means; and having the demand information is based on the image purchase information by the user, a control device characterized by this.

2. The control device according to claim 1, wherein the setting means sets a higher adoption frequency for a type of composition with a higher demand in the demand information.

3. The setting means sets a higher adoption frequency for a type of composition having a first demand level in the demand information than for a type of composition having a second demand level lower than the first demand level in the demand information. The control device according to claim 1, characterized by this.

4. Each of the plurality of types of compositions has a different occupancy rate of the subject area in the captured image, the demand information indicates the height of the user's demand for each occupancy rate of the subject area in the image purchased by the user The control device according to claim 1, characterized by this.

5. Each of the plurality of types of compositions has a different position of the subject area in the captured image, the demand information indicates the height of the user's demand for each position of the subject area in the image purchased by the user The control device according to claim 1, characterized by this.

6. The control device according to claim 4, wherein the subject area in the captured image is a face area of a person.

7. The demand information is information configured based on the composition of the subject related to the image purchased by the user The control device according to claim 1, characterized by this.

8. The control device according to claim 7, wherein the demand information is a learning result obtained by machine learning of the demand trend of the composition of the subject based on the image purchased by the user.

9. identification means for identifying a user; detection means for detecting a subject associated with the user identified by the identification means as a target subject; further having The first acquisition means acquires the demand information associated with the user identified by the identification means. The determination means determines the composition for causing the imaging for the target subject detected by the detection means. The control device according to claim 1, characterized in that.

10. The demand information is information configured based on the composition of the subject in a plurality of images determined to have high demand based on the operations of a plurality of users among a plurality of images taken in a predetermined event. The control device according to claim 1, characterized in that.

11. The control device according to claim 1, further comprising a changing means for changing the imaging parameters of the imaging device based on the composition for causing the imaging determined by the determination means.

12. A second acquisition means for acquiring an imaging image imaged by the imaging device with respect to the composition for causing the imaging; Control means for controlling whether or not to cause the imaging device to perform imaging based on the imaging image acquired by the second acquisition means; The control device according to claim 1, further comprising.

13. Imaging means; The control device according to any one of claims 1 to 12; Means for causing the imaging means to capture an image for recording based on the information on the composition for causing the imaging determined by the control device. An imaging device, characterized in that it has.

14. A control method for a control device that controls the operation of an imaging device that acquires images sold to users by automatic shooting with changed compositions, A first acquisition step of acquiring demand information indicating the user's demand tendency regarding the composition of the subject; A setting step of setting the adoption frequency for shooting for each of a plurality of predetermined types of compositions based on the demand information acquired in the first acquisition step; A determination step of determining the composition for causing the imaging device to capture the subject based on the adoption frequency of each of the plurality of types of compositions set in the setting step; having, The control method, characterized in that the demand information is based on the purchase information of images by the user.

15. A program for causing a computer to function as each means of the control device according to any one of claims 1 to 12.

16. An imaging system that sells to a user an image automatically captured by an imaging device with a changed composition, the system including the imaging device, a control device that controls the operation of the imaging device, and a collection server that collects the user's demand tendency based on the image purchased by the user. The control device has an acquisition means for acquiring demand information indicating the user's demand tendency regarding the composition of the subject, a setting means for setting the adoption frequency for shooting for each of a plurality of predetermined types of compositions based on the demand information acquired by the acquisition means, and a determination means for determining a composition for causing the imaging device to shoot a subject based on the adoption frequency of each of the plurality of types of compositions set by the setting means. The imaging system having the above features is characterized in that the demand information is based on the purchase information of the image by the user.

Images