Imaging control device, imaging control method, program, and imaging equipment
The imaging control device recognizes and selects the target subject and automatically adjusts the imaging parameters, which solves the problem of difficulty in operating the imaging equipment by users and improves the quality and convenience of video shooting.
Patent Information
- Application Number
- CN202080060004.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-03
- Filing Date
- 2020-06-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-06-12
AI Technical Summary
In video publishing websites and social network services, it is difficult for users to operate the imaging device correctly, especially the automatic function, resulting in poor video shooting quality.
Through the imaging control device, the introduction object and the object introducer are identified based on the captured image data, and the appropriate target subject is selected, and imaging operations such as automatic focus control and exposure control are performed to ensure the quality of video shooting.
It realizes that when the user does not directly operate the imaging device, automatically adjusts the imaging parameters, improves the quality and convenience of video shooting, and adapts to different shooting scenes.
Smart Images

Figure CN114342350B_ABST
Abstract
Description
Technical Field
[0001] The present technology relates to an imaging control apparatus, an imaging control method, a program, and an imaging device, and more particularly, to a technology for imaging control suitable for a subject. Background Art
[0002] There are known technologies for performing various types of processing related to imaging, such as focus control of a moving image captured by an imaging device.
[0003] Citation List
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-33013 Summary of the invention
[0006] Problems to be solved by the present invention
[0007] Meanwhile, nowadays, it is popular among users to post videos shot by themselves with imaging devices such as digital cameras or smartphones on video posting sites, social network services (SNS), and the like.
[0008] In such an environment, the user himself is often the subject. It is difficult for the user to operate the imaging device by himself, so it is difficult to properly adjust the operation of the imaging device. Then, automatic functions such as autofocus are often used, but sometimes it is difficult to operate the automatic function correctly.
[0009] Then, the present disclosure proposes a technology that allows imaging-related functions to be properly operated.
[0010] Solutions to the Problem
[0011] According to the present technology, the imaging control device includes an identification unit, which identifies an introduction object and an object introducer who introduces the introduction object as subjects, respectively, based on captured image data obtained by an imaging unit of an imaging device; a selection unit, which selects one of the introduction object and the object introducer as a target subject based on a positional relationship between the introduction object, the object introducer and at least any two of the imaging device; and an imaging control unit, which performs imaging control suitable for the subject selected as the target subject by the selection unit.
[0012] The introduction object is, for example, an article, a product, etc., which is a photographic subject. For example, the object introducer is a person, which is a photographic subject. Then, based on the positional relationship between any two of the introduction object, the object introducer, and the imaging device, the imaging control is determined by estimating which one of the introduction object and the object introducer is the main photographic subject.
[0013] As imaging control, for example, auto focus control, automatic exposure (AE) control (aperture control, shutter speed (SS) control, and gain control), etc. are assumed as control of the imaging operation of the imaging unit. In addition, as imaging control, control of image processing of captured image data is also considered. For example, signal processing control such as white balance processing and contrast adjustment processing is also assumed.
[0014] Note that the positional relationship between at least any two of the introduced object, the object introducer and the imaging device is assumed to be the positional relationship between the object introducer and the introduced object, the positional relationship between the introduced object and the imaging device, the positional relationship between the imaging device and the object introducer, the positional relationship between the object introducer, the introduced object and the imaging device, and the like.
[0015] In the above-mentioned imaging control device according to the present technology, the selection unit may select one of the introduction object and the object introducer as the target subject based on a positional relationship between the introduction object and the object introducer.
[0016] There is a case where a scene or situation can be estimated based on the positional relationship between an introduction object and an object introducer, thereby utilizing it to select a target subject.
[0017] In the above-mentioned imaging control device according to the present technology, the selection unit may select one of the introduction object and the object introducer as the target subject based on the positional relationship between the introduction object and the imaging device.
[0018] There is a case where a scene or situation can be estimated based on the positional relationship between an introduction object and an imaging device, and thus utilized to select a target subject.
[0019] In the above-mentioned imaging control device according to the present technology, the identification unit may identify the introduction object by recognizing the introduction object based on the captured image data.
[0020] That is, the items to be introduced are directly identified through image recognition.
[0021] In the above-mentioned imaging control device according to the present technology, the identification unit may recognize a hand of the object introducer based on the captured image data, and identify the introduction object based on the recognition result of the hand.
[0022] For example, even if the introduced object cannot be directly identified or is not directly identified, the introduced object can be indirectly identified based on the hand recognition result.
[0023] In the above-mentioned imaging control device according to the present technology, the identification unit may virtually identify the hand as the introduction object as a substitute for the real object to be introduced.
[0024] For example, in the case where a hand is assumed to hold an introduction object, the introduction object can be identified by recognizing the hand.
[0025] In the above-mentioned imaging control device according to the present technology, the identification unit may identify the introduction object based on the state of the hand.
[0026] For example, the introduction object is identified based on the fact that a hand, which is a part of the body of the object introducer, is in a state such as holding, pinching, or grasping an object.
[0027] In the above-mentioned imaging control device according to the present technology, the selection unit can select one of the introduction object and the object introducer as the target subject based on the introduction object and the positional relationship between the introduction object and the object introducer depending on the state of the object introducer's hand.
[0028] In this case, the state of the hand is also a state in which the hand as a part of the object presenter's body, for example, holds, pinches, or grasps an object, etc. Based on the positional relationship accompanying these states, target subject selection is performed.
[0029] In the above-mentioned imaging control device according to the present technology, the state of the hand may be a state in which the hand of the object introducer is in contact with the introduction object.
[0030] Based on the fact that the hand is in contact with the article or the like, the article or the like is identified as an introduction object.
[0031] In the above-mentioned imaging control device according to the present technology, the state of the hand may be a state in which a hand of an object introducer points to the introduction object.
[0032] Based on the fact that the hand is pointing to the object or the like, the object or the like is identified as the introduction object.
[0033] In the above-mentioned imaging control device according to the present technology, the selection unit can select one of the introduction object and the object introducer as the target subject based on the distance relationship as a positional relationship between at least any two of the introduction object, the object introducer and the imaging device.
[0034] The positional relationship can be regarded as a distance relationship between them. In this case, the imaging control is determined by estimating which one of the introduction object and the object introducer is the main subject based on the distance relationship, its change, etc.
[0035] In the above-mentioned imaging control apparatus according to the present technology, the distance relationship may be a distance between the introduction object and the imaging device.
[0036] There is a case where a scene or situation can be estimated based on the distance relationship between an introduction object and an imaging device, thereby utilizing it to select a target subject.
[0037] In the above-mentioned imaging control device according to the present technology, the distance relationship may be a distance between the object introducer and the introduced object.
[0038] There is a case where a scene or situation can be estimated based on the distance relationship between an introduction object and an object introducer, thereby utilizing it to select a target subject.
[0039] In the above-mentioned imaging control apparatus according to the present technology, the distance relationship may be a distance between the object introducer, the introduction object, and the imaging device.
[0040] There is a case where a scene or situation can be estimated based on the respective distance relationships between an introduction object, an object introducer, and an imaging device, and thus utilized to select a target subject.
[0041] In the above-mentioned imaging control device according to the present technology, the selection unit may detect the distance relationship based on a ratio of a region of at least one of the introduction object or the object introducer to an entire frame of the captured image data.
[0042] For example, based on the fact that the ratio of the object to be introduced on the captured image has become larger than a predetermined value, the introduction object is determined as the target subject, and imaging control is performed.
[0043] The above-mentioned imaging control device according to the present technology may further include a presentation control unit which performs presentation control of presenting a difficult control state in which imaging control is difficult to the subject introducer when the distance between the imaging device and the subject of introduction is less than a predetermined value.
[0044] For example, the user may be notified of a state in which a subject is too close to be properly imaged.
[0045] The above-mentioned imaging control device according to the present technology may further include an association control unit that performs association control of associating metadata related to the selection result of the selection unit with the captured image data.
[0046] For example, metadata may be used at a later point in time (eg, at the time of reproduction) to determine which is the target subject that is the target of imaging control.
[0047] The imaging device of the present technology includes an imaging unit and the above-mentioned imaging control device. For example, an information processing device in the imaging device is used as the imaging control device.
[0048] The imaging method of the present technology is an imaging method including: identification processing for identifying an introduction object and an object introducer who introduces the introduction object as subjects, respectively, based on captured image data obtained by an imaging unit of an imaging device; selection processing for selecting one of the introduction object and the object introducer as a target subject, based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; and imaging control processing for performing imaging control suitable for the subject selected as the target subject in the selection unit. This allows the subject that should be subjected to imaging control to be appropriately set at the time of imaging.
[0049] The program according to the present technology is a program that causes the imaging control device to execute processing corresponding to the imaging control method as described above. This allows the above-mentioned imaging control device to be realized by an information processing device, a microcomputer, or the like. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is an explanatory diagram of the equipment used in the embodiment of the present technology.
[0051] Figure 2 is a block diagram of an imaging device according to an embodiment.
[0052] Figure 3 is a block diagram of a computer device according to an embodiment.
[0053] Figure 4 1 is a first explanatory diagram of scene determination according to the first embodiment.
[0054] Figure 5 This is a second explanatory diagram of scene determination according to the first embodiment.
[0055] Figure 6 This is a first explanatory diagram of scene determination according to the second embodiment.
[0056] Figure 7 2 is a second explanatory diagram of scene determination according to the second embodiment.
[0057] Figure 8 This is a first explanatory diagram of scene determination according to the third embodiment.
[0058] Fig. 9 is a second explanatory diagram of scene determination according to the third embodiment.
[0059] Fig.10 3 is a third explanatory diagram of scene determination according to the third embodiment.
[0060] Fig.11 It is a fourth explanatory diagram of scene determination according to the third embodiment.
[0061] Fig.12This is a first explanatory diagram of scene determination according to the fourth embodiment.
[0062] Fig.13 This is a second explanatory diagram of scene determination according to the fourth embodiment.
[0063] Fig.14 is a flowchart of a processing example according to each embodiment.
[0064] Fig.15 is a diagram showing an example of imaging control according to the embodiment.
[0065] Fig.16 is a diagram showing an example of imaging control according to the embodiment.
[0066] Fig.17 is a flowchart of a processing example according to the first embodiment.
[0067] Fig.18 is a flowchart of a processing example according to the second embodiment.
[0068] Fig.19 is a flowchart of a processing example according to the third embodiment.
[0069] Fig. 20 is a flowchart of a processing example according to a modification of the third embodiment.
[0070] Fig.21 is a flowchart of a processing example according to the fourth embodiment. DETAILED DESCRIPTION
[0071] Hereinafter, the embodiments will be described in the following order.
[0072] <1. Configuration of equipment usable as imaging control device>
[0073] <2. Configuration of imaging equipment>
[0074] <3. Overview of Scene Determination and Imaging Control>
[0075] <4. Processing to implement the embodiment>
[0076] <5. Summary and revision>
[0077] Note that the same reference numerals are used to denote the contents and structures that have been described before to omit explanation.
[0078] The present technology identifies an introduction object and an object introducer who introduces the introduction object based on the inside of captured image data obtained by capturing a moving image. Then, for example, based on the positional relationship between at least any two of the introduction object, the object introducer, and the imaging device, one of the introduction object and the object introducer is selected as a target subject. Imaging control suitable for the area of the subject selected as the target subject is performed.
[0079] As an example, the present embodiment describes an imaging device that performs appropriate imaging control on an image area of an introduced object or an introducer of an object when shooting a video to be posted on a video posting site, SNS, or the like.
[0080] Here, as an example of a video to be posted, a product review video in which a video poster introduces a product will be described as an example.
[0081] Then, in the product review video, the product to be introduced and the object introducer who introduces the product are imaged by an imaging device. The object introducer is mostly a video publisher, who is the person who introduces the product himself in the product review video.
[0082] In the present embodiment, a case is assumed where the object introducer shoots a product review video by shooting a video of himself with a fixed imaging device.
[0083] Here, the object to be introduced refers to an article. In this embodiment, a product will be described as an example of an article. Note that the article mentioned here is not necessarily an object of commercial transaction, but may be, for example, a work created by him or her himself.
[0084] Furthermore, in the embodiment, it is assumed that the product review video includes a performance scene and a product introduction scene.
[0085] The performance scene is a scene for imaging the performance of the object introducer (such as the object introducer's self-introduction and the overview of the product). The product introduction scene is a scene for imaging the shape and operation of the product, the way to actually use the product, etc.
[0086] The imaging device according to the present embodiment determines whether the product review video is a performance scene or a product introduction scene based on the positional relationship between at least any two of, for example, the introduction object, the object introducer, and the imaging device, and selects a target subject according to each scene. Then, imaging control suitable for the selected target subject is performed.
[0087] <1. Configuration of devices applicable as imaging control devices>
[0088] An example in which the imaging control device according to the present disclosure is realized mainly by an imaging device is described below. The imaging control device according to an embodiment of the present disclosure takes a form incorporated in various devices (particularly, an imaging device).
[0089] Figure 1 An example of a device which may be the imaging device 1 is shown.
[0090] The device that may be the imaging device 1 is a device having a function of capturing moving images, and may be a digital video camera 1A, a digital still camera 1B, a mobile terminal 1C such as a smartphone, etc. For example, an imaging control apparatus is incorporated in the above-described imaging device 1 .
[0091] Note that the imaging device 1 is not limited to the above examples and may be any device that may include an imaging control device or any device controlled by an imaging control device, and may be various other types of devices. In addition, the imaging control device may be provided as a separate body instead of being incorporated in the imaging device 1.
[0092] In the imaging apparatus 1 , a microcomputer or the like within the imaging apparatus 1 performs imaging control.
[0093] Imaging control refers to imaging control of the imaging device 1, and is, for example, imaging operation control (which is control of the optical system and light receiving operation for focusing the subject light onto the image sensor (imaging element) of the imaging unit) and captured image processing control (which is control of signal processing of captured image data).
[0094] As for the imaging operation control, for example, automatic focus control, AE control (aperture control, SS control, and gain control), zoom control, and the like are assumed.
[0095] Furthermore, as for the captured image processing control, for example, white balance processing control, contrast adjustment processing control, image effect processing control, and the like are assumed.
[0096] In the imaging device 1, light receiving and imaging operations and image signal processing are performed according to imaging operation control and captured image processing control, and a captured image is output. That is, the captured image is recorded in a recording medium and displayed on a display unit.
[0097] As described above, various devices are used as the imaging control apparatus according to the embodiment. An example in which the imaging apparatus 1 as a digital camera 1A is provided as the imaging control apparatus of the present disclosure is described below.
[0098] <2. Configuration of imaging equipment>
[0099] Will refer to Figure 2 A configuration example of a digital video camera 1A as the imaging apparatus 1 is described.
[0100] like Figure 2 As shown, the imaging device 1 includes an optical system 11, a driver unit 12, an imaging unit 13, a camera signal processing unit 16, a recording control unit 17, a presentation unit 18, an output unit 19, an operation unit 20, a camera control unit 21, a memory unit 23 and a sensor unit 24.
[0101] The optical system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The optical system 11 guides light (incident light) from a subject and focuses the light on the imaging unit 13 .
[0102] The driver unit 12 is equipped with, for example, a motor driver for a zoom lens driving motor, a motor driver for a focus lens driving motor, a motor driver for an aperture mechanism driving motor, a shutter driver for a shutter driving motor, and the like.
[0103] According to instructions from the camera control unit 21 and the camera signal processing unit 16 , the driver unit 12 applies drive current to the corresponding drivers to cause them to perform movement of the focus lens and zoom lens, opening and closing of the aperture blades of the aperture mechanism, shutter operation, and the like.
[0104] The aperture mechanism is driven by an aperture mechanism driving motor to control the amount of light incident on the imaging unit 13 described later. The focus lens is driven by a focus lens driving motor for focus adjustment. The zoom lens is driven by a zoom lens driving motor for zoom adjustment. The shutter mechanism is driven by a shutter driving motor to perform a shutter operation.
[0105] The imaging unit 13 includes an image sensor 14 (imaging element) of, for example, a complementary metal oxide semiconductor (CMOS) type, a charge coupled device (CCD) type, etc. The image sensor 14 includes imaging pixels for capturing an image of a subject and image plane phase difference pixels for detecting a phase difference in an optical image of the subject. Note that the image sensor 14 may not include phase difference pixels.
[0106] The imaging unit 13 performs, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc., and further performs analog-to-digital (A / D) conversion processing on an electric signal obtained by photoelectrically converting light received by the image sensor 14. The imaging unit 13 outputs the captured image signal to the camera signal processing unit 16 and the camera control unit 21 as digital data.
[0107] The image sensor 14 includes a plurality of imaging pixels, each of which stores electric charge corresponding to the intensity of received light.
[0108] The image sensor 14 may be covered by, for example, a Bayer array color filter.The captured image signal may be read from an electrical signal obtained by photoelectrically converting light received by the imaging pixel group.
[0109] The image sensor 14 outputs a captured image signal to the camera signal processing unit 16 and the camera control unit 21 .
[0110] The image sensor 14 may include an image plane phase difference pixel. The image plane phase difference pixel detects phase difference information. The image plane phase difference pixel detects a pair of phase difference signals. The imaging unit 13 outputs a pair of phase difference signals detected by the image plane phase difference pixel. For example, the phase difference signal is used in a correlation calculation for calculating the distance from the imaging device 1 to the introduction object or the object introducer.
[0111] Note that the image sensor 14 does not necessarily have image plane phase difference pixels. In order to calculate the distance from the imaging device 1 to the product to be introduced or the object introducer, a dedicated phase difference sensor, a time-of-flight (TOF) sensor, or the like arranged separately from the image sensor 14 may be used. In addition, for the distance from the imaging device 1 to the product or object introducer, a value corresponding to the distance may be determined instead of detecting the distance itself. For example, the size of the area of the product or object introducer in the captured image (the number of pixels included in the area), position information about the focusing lens, and the like are information that indirectly indicates the distance to the imaging device 1.
[0112] The imaging unit 13 outputs the phase difference signal to the camera signal processing unit 16 and the camera control unit 21 .
[0113] The camera signal processing unit 16 is configured as an image processing processor by, for example, a digital signal processor (DSP) or the like.
[0114] The camera signal processing unit 16 performs various types of signal processing on the digital signal (captured image signal) from the imaging unit 13. For example, the camera signal processing unit 16 performs preprocessing, synchronization processing, YC generation processing, various types of correction processing, resolution conversion processing, codec processing, and the like.
[0115] In the preprocessing, a pinching process of pinching the black levels of R, G, and B to a predetermined signal level, a correction process between R, G, and B color channels, and the like are performed on the captured image signal from the imaging unit 13 .
[0116] In the synchronization process, color separation processing is performed so that image data on each pixel includes all color components of R, G, and B. For example, for an imaging element using a Bayer array color filter, demosaicing is performed as the color separation processing.
[0117] In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data.
[0118] In the resolution conversion process, the resolution conversion process is performed on the image data before various types of signal processing are performed or after signal processing is performed.
[0119] In the codec processing in the camera signal processing unit 16, for example, encoding processing for recording and communication and file generation are performed on the image data subjected to the above-mentioned various types of processing. For example, the image file MF is generated in the MP4 format or the like for recording video and audio conforming to MPEG-4. In addition, as a still image file, a file may be generated in a format such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Graphics Interchange Format (GIF).
[0120] Note that the camera signal processing unit 16 performs a process of adding metadata transmitted from the camera control unit 21 to an image file.
[0121] The metadata includes parameters of various types of processing in the camera signal processing unit 16 and detection information obtained by the sensor unit 24 described later, and is added to, for example, each corresponding frame constituting a video, the entire corresponding video, or a corresponding predetermined single such as a scene unit.
[0122] In the present embodiment, the camera control unit 21 (imaging control means 22) performs control suitable for identifying product introduction scenes and performance scenes as described below. It is assumed that metadata related thereto is also generated and added to the image file.
[0123] Specifically, information indicating whether each frame is a product introduction scene or a performance scene, information indicating whether the scene is successfully recognized or not recognized, information on whether the object to be introduced and the object introducer are identified, information indicating the area of the introduced object and the object introducer identified in the image, an error flag (refer to later Fig.18 etc. description) etc. can be added as metadata.
[0124] Note that although an example in which the camera signal processing unit 16 performs metadata adding processing is described here, an example in which the recording control unit 17 or the output unit 19 performs metadata adding processing is also possible.
[0125] Despite Figure 2 The audio processing system is not shown in the figure, but actually includes an audio recording system and an audio processing system. The image file may include audio data as well as image data as video.
[0126] In the case of performing audio recording, an audio signal input from an audio input unit such as a microphone (not shown) is converted into a digital audio signal in an audio processing system and then sent to the camera control unit 21. The camera control unit 21 performs control to record the digital audio signal in a recording medium such as a nonvolatile memory in association with an image signal.
[0127] The recording control unit 17 performs recording and reproduction on a recording medium such as a nonvolatile memory. The recording control unit 17 performs processing such as recording image files of moving image data, still image data, etc., thumbnails, etc. on the recording medium.
[0128] Note that the recording control unit 17 may be provided in the imaging control device 22 .
[0129] The recording control unit 17 may be in various practical forms. For example, the recording control unit 17 may be configured as a flash memory and its writing and reading circuits built into the imaging device 1, or may be in the form of a card recording and reproducing unit that performs recording and reproducing access to a recording medium (e.g., a memory card such as a portable flash memory) removably installed in the imaging device 1. In addition, it may be implemented as a hard disk drive (HDD) or the like as a form incorporated in the imaging device 1.
[0130] The presentation unit 18 includes a display unit that performs various displays for person photography. The display unit is, for example, a display panel or a viewfinder of a display device such as a liquid crystal display (LCD) or an organic electroluminescent (EL) display arranged in a housing of the imaging device 1.
[0131] The presentation unit 18 also includes an audio output unit such as a speaker. The digital audio signal read by the camera control unit 21 is converted into an audio signal by the camera signal processing unit 16 and then output by the audio output unit.
[0132] The display unit in the presentation unit 18 performs various displays on the display screen based on the instructions from the camera control unit 21. For example, captured image data whose resolution has been converted for display by the camera signal processing unit 16 is provided. In response to the instructions from the camera control unit 21, the display unit performs display based on the captured image data. Thus, so-called through images (subject monitoring images) which are images captured during standby or recording are displayed.
[0133] Furthermore, the display unit displays a reproduced image of the captured image data read from the recording medium in the recording control unit 17 .
[0134] The display unit displays various operation menus, icons, messages, and the like, that is, performs display as a graphical user interface (GUI) on the screen based on an instruction from the camera control unit 21 .
[0135] The output unit 19 performs data communication and network communication with an external device through wires or radio.
[0136] For example, captured image data (still image file or moving image file) is transmitted and output to an external display device, a recording device, a reproducing device, or the like.
[0137] Furthermore, as a network communication unit, the output unit 19 can perform communication via various networks such as the Internet, a home network, and a local area network (LAN) to send and receive various data to and from a server, a terminal, and the like on the network.
[0138] The operation unit 20 collectively indicates an input device for a user to perform various operation inputs. Specifically, the operation unit 20 indicates various operation elements (such as keys, dials, touch panels, and touch pads) provided at a housing of the imaging device 1.
[0139] The operation unit 20 detects an operation by a user, and transmits a signal corresponding to the input operation to the camera control unit 21 .
[0140] The camera control unit 21 is constituted by a microcomputer (arithmetic processing unit) including a central processing unit (CPU).
[0141] The memory unit 23 stores information and the like used in processing of the camera control unit 21. The memory unit 23 shown in the figure collectively indicates, for example, a read only memory (ROM), a random access memory (RAM), a flash memory, and the like.
[0142] The memory unit 23 may be a memory area built in a microcomputer chip as the camera control unit 21 , or may be constituted by a separate memory chip.
[0143] The camera control unit 21 executes a program stored in a ROM, a flash memory, or the like of the memory unit 23 to control the entire imaging device 1 .
[0144] For example, the camera control unit 21 controls the operation of the units required to control the shutter speed of the imaging unit 13, the indication of various types of signal processing in the camera signal processing unit 16, the acquisition of lens information, the imaging operation and the recording operation according to the user's operation, the start and end control of the video recording, the operation of reproducing the recorded image file, the camera operation (such as zoom, focus and exposure adjustment in the lens barrel), the user interface operation, etc.
[0145] The RAM in the memory unit 23 is used to temporarily store data, programs, and the like as a work space during various types of data processing by the CPU of the camera control unit 21 .
[0146] The ROM and flash memory (nonvolatile memory) in the memory unit 23 are used to store an operating system (OS) for the CPU to control each unit, content files such as image files, application programs for various operations, firmware, and the like.
[0147] The camera control unit 21 has a function as an imaging control device 22. It is assumed that the imaging control device 22 has functions as, for example, an identification unit 22a, a selection unit 22b, an imaging control unit 22c, a presentation control unit 22d, and an association control unit 22e. These functions are realized by software (application) in the camera control unit 21 which is a microcomputer or the like.
[0148] The identification unit 22a performs processing for identifying a product as a subject and an object introducer who introduces the object to be introduced based on the inside of the captured image data obtained by the imaging unit 13 of the imaging device 1. For example, the identification unit 22a identifies the faces of the product and the object introducer by performing analysis processing on the acquired captured image data.
[0149] The identification of the product mentioned here includes not only selecting the product to be introduced from the products detected from the subject appearing in the captured image data, but also estimating the position of the product based on, for example, the position or state of the hand of the subject introducer.
[0150] The selection unit 22b selects one of the product and the object introducer as the target subject based on, for example, the positional relationship between any two of the product to be introduced, the object introducer, and the imaging device 1. More specifically, for the selection of the target subject, the selection unit 22b performs scene determination, that is, determines whether the scene of the video currently being shot is a performance scene or a product introduction scene, and selects one of the product and the object introducer as the target subject based on the determined scene.
[0151] In the present disclosure, the positional relationship between the product, the object introducer, and the imaging device 1 is referred to as the object positional relationship. For example, the object positional relationship is determined based on the distance between the imaging device 1 and the product, the distance between the product and the object introducer, the distance between the imaging device 1 and the object introducer, etc.
[0152] Note that the distance may not be the distance itself. A value related to the distance may be used. For example, the ratio of the area of the product or object introducer to the entire frame of the captured image may be used as a value corresponding to the distance. In addition, information about the position of the focusing lens, etc. may be used as information corresponding to the distance.
[0153] The imaging control unit 22c performs imaging control suitable for the area of the subject selected as the target subject by the selection unit 22b. For example, the imaging control unit 22c performs imaging operation control (such as autofocus control) and captured image processing control (such as white balance processing control) suitable for the area of the target subject.
[0154] When the distance between the imaging device 1 and the product is less than a predetermined value, the presentation control unit 22d performs presentation control to present a difficult control state in which the imaging control is difficult to the object introducer. For example, depending on the conditions, the presentation control unit 22d can control the execution of the output of a message, an icon, etc. on the display unit in the presentation unit 18, the lighting or flashing of a warning light, etc.
[0155] The association control unit 22e performs association control to associate metadata related to the selection result of the selection unit 22b with the captured image data.
[0156] The metadata related to the result of the selection by the selection unit 22b may be, for example, information indicating whether it is a product introduction scene or a performance scene, information indicating whether the scene is successfully recognized or not, information about whether the introduction object and the object introducer are identified, information indicating the areas of the introduction object and the object introducer identified in the image, information about the target subject (information about which one is selected, the object introducer 60 or the product 70), information about what the object to be introduced or the target subject is (e.g., the type of item), an error flag indicating a determination impossible state (refer to later Fig.18 etc. description) and so on.
[0157] For example, the association control unit 22 e transmits such metadata associated with the selection result of the selection unit 22 b to the camera signal processing unit 16 , so that the metadata associated with the selection result of the selection unit 22 b is included in the image file in the camera signal processing unit 16 .
[0158] That is, the association control unit 22 e supplies information to the camera signal processing unit 16 , for example, frame by frame, according to the results of the scene determination and target subject selection by the selection unit 22 b .
[0159] The association control unit 22e provides metadata in such a manner that the camera signal processing unit 16 performs processing to add the metadata to the image file, for example, so that the metadata associated with the result of the selection by the selection unit 22b is recorded in the same recording medium as the captured image data, or is put into the same file to be recorded and transmitted. Of course, being configured as a metadata file separate from the image file, each piece of metadata can be associated with the image file and the frame of the captured image data in the image file.
[0160] As a result, metadata related to the selection result of the selection unit 22b (for example, metadata indicating a product introduction scene or a performance scene) is associated with the captured image data frame by frame.
[0161] Note that metadata related to the result of the selection by the selection unit 22b may be associated with the captured image data, for example, on a scene-by-scene basis, rather than on a frame-by-frame basis.
[0162] The sensor unit 24 collectively indicates various sensors mounted on the imaging device 1. As the sensor unit 24, for example, a position information sensor, an illuminance sensor, an acceleration sensor, and the like are mounted.
[0163] The processing for realizing the present technique is executed by the digital video camera 1A including the imaging control device 22 having the above-described functions.
[0164] Meanwhile, the control processing performed by the imaging control device 22 described later can be implemented not only in the digital camera 1A but also in other devices such as Figure 1 Therefore, a configuration example of the mobile terminal 1C will also be described.
[0165] The mobile terminal 1C may be configured to have, for example, Figure 3 The configuration of the computer device 30 is shown.
[0166] exist Figure 3 In the computer device 30, a central processing unit (CPU) 31 performs various types of processing according to a program stored in a read-only memory (ROM) 32 or a program loaded from a storage unit 39 into a random access memory (RAM) 33. In the RAM 33, data and the like required for the CPU 31 to perform various types of processing are also appropriately stored. The CPU 31 is equipped with the same functional configuration as the above-mentioned imaging control device 22 through, for example, an application program.
[0167] The CPU 31, the ROM 32, and the RAM 33 are interconnected via a bus 34. To the bus 34, an input-output interface 35 is also connected.
[0168] An input unit 36 , an imaging unit 37 , an output unit 38 , a storage unit 39 , and a communication unit 40 are connected to the input-output interface 35 .
[0169] The input unit 36 includes a keyboard, a mouse, a touch panel, and the like.
[0170] The imaging unit 37 includes a lens system including an imaging lens, an aperture, a zoom lens, a focusing lens, etc., a driving system for causing the lens system to perform focusing operations and zooming operations, a solid-state imaging element array that detects captured image light obtained by the lens system and generates a captured image signal by performing photoelectric conversion, etc.
[0171] The output unit 38 includes a display such as a liquid crystal display (LCD), a cathode ray tube (CRT), or an organic electroluminescence (EL) panel, a speaker, and the like.
[0172] For example, the output unit 38 performs display of images for various types of image processing, moving images to be processed, etc. on the display screen based on the instructions from the CPU 31. In addition, the output unit 38 displays various operation menus, icons, messages, etc., that is, performs display as a graphical user interface (GUI) based on the instructions from the CPU 31.
[0173] The storage unit 39 includes a hard disk drive (HDD), a solid-state memory, or the like, and stores various types of information.
[0174] The communication unit 40 performs communication processing via a transmission line such as the Internet, and performs communication with various devices through wired / wireless communication, bus communication, and the like.
[0175] To the input-output interface 35 , a drive 41 is also connected as necessary, in which a removable recording medium 42 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.
[0176] The drive 41 allows data files such as image files, various computer programs, and the like to be read from the removable recording medium 42. The read data files are stored in the storage unit 39, and the video and audio included in the data files are output by the output unit 38. In addition, the computer program and the like read from the removable recording medium 42 are installed in the storage unit 39 as necessary.
[0177] For example, the computer device 30 allows software for executing processing as the imaging control apparatus of the present disclosure to be installed via network communication through the communication unit 40 or the removable recording medium 42. Alternatively, the software may be stored in advance in the ROM 32, the storage unit 39, or the like.
[0178] Note that the computer device 30 is not limited to Figure 3 The plurality of computer devices may be configured not individually as in the example above, but by systematizing a plurality of computer devices. The plurality of computer devices may include computer devices as a server group (cloud) that can be used by a cloud computing service.
[0179] <3. Summary of scene determination>
[0180] Hereinafter, an outline of scene determination in the present technology will be described. Here, different types of imaging control are performed on imaging of the imaging device 1 according to each scene. In the present embodiment, switching of a focus control target will be mainly described as an example of imaging control.
[0181] In recent years, the popularity of video posting sites and SNS has made it easy for individuals to post videos they have shot, and the number of publishers of product review videos that introduce products through videos shot by individuals has also increased.
[0182] In many cases, such product review videos mainly include: performance scenes for imaging the performance of the object introducer, such as the object introducer's self-introduction and product overview, which are usually performed before or after the product introduction scene; and product introduction scenes for imaging the shape and operation of the product, the way to actually use the product, etc.
[0183] In a performance scene, it is desirable to perform focus control on an object introducer who introduces a product. In a product introduction scene for explaining a specific product, it is desirable to perform focus control on the product so that the shape of the product, etc. can be easily seen.
[0184] However, in the case of personally shooting a product review video, the object introducer who introduces the product often performs the operation of shooting the video by himself, that is, self-shooting, and performs shooting with the imaging device 1 still. Therefore, it is difficult for the object introducer to switch the focus control target by performing the operation of changing the focus control target according to the imaging scene during shooting.
[0185] Furthermore, in actual video shooting, in addition to the product, an object introducer etc. also appears, making it impossible for the imaging device 1 side to identify which one is the product to be introduced, resulting in the problem that focus control focusing on the product that should be the focus control target is not performed in the product introduction scene.
[0186] Therefore, in the present technology, it is determined which scene the captured image being captured is, ie, a performance scene or a product introduction scene, a subject suitable for each scene is identified, and then imaging control such as focus control is performed.
[0187] Will refer to Figures 4 to 13 Describes the outline of a specific focus control.
[0188] Figure 4 and 5 as well as Figure 6 and Figure 7 The distance relationship between the introduction object, the object introducer and the imaging device is shown. In addition, Figures 8 to 13 A captured image display screen 50 displayed as a through image on a display unit when shooting a product review video is shown. An object introducer 60 introducing a product 70 and a product to be introduced 70 are displayed on the captured image display screen 50. The object introducer 60 has a hand 61 and a face 62 as a part of the body.
[0189] In addition, Figures 4 to 13, for convenience of explanation, the indicating hand 61, the face 62, and parts of the product 70 are illustrated as areas surrounded by dotted lines.
[0190] First, refer to Figure 4 and 5 A first embodiment is described. In the first embodiment, scene determination is performed based on a distance relationship generated from an object position relationship.
[0191] The first embodiment is an example of determining a scene based on the distance from the imaging device 1 to the product 70 , selecting a target subject suitable for each scene, and performing imaging control.
[0192] Note that when shooting a product review video, the product introducer is often located in front of the imaging device 1 and performs shooting at a specific position without moving. The present embodiment can be applied to such a case.
[0193] It can be considered that the distance Loc from the imaging device 1 to the product 70 varies with the scene. For example, when shooting a product review video, the object introducer 60 can bring the product 70 held in the hand 61 close to the imaging device 1 to explain the product 70 while highlighting it.
[0194] Therefore, based on the positional relationship between the product 70 and the imaging device 1 , in particular, the distance relationship represented by the distance Loc, subject selection based on scene determination is performed.
[0195] Note that the product 70 mentioned here is the introduction object identified by the imaging control device 22. In addition, in the case where the product 70 cannot be identified, the case where the hand 61 of the object introducer 60 is identified in place of the product 70 is also included. That is, in such a case, the real object to be introduced is the product 70, but the hand 61 of the object introducer 60 is identified as the introduction object instead of the product 70. This means that, for example, in the case where the product 70 is too small to be identified in the image, it is assumed that the product 70 is held in the hand 61, and the position of the product 70 in the image is identified by treating the hand as the product 70.
[0196] Figure 4 The case where the value of the distance Loc from the product 70 to the imaging device 1 is a value greater than the predetermined value Lth is shown. This is a state where the distance from the imaging device 1 to the product 70 is relatively long. Then, this is a state where the object introducer 60 does not bring the product 70 close to the imaging device 1 to make the product 70 stand out, and thus it is considered to be a performance scene where the object introducer 60 performs a performance.
[0197] Therefore, the imaging control device 22 selects the face 62 of the object introducer 60 who is performing as the target subject, and performs focus control for the face 62. This allows video shooting to make the audience pay attention to the object introducer 60 who is speaking in the performance scene.
[0198] Note that although the focus control is performed with the face 62 of the object introducer 60 as the target subject, the focus control may be performed with the eyes of the object introducer 60 as the target subject, for example.
[0199] On the other hand, when the value of the distance Loc from the product 70 to the imaging device 1 is smaller than Figure 5 In the case of the value of the predetermined value Lth1 in , it can be estimated that the object introducer 60 is in a state of bringing the product 70 close to the imaging device 1 , and it is considered to be a product introduction scene in which the object introducer 60 is introducing the product 70 .
[0200] In such a product introduction scene, the imaging control device 22 selects the product 70 as the target subject and performs focus control for the product 70. Therefore, focus control focusing on the product 70 to be introduced by the subject introducer 60 is performed, thereby allowing video shooting to make the audience pay attention to the product 70.
[0201] As described above, in the first embodiment, a scene is determined based on the distance from the imaging device 1 to the product 70, and focus control that focuses on a target subject suitable for each scene is performed.
[0202] Will refer to Figure 6 and 7 The second embodiment is described. The second embodiment also performs scene determination based on the distance relationship obtained from the object position relationship, but the second embodiment is an example of determining the scene based on the distance between the object introducer 60 and the product 70, selecting a target subject suitable for each scene, and performing imaging control.
[0203] As mentioned above Figure 4 and 5 As in the example, it is assumed that, for example, when shooting a product review video, the object introducer 60 brings the product 70 held in the hand 61 close to the imaging device 1 to explain the product 70 while making it stand out. This can also be regarded as Figure 6 and 7 That is, it can be considered that the distance Lho between the object introducer 60 and the product 70 changes with the scene.
[0204] Therefore, based on the positional relationship between the object introducer 60 and the product 70, in particular, the distance relationship represented by the distance Lho, subject selection based on scene determination is performed.
[0205] Figure 6 The case where the distance Lho is relatively small is shown.
[0206] In the imaging device 1, the distance Lhc from the imaging device 1 to the object introducer 60 and the distance Loc from the imaging device 1 to the product 70 can be measured, and thus the distance Lho between the object introducer 60 and the product 70 can be determined (Lho=Lhc-Loc).
[0207] exist Figure 6 In the case of , the distance Lho is smaller than the predetermined value Lth2. This is a state where the distance from the object introducer 60 to the product 70 is relatively short.
[0208] Then, this is a state in which the object introducer 60 does not bring the product 70 close to the imaging device 1 to make it protrude, and thus is regarded as a performance scene in which the object introducer 60 gives a performance.
[0209] Therefore, the imaging control device 22 selects the face 62 of the object introducer 60 giving the performance as the target subject, and performs focus control on the face 62 (or eyes, etc.). This allows video shooting to make the audience pay attention to the object introducer 60 speaking in the performance scene.
[0210] on the other hand, Figure 7 The case where the distance Lho is a value greater than the predetermined value Lth2 is shown. This is a state where the distance from the object introducer 60 to the product 70 becomes relatively long.
[0211] Then, this is a state in which the object introducer 60 brings the product 70 close to the imaging device 1 to make it protrude, and thus is regarded as a product introduction scene in which the object introducer 60 introduces the product 70 .
[0212] In such a product introduction scene, the imaging control device 22 selects the product 70 as the target subject and performs focus control for the product 70. Therefore, focus control focusing on the product 70 to be introduced by the subject introducer 60 is performed, thereby allowing video shooting to make the audience pay attention to the product 70.
[0213] As described above, in the second embodiment, the scene is determined based on the distance from the object introducer 60 to the product 70, and focus control suitable for a target subject appropriate for each scene is performed.
[0214] This second embodiment and the first embodiment are all based on the distance relationship execution scene determination. However, in the case where only the distance Lho between the object introducer 60 and the product 70 is used to perform determination as in the second embodiment, the following advantage is produced, that is, no matter how the movement (position) of the object introducer 60 is, determination can be performed.
[0215] That is, in the scene determination in the first embodiment, the object introducer 60 is required not to move (keep the same distance) relative to the imaging device 1. In the case where the object introducer 60 moves back and forth holding the product 70 in his hand, the scene determination may be inaccurate.
[0216] On the other hand, in the case of the second embodiment, determination is performed focusing only on the distance Lho between the object introducer 60 and the product 70 so that the accuracy of scene determination can be maintained even if the object introducer 60 moves back and forth.
[0217] By the way, the above is an example of focusing on the distance Lho and comparing the distance Lho with the predetermined value Lth2. However, the scene determination may be performed while focusing on the change in the difference between the distance Lho and the distance Loc. That is, the performance scene and the product introduction scene are determined based on whether the difference (or ratio) between the distance Lho and the distance Loc is greater than, equal to, or less than a predetermined value.
[0218] Furthermore, in the above-described example, the behavior of bringing the product 70 close to the imaging device 1 is assumed in the product introduction scenario, but sometimes it may be better to assume the opposite behavior.
[0219] That is, the object introducer 60 may take actions of keeping the product 70 away from him or her in the performance scene, and bringing the product 70 close to the object introducer 60 by, for example, holding the product 70 in the product introduction scene.
[0220] To cope with this behavior, the logic of scene determination can be reversed. For example, if the distance Lho is greater than the predetermined value Lth2, it is regarded as a performance scene, and if the distance Lho is less than or equal to the predetermined value Lth2, it is regarded as a product introduction scene.
[0221] For example, the user may be allowed to select which scenario determination logic to use.
[0222] Furthermore, for each distance, attention has been paid to the distance from the imaging device 1 in the depth direction (depth), but the distance between the object introducer 60 and the product in the vertical or horizontal direction may be considered.
[0223] The third embodiment will refer to Figures 8 to 11 The third embodiment is an example in which, based on the state of the hand 61 of the object introducer 60, in particular, the relationship between the hand 61 and the product 70, the product 70 is identified, and scene determination is performed.
[0224] Figure 8 and 9It is shown that based on the state of the hand 61 of the object introducer 60 (for example, holding the product 70), scene determination is performed using the positional relationship between the object introducer 60 and the product 70, and focus control suitable for the target subject selected depending on each scene is performed.
[0225] Here, the state of the hand 61 of the object introducer 60 is not limited to the state of holding the product 70, but includes various states of the hand 61 in contact with the product 70, such as pinching, grasping, and placing the product 70 on the hand.
[0226] In the object introducer 60 not as Figure 8 In the state shown of holding the product 70, it is considered as a performance scene where the object introducer 60 gives a performance.
[0227] Therefore, focus control is performed with the face 62 of the object introducer 60 as the target subject. This allows video shooting to make the audience pay attention to the object introducer 60 speaking in the performance scene.
[0228] On the other hand, in the object introducer 60 such as Fig. 9 In the state shown in which the product 70 is held, the product 70 can be identified and can be regarded as a scene in which the object introducer 60 specifically introduces the product 70, that is, a product introduction scene.
[0229] In the product introduction scene, specific aspects of the product 70, such as the shape of the product 70, will be explained, so that focus control and the like are performed with the product 70 as the target subject.
[0230] Therefore, focus control that focuses on the product 70 that the object introducer 60 is to introduce is performed, thereby allowing video shooting to make the audience pay attention to the product 70.
[0231] As described above, in the third embodiment, the scene is determined based on the state of the hand 61 of the object introducer 60 (eg, holding the product 70), and focus control focused on a target subject suitable for each scene is performed.
[0232] Furthermore, in the third embodiment, the product 70 can be identified and the scene can be determined based on the gesture of the hand 61 of the object introducer 60 (eg, pointing to the product 70) to perform imaging control suitable for the target subject selected according to each scene.
[0233] The gesture mentioned here is the state of the hand 61 of the object introducer 60 , and may be detected statically as a momentary state of the hand 61 , or may be detected dynamically as a movement of the hand 61 .
[0234] In the object introducer 60, no Fig.10In the state of the gesture shown, such as pointing to the product 70, it is considered that the object introducer 60 gives a performance scene.
[0235] Therefore, imaging control such as focus control is performed with the face 62 of the object introducer 60 as the target subject. This enables video shooting to allow the audience to notice that the object introducer 60 is speaking.
[0236] On the other hand, in Fig.11 In a state where the object introducer 60 is shown making a gesture such as pointing to the product 70 , it is regarded as a product introduction scene in which the object introducer 60 is introducing the product 70 .
[0237] In the product introduction scene, the identified product 70 is set as the target subject, and focus control is performed to focus on the product 70. This allows video shooting to make the audience pay attention to the product 70 that the object introducer 60 is about to introduce.
[0238] In this example, by performing scene determination based on the gesture of the object introducer 60 (eg, pointing to the product 70 ) and performing focus control on a target subject selected according to each scene, focus control suitable for each scene can be performed.
[0239] The relative positional relationship between the target subject 60 and the product 70 is defined based on the state of the hand 61 pointing to the product 70 by gesture. Therefore, the subject selection based on scene determination can be said to be performed based on the object positional relationship between the product 70 and the object introducer 60.
[0240] This example is applicable to, for example, a case where the product 70 cannot be held by the hand 61 , a case where the object introducer 60 introduces the product 70 at a remote location, and the like.
[0241] Note that in the above description, identification of the product 70 to be introduced and scene determination are performed based on the object introducer 60 holding or pointing to the product 70. There may be a case where the product 70 is already identified in the image, and there may be a case where the product 70 can be identified only based on the state of the hand, and any of them is possible.
[0242] For example, even if the product 70 has not been identified, by recognizing a state in which the object introducer 60 holds or points to a specific object, the product 70 can be identified, and scene determination can be performed.
[0243] Furthermore, in a state where the product 70 has been identified, by recognizing a state where the object introducer 60 holds or points to the product 70 , scene determination may be performed.
[0244] Will refer to Fig.12 and 13A fourth embodiment is described. The fourth embodiment is an example in which a scene is determined based on a ratio of an area of a product 70 generated from an object positional relationship to an entire frame of a captured image, and focus control is performed on a target subject suitable for each scene.
[0245] In the object introducer 60 there is no Fig.12 In the state where the product 70 is brought close to the imaging device 1, the ratio of the product 70 to the captured image display screen 50 is not large. Therefore, it is considered that the performance scene is not a state where the object introducer 60 is introducing the product 70.
[0246] Therefore, based on the fact that the ratio of the product 70 to the captured image display screen 50 is less than a predetermined value, it is determined that it is a performance scene, and focus control is performed with, for example, the face 62 of the object introducer 60 as the target subject. This enables video shooting to allow the audience to notice that the object introducer 60 is talking.
[0247] On the other hand, in the object introducer 60, for example Fig.13 In the case where the product 70 is brought close to the imaging device 1 by holding the product 70, the ratio of the product 70 to the captured image display screen 50 becomes large. In this case, it is assumed that the object introducer 60 is about to explain the product 70.
[0248] Therefore, based on the fact that the ratio of the product 70 to the captured image display screen 50 becomes larger than a predetermined value, it is determined to be a product introduction scene, and focus control is performed with the product 70 as the target subject. This enables video shooting to allow viewers to focus on the product 70.
[0249] As described above, in the fourth embodiment, the scene is determined based on the ratio of the area of the product 70 to the area of the captured image to allow imaging control, such as focus control, suitable for a target subject appropriate for each scene.
[0250] As the distance between the product 70 and the imaging device 1 changes, that is, the positional relationship between the imaging device 1 and the product 70 changes, a change in the ratio of the area of the product 70 to the entire frame of the captured image occurs. Therefore, the subject selection based on scene determination can be said to be performed based on the object positional relationship between the imaging device 1 and the product 70. Note that in this case, it can be regarded as a change in the object positional relationship between the object introducer 60 and the product 70.
[0251] As described above, scene determination and imaging control suitable for each scene in the present technology can be performed in various modes.
[0252] <4. Processing to implement the embodiment>
[0253] Will refer to Figures 14 to 21The processing of the imaging control apparatus performed to realize the imaging control in the above-described embodiments is described.
[0254] First, refer to Fig.14 The overall processing procedure in each embodiment is described.
[0255] In each embodiment, the imaging control device 22 of the imaging device 1 identifies the product 70 to be introduced and the object introducer 60 in the captured image data, and performs scene determination based on the object position relationship. Then, one of the introduced object and the object introducer is selected as the target subject according to the determined scene, and focus control suitable for the selected target subject is performed.
[0256] That is, the imaging control device 22 selects one of the product 70 and the object introducer 60 as a target subject based on the object position relationship detected from the captured image data, and performs imaging control suitable for the selected target subject.
[0257] Note that, for the shooting mode of the imaging device 1 according to the present embodiment, there is provided an introduction video mode in which focus control is performed on a target subject selected according to the scene determined as described above. Assume that the imaging control means 22 performs the following operation in the case where video recording is started with the shooting mode set to the introduction video mode. Fig.14 in the processing.
[0258] For example, before starting to record a video, the introduction video mode is set in response to a mode setting operation of the imaging device 1 by the object introducer 60 .
[0259] The following describes the operations performed by the imaging control device 22. Fig.14 in the processing.
[0260] First, in step S101, the imaging control device 22 recognizes the subject. The imaging control device 22 acquires one frame of image data or a plurality of frames of image data from the imaging unit 13, and performs image analysis processing or the like using image signals of the acquired frames to recognize, for example, the hand 61 and the face 62 of the object introducer 60 and the product 70, such as Figures 4 to 13 shown.
[0261] Specifically, it is assumed that the imaging control device 22 recognizes the hand 61 and the face 62 of the object introducer 60 by, for example, estimating the posture of the object introducer 60 or extracting the skin color in the image data.
[0262] Furthermore, for the product 70 to be introduced, the imaging control device 22 recognizes the object portion in the image by shape recognition, pattern recognition, etc. to distinguish it from the object introducer 60, background, etc., thereby identifying the product 70 to be introduced.
[0263] Furthermore, for example, the imaging control device 22 can identify the product 70 to be introduced based on the state of the recognized hand 61. In the case where the imaging control device 22 has recognized the state in which the hand 61 is in contact with the product 70 (e.g., holding, pinching, or grabbing the product 70), it identifies the product 70 in contact with the hand 61 as the product 70 to be introduced. Therefore, in a state in which various products arranged in the shooting location appear, the product 70 touched by the hand 61 can be identified as the product 70 to be introduced in the product review video. That is, the product to be introduced is identified based on the positional relationship in which the product 70 is in contact with the hand 61.
[0264] In addition, the imaging control device 22 can recognize a state in which the hand 61 of the user object introducer 60 makes a gesture for identifying the product 70, and identify the product 70 to be introduced based on the gesture. For example, in the case of recognizing a gesture in which the hand 61 points to the product 70, the product 70 located on the extension line of the direction pointed by the hand 61 can be identified as the product 70 to be introduced. That is, the product to be introduced is identified based on the positional relationship of the product 70 in the direction pointed by the hand 61.
[0265] As described above, the imaging control device 22 can identify the product 70 to be introduced based on the positional relationship between the object introducer 60 and the product 70 .
[0266] Note that the identification of the product to be introduced 70 here includes not only identifying the object to be introduced from the recognized product 70 , but also includes identification by estimating the position of the product to be introduced 70 based on the position of the hand 61 of the object introducer 60 .
[0267] In this case, the imaging control device 22 identifies the hand 61 of the object introducer 60 to estimate that the product 70 to be introduced is located at the position of the hand 61 to identify the position of the product 70 to be introduced. For example, in a case where the product 70 is too small to be recognized on the image, the hand 61 is assumed to be regarded as the product 70 (assuming that the product 70 is held in the hand), and the hand 61 is recognized in place of the product 70 as the real object to be introduced, so that the product 70 can be identified.
[0268] In this way, by identifying the product 70 to be introduced based on the state of the hand 61 of the object introducer 60 detected by the imaging control device 22, or identifying the hand 61 as a substitute for the product 70 which is the real object to be introduced, the product 70 to be introduced in the product review video can be identified even in a state where various products arranged in the shooting location appear.
[0269] Subsequently, in step S102 , the imaging control device 22 executes a scene determination process using the acquired recognition result of the subject.
[0270] The imaging control device 22 determines whether the current scene is a performance scene or a product introduction scene based on the object position relationship between each recognized subject and the imaging device 1. As a specific scene determination process, the examples of the first to fourth embodiments described above are assumed. Figures 17 to 21 The scene determination processing examples corresponding to the respective embodiments are described in sequence.
[0271] In step S103 , the imaging control device 22 branches the process according to the result of the scene determination process.
[0272] When it is determined to be a product introduction scene in the scene determination process, the imaging control device 22 advances the process from step S103 to step S104 and selects the product 70 identified in step S101 as the target subject.
[0273] Then, in step S105, the imaging control device 22 performs imaging control suitable for the area of the product 70 as the target subject. For example, as an example of imaging control, control is performed so that focus control is performed for the product 70. Note that the following description uses focus control as an example of imaging control for explanation.
[0274] Therefore, in the product introduction scene, the imaging device 1 is in a state of performing imaging focused on the product 70. The imaging control device 22 performs control so that a captured image that has been subjected to focus control in the entire screen (frame) using detection information about the area of the product 70 in the frame is presented on the display unit of the imaging device 1.
[0275] Note that, as imaging operation control, the imaging control device 22 may also perform aperture control to reduce the F value in accordance with focus control on the product 70 to narrow the depth of field to blur the foreground and background of the product 70 .
[0276] On the other hand, when it is determined to be a performance scene in the scene determination processing, the imaging control device 22 advances the processing from step S103 to step S108, and selects the object introducer 60 identified in step S101 as the target subject.
[0277] Then, in step S109, the imaging control means 22 performs focus control to focus on the face 62 of the object introducer 60 as the target subject. Therefore, in the performance scene, imaging is performed focusing on the face 62 of the object introducer 60. The imaging control means 22 performs control so that a captured image that has been subjected to focus control in the entire screen (frame) using detection information about the area of the face 62 in the frame is presented on the display unit of the imaging device 1.
[0278] After the processing in the above-mentioned step S105 or step S109, the imaging control device 22 also advances the processing to step S106, and performs presentation control to present information indicating what the currently determined scene or the product 70 that has undergone focus control is through on-screen display, turning on / off a specific indication unit (such as an LED), voice, etc.
[0279] For example, the imaging control device 22 may display an icon or a message indicating that it is a product introduction scene or a performance scene.
[0280] In addition, the imaging control device 22 can superimpose and display a focus frame around the product part on the captured image to indicate that focus control is being performed on the product 70 (if it is a product introduction scene), and can superimpose and display a focus frame around the face part on the captured image to indicate that focus control is being performed on the face 62 (if it is a performance scene).
[0281] Then, in step S107, the imaging control device 22 performs metadata association. For example, the imaging control device 22 generates metadata about scene information, error flags, imaging control parameters, and the like on the current frame.
[0282] The scene information is information indicating whether it is a performance scene or a product introduction scene. The error flag is referenced later. Fig.14 The imaging control parameters are parameters related to the above-mentioned imaging operation control and captured image processing control.
[0283] Then, the imaging control device 22 sends the generated metadata as metadata of the current frame to the camera signal processing unit 16 so that the metadata is associated with the captured image data. Thereafter, the imaging control device 22 returns to step S101 to perform processing.
[0284] Note that the metadata is associated with the captured image data by being incorporated into the image file associated with the corresponding frame, for example, through processing by the camera signal processing unit 16, but is not limited thereto. For example, a metadata file in which an ID associated with the captured image data (image file) and the corresponding frame are indicated may be formed separately from the captured image data to be recorded or transmitted. In any form, it is only required that the captured image data and the metadata can be associated with each other at a later point in time.
[0285] In addition, despite the Fig.12 The video recording process is not shown, but in the implementation Fig.12 During the processing period in , the camera signal processing unit 16 performs processing for video recording on each frame obtained from the imaging unit 13. Fig.12The metadata generated by the imaging control device 22 in step S107 in is recorded in the recording medium in association with the frame of captured image data. Thus, scene determination information, parameters of appropriate imaging control, and the like are associated with the captured image data.
[0286] The imaging control device 22 repeatedly executes Fig.14 The processing in is continued until, for example, the object introducer 60 performs an operation to complete video recording, or the object introducer 60 changes the shooting mode to a mode other than the introduction video mode.
[0287] The above-described processing allows focus control to be performed on a target subject suitable for the scene by the imaging control device 22 in each embodiment.
[0288] Therefore, by performing imaging while focusing on the product to be introduced 70 in a product introduction scene where attention needs to be paid to the product 70, and while focusing on the object introducer 60 in a performance scene where attention needs to be paid to the performance of the object introducer 60, focus control can be effectively performed on objects requiring the audience's attention in each scene.
[0289] Note that in the present embodiment, the introduction video mode is set in response to the mode setting operation of the imaging device 1 by the object introducer 60 as an example, but the setting of the introduction video mode may be performed in various modes.
[0290] For example, suppose Figure 1 The digital camera 1A in the embodiment has an imaging device body 2 and a display housing 3 having a display unit, and by moving the display housing 3 relative to the imaging device body 2, the display unit of the display housing 3 can be turned to the same side as the imaging lens. Such an imaging device 1 can determine that a Selfie is being performed based on this state, and based on this, an introduction video mode can be set. That is, when a state in which the display unit of the display housing 3 is turned to the same side as the imaging lens is detected, the imaging control device 22 sets the shooting mode to the introduction video mode.
[0291] Note that in the filming of product review videos, it is common for the subject introducer 60 to film the video himself, such as Figure 4 Therefore, by allowing the object introducer 60 to recognize the current state of the imaging control through the content displayed on the display unit during the recording of the product review video, the object introducer 60 himself can determine the retake, continuation, etc. of the video during the recording of the product review video.
[0292] Furthermore, the present embodiment describes an example in which the imaging control device 22 identifies the product to be introduced 70 based on the state of the hand 61 recognized in the process of step S101 , but the product to be introduced 70 may be identified based on audio data obtained by the audio input unit.
[0293] For example, by associating the product 70 with the name in advance, the imaging control device 22 can acquire audio data from the speech collected from the imaging apparatus 1, perform language analysis on the acquired audio data, and identify the product 70 corresponding to the name acquired by the language analysis.
[0294] Furthermore, the imaging control device 22 may identify the product 70 to be introduced based on the line of sight direction of the object introducer 60 .
[0295] For example, in step S101 , the imaging control device 22 may identify the sight line direction of the object introducer 60 by performing image analysis processing using the acquired frame information, and identify the product 70 located on the extension line of the identified sight line direction as the product 70 to be introduced.
[0296] Furthermore, before starting to record a product review video in the introduction video mode, the product 70 to be introduced may be registered by the operation of the object introducer 60. For example, the imaging control device 22 recognizes the product 70 from the captured image data and displays the recognized product 70 on the display unit. The object introducer 60 performs an operation to select the product 70 to be introduced from the displayed products 70. The imaging control device 22 registers the product 70 to be introduced from the recognized products 70 in response to the selection operation.
[0297] Alternatively, the imaging control device 22 may register the identified product 70 without causing the object introducer 60 to make a selection.
[0298] In this case, during recording of the product review video, the imaging control device 22 identifies the product 70 registered before the recording among the products 70 identified in step S101 as an introduction object.
[0299] Furthermore, the present embodiment has described the imaging control of the target subject as focus control, but the imaging control of the target subject may be various other types of processing.
[0300] For example, when the imaging control device 22 determines in step S102 that it is a product introduction scene, it advances the processing in the order of steps S103, S104, and S105, and performs AE control so that the exposure of the selected area of the product 70 becomes appropriate. When the imaging control device 22 determines in step S102 that it is a performance scene, it advances the processing in the order of steps S103, S108, and S109, and performs AE control so that the exposure of the selected area of the face 62 of the target subject 60 becomes appropriate.
[0301] Furthermore, in steps S105 and S109, the imaging control device 22 can perform captured image processing control suitable for the target subject as imaging control. Specifically, white balance processing control, contrast adjustment control, and the like can be performed.
[0302] Using detection information about the area of the target subject in the frame, the imaging control device 22 performs automatic white balance control so that the white balance of the area of the target subject becomes appropriate, or performs image quality correction processing or the like as contrast adjustment so that the contrast of the area of the target subject becomes appropriate.
[0303] For example, when the imaging control device 22 determines in step S102 that it is a product introduction scene, it advances the processing in the order of steps S103, S104, and S105, and performs captured image processing control on the selected product 70. When the imaging control device 22 determines in step S102 that it is a performance scene, it advances the processing in the order of steps S103, S108, and S109, and performs captured image processing control on the face 62 of the selected target subject 60.
[0304] Furthermore, in the case where the imaging control device 22 determines that the current scene is a product introduction scene, processing may be performed to cause the product 70 to effectively draw attention.
[0305] For example, when the imaging control device 22 determines in step S102 that it is a product introduction scene, it advances the processing in the order of steps S103, S104 and S105, and can perform image processing to blur the background portion other than the selected product 70 as a captured image processing control suitable for the target subject.
[0306] For example, in Fig.15 On the captured image display screen 50 in FIG. 1 , the portion other than the product 70 is displayed blurred. Fig.15 In the image, blurred subjects are indicated by dotted lines.
[0307] In addition, if Fig.16 As shown, the product 70 can be highlighted by arranging a plurality of concentration lines around the selected product 70. Here, the concentration line refers to a plurality of lines arranged radially around the area from the center thereof.
[0308] For example, when the imaging control device 22 determines in step S102 that it is a product introduction scene, it advances the processing in the order of steps S103, S104 and S105, and as a captured image processing control suitable for the target subject, the acquired captured image data is combined with the concentration line effect data to generate a concentration line effect image in which multiple concentration lines are arranged around the product 70.
[0309] Furthermore, imaging control suitable for the type of the product 70 can be performed on the product 70 as the target subject in the product introduction scene.
[0310] At this time, when the product 70 is identified in step S101, the imaging control device 22 has already determined the type of the product 70 through the image analysis processing, etc. Then, when the imaging control device 22 determines that the current scene is a product introduction scene in the scene determination processing of step S102, it advances the processing in the order of steps S103 and S104, and selects the product 70 as the target subject.
[0311] Thereafter, the imaging control device 22 transmits information on the type of the product 70 determined in step S101 to a cloud server storing imaging control information associated with the type of the product 70 , and acquires imaging control information suitable for the product 70 from the cloud server.
[0312] In step S105 , the imaging control device 22 performs imaging control suitable for the product 70 based on the imaging control information acquired from the cloud server.
[0313] Note that the imaging control information associated with the type of the product 70 may be stored in advance in the imaging device 1 .
[0314] Various examples can be considered for the type of product 70 and the imaging control associated with the type. For example, in the case where the type of product 70 is a television device, a digital camera device, or the like having a display unit in which a flicker phenomenon may occur, SS control for changing to a shutter speed at which a flicker phenomenon does not occur is considered.
[0315] Note that when these types of control processing are performed in the case of a product introduction scene, in the case where the current scene is determined to be a performance scene rather than a product introduction scene, the imaging control device 22 is considered to advance the processing in the order of steps S103, S108, and S109, and perform focus control to focus on the face 62 of the object introducer 60, as described in the above embodiment. For example, it is assumed that blurring of other parts, a concentrated line effect, control suitable for the product type, etc. are not performed.
[0316] The scene determination process in each embodiment is described below.
[0317] Fig.17 An example of scene determination processing according to the first embodiment is shown. The first embodiment is an example of performing scene determination using the distance Loc from the imaging device 1 to the product 70, such as Figure 4 and 5 shown.
[0318] First, in step S201, the imaging control device 22 calculates the distance Loc from the imaging device 1 to the product to be introduced 70. For example, when a frame of captured image data is acquired, a phase difference signal is acquired, and a correlation operation is performed using the acquired phase difference signal to calculate the distance Loc from the imaging device 1 to the product to be introduced 70.
[0319] Note that the imaging control device 22 can calculate the distance Loc from the imaging device 1 to the product 70 based on the position of the focus lens in the contrast AF, or can calculate the distance Loc from the imaging device 1 to the product 70 using detection information of a dedicated ranging sensor using a phase difference sensor, an infrared sensor, etc.
[0320] In step S202 , the imaging control device 22 determines whether the distance Loc from the imaging apparatus 1 to the product 70 is greater than or equal to the shortest imaging distance.
[0321] Among the distances from the imaging device 1 to the subject, the shortest imaging distance refers to the shortest distance at which focus control can be performed without blurring the subject. Therefore, in the case where the distance Loc is shorter than the shortest imaging distance mentioned here, the product 70 is blurred even when focus control is performed on the product 70. For example, in the case where the object introducer 60 brings the product 70 too close to the imaging device 1 to emphasize it, the distance Loc becomes shorter than the shortest imaging distance.
[0322] In this case, appropriate imaging cannot be performed. Therefore, in the case where the distance Loc is shorter than the shortest imaging distance, the imaging control device 22 advances the process from step S202 to step S203 and sets the error flag to on.
[0323] Here, the error flag is a flag indicating whether it is in an appropriate imaging state where scene determination can be performed. The error flag is turned on to indicate a determination impossible state where the focus controllable range is exceeded, blur-free video shooting cannot be performed, and target subject selection based on scene determination cannot be performed.
[0324] If the error flag is turned on in step S203, the imaging control device 22 will process from Fig.14 The process proceeds to step S106 from step S204 in the above step, and performs control to present the determination impossible state. For example, the presentation control for presenting the determination impossible state is performed by displaying an icon or message indicating the determination impossible state on the screen, turning on / off a specific indication unit such as an LED, an error sound or voice, etc.
[0325] Note that, in the case where the display unit displaying the captured image does not face the object introducer 60 side, it is desirable to perform presentation control to the object introducer 60 using an instruction unit other than the display unit, voice, or the like.
[0326] Then, in Fig.14 In step S107, the imaging control device 22 generates metadata indicating that the error flag is turned on. The generated metadata is sent to the camera signal processing unit 16 and recorded in the recording medium in association with the corresponding frame of the captured image data, for example. Thereafter, the imaging control device 22 returns to Fig.14 Step S101 in to perform the above processing.
[0327] exist Fig.15 The case where the distance Loc is greater than the shortest imaging distance in step S202 is a case where the error processing as described above is not particularly required.
[0328] In step S210, the imaging control device 22 determines whether the error flag is on. If the error flag is on, the error flag is turned off in step S205, and the process proceeds to step S206.
[0329] Furthermore, if it is determined in step S210 that the error flag is off, the imaging control device 22 advances the process to step S206 without executing the process in step S205 .
[0330] In step S206 , the imaging control device 22 determines whether the calculated distance Loc is smaller than a predetermined value Lth1 .
[0331] As mentioned above Figure 4 and 5 As understood from the description, the predetermined value Lth1 is a reference value for determining whether the current scene is a performance scene or a product introduction scene. For example, if the distance Loc is less than the predetermined value Lth1, it is determined to be a product introduction scene, and if the distance Loc is greater than or equal to the predetermined value Lth1, it is determined to be a performance scene.
[0332] The predetermined value Lth1 may be set as needed by the person who performs the shooting, or may be set in advance in the imaging device 1. Alternatively, for example, before starting the imaging or during the imaging, a process may be performed to set the predetermined value Lth1 for the current imaging based on the actually measured distance data. For example, the predetermined value Lth1 may be determined by measuring the distance to the object introducer 60 and performing subtraction of an appropriate distance from the distance, etc.
[0333] When the imaging control device 22 determines in step S206 that the distance Loc is shorter than the predetermined value Lth1, it determines in step S207 that the current scene is a product introduction scene.
[0334] Then, the imaging control device 22 performs the steps S204 and Fig.14The process of steps S103 and S104 in the sequence of steps S103 and S104 is advanced, the product 70 identified in step S101 is selected as the target subject, and in step S105, for example, focus control is performed on the product 70 as the target subject as imaging control. Of course, as described above, the imaging control device 22 can perform various types of control such as SS control for preventing a flicker phenomenon, brightness processing, and image effect processing as imaging control separately from or in addition to the focus control.
[0335] Thereafter, the imaging control device 22 performs the processing in steps S106 and S107 .
[0336] If the imaging control device 22 Fig.17 If it is not determined in step S206 that the distance Loc is less than the predetermined value Lth1, it is determined in step S208 that the current scene is a performance scene.
[0337] Afterwards, the imaging control device 22 performs the steps S204 and Fig.14 The object introducer 60 identified in step S101 is selected as the target subject, and, for example, focus control on the face 62 of the object introducer 60 as the target subject is performed as imaging control in step S109. Then, the imaging control device 22 performs the processing in steps S106 and S107.
[0338] As described above, according to the first embodiment, by performing scene determination based on the distance Loc from the imaging device 1 to the product 70 , scene determination based on the object positional relationship between the object introducer 60 and the product to be introduced 70 is performed.
[0339] Will refer to Fig.18 The scene determination process according to the second embodiment is described. The second embodiment is an example of performing scene determination using the distance Lho from the object introducer 60 to the product 70, such as Figure 6 and 7 shown.
[0340] In step S251, the imaging control device 22 calculates the distance Loc from the imaging device 1 to the product 70 and the distance Lhc from the imaging device 1 to the object introducer 60. In addition, the distance Lho between the object introducer 60 and the product 70 is determined by calculating the distance Lhc-the distance Loc.
[0341] In step S252, the imaging control device 22 determines whether the distance Loc from the imaging apparatus 1 to the product 70 or the distance Lhc from the imaging apparatus 1 to the object introducer 60 is greater than or equal to the shortest imaging distance.
[0342] If the distance Loc or the distance Lhc is shorter than the shortest imaging distance, appropriate imaging cannot be performed. Therefore, the imaging control device 22 advances the process from step S252 to step S258, sets the error flag to on, and determines that it is in a determination impossible state.
[0343] In this case, the imaging control device 22 advances the processing from step S259 to Fig.14 Step S106 in , and control is performed to present a determination impossible state.
[0344] Then, in Fig.14 In step S107, the imaging control device 22 generates metadata indicating that the error flag is on, and then returns to step S101 to perform the above-mentioned processing. Fig.17 A first embodiment is described.
[0345] exist Fig.18 In the case where the distance Loc is greater than or equal to the shortest imaging distance in step S252, the imaging control device 22 determines in step S253 whether the error flag is on. If the error flag is on, the error flag is turned off in step S254, and the process proceeds to step S255.
[0346] Furthermore, if it is determined in step S253 that the error flag is off, the imaging control device 22 advances the process to step S255 without executing the process in step S254 .
[0347] In step S255 , the imaging control device 22 determines whether the calculated distance Lho is smaller than a predetermined value Lth2 .
[0348] As mentioned above Figure 6 and 7 As understood from the description of , the predetermined value Lth2 is a reference value for determining whether the current scene is a performance scene or a product introduction scene. For example, if the distance Lho is less than the predetermined value Lth2, it is determined to be a performance scene, and if the distance Lho is greater than or equal to the predetermined value Lth2, it is determined to be a product introduction scene.
[0349] The predetermined value Lth2 may be set as needed by the person who performs the shooting, or may be set in advance in the imaging device 1. Alternatively, for example, before starting imaging or during imaging, processing may be performed to set the predetermined value Lth2 for the current imaging based on the actually measured distance data. For example, the predetermined value Lth2 may be appropriately determined based on the scene distance Lhc and the distance Loc measured at a time point before reaching the product introduction (for example, at the start of imaging).
[0350] When the imaging control device 22 determines in step S255 that the distance Lho is smaller than the predetermined value Lth2, it determines in step S257 that the current scene is a performance scene.
[0351] Then, the imaging control device 22 performs the steps S259 and Fig.14 The object introducer 60 identified in step S101 is selected as the target subject, and, for example, focus control on the face 62 of the object introducer 60 as the target subject is performed as imaging control in step S109. Then, the imaging control device 22 performs the processing in steps S106 and S107.
[0352] If the imaging control device 22 Fig.18 If it is not determined in step S255 that the distance Lho is less than the predetermined value Lth2, it is determined in step S256 that the current scene is a product introduction scene.
[0353] Then, the imaging control device 22 performs the steps S259 and Fig.14 The imaging control device 22 sequentially advances the processing of steps S103 and S104 in step S106, selects the product 70 identified in step S101 as the target subject, and performs, for example, focus control on the product 70 as the target subject as imaging control in step S105. Thereafter, the imaging control device 22 performs the processing in steps S106 and S107.
[0354] As described above, according to the second embodiment, by performing scene determination based on the distance Lho from the object introducer 60 to the product 70 , scene determination based on the object positional relationship between the object introducer 60 and the product to be introduced 70 is performed.
[0355] Note that, as described above, in a scenario where the object introducer 60 introduces the product 70, the object introducer 60 may introduce the product 70 by bringing the product 70 close to himself or herself with the hand 61. When such behavior is assumed, the logic of step S255 may be reversed.
[0356] That is, the imaging control device 22 determines that the current scene is a product introduction scene based on the fact that the distance Lho has become smaller than a predetermined value, and determines that the current scene is a performance scene if the distance Lho is greater than or equal to the predetermined value.
[0357] Note that in the first and second embodiments, the imaging control device 22 performs scene determination based on the distance Loc from the imaging device 1 to the product 70 or the distance Lho between the object introducer 60 and the product 70, but the scene determination may be performed based on the distance from the imaging device 1 to the object introducer 60.
[0358] Will refer to Fig.19 The scene determination process according to the third embodiment is described. The third embodiment is an example of performing scene determination according to the state of a part of the body of the object introducer 60. Here, as an example, an example of performing scene determination according to the state of the hand 61 of the object introducer 60 will be described.
[0359] In step S301, the imaging control device 22 determines the state of the hand 61 of the object introducer 60. That is, the imaging control device 22 determines the state of the hand 61 of the object introducer 60 by using Fig.14 The image data frame acquired in step S101 performs image analysis processing to determine whether the hand 61 is in the object identification state.
[0360] Here, the object identification state refers to a state in which the product 70 to be introduced can be identified from the hand 61 of the object introducer 60, and is, for example, a state in which the hand 61 is in contact with the product 70 (for example, the hand 61 holds, pinches or grabs the product 70), a state in which a gesture for identifying the product 70 is made with the hand 61 (for example, the hand 61 of the object introducer 60 points to the product 70), etc.
[0361] Note that it is considered that the imaging control device 22 determines whether the hand 61 is in the object identification state by performing image analysis processing using the acquired image data frame. However, by acquiring multiple frames of image data and performing image analysis processing, etc., the movement of the hand 61 can be detected from the change of the image data between the frames to determine whether the hand 61 is in the object identification state based on the movement.
[0362] When the imaging control device 22 determines in step S302 that the hand 61 is in the object identification state, it determines in step S303 that the current scene is a product introduction scene.
[0363] Thereafter, the imaging control device 22 performs the following operations: Fig.14 The process proceeds in the order of steps S103 and S104 shown, the product 70 identified in step S101 is selected as the target subject, and in step S105, focus control is performed to focus on the product 70 as the target subject. Then, the imaging control device 22 performs the processes in steps S106 and S107.
[0364] Note that when the hand 61 is in an object identification state where the hand 61 is in contact with the product 70 (for example, it holds the product 70), the imaging control device 22 can perform focus control etc. suitable for the area of the hand 61 in contact with the product 70 in step S105.
[0365] Performing focus control on the area of the hand 61 of the object introducer 60 allows video shooting to also focus on the product 70 in contact with the hand 61. This is particularly effective in a case where the product 70 is too small to perform focus control on the product 70 itself.
[0366] On the other hand, when the imaging control device 22 determines in step S302 that the hand 61 is not in the object identification state, it determines in step S304 that the current scene is a performance scene.
[0367] Afterwards, the imaging control device 22 performs Fig.14 The imaging control device 22 sequentially advances the processing of steps S103 and S108, selects the object introducer 60 identified in step S101 as the target subject, and performs focus control on the face 62 of the object introducer 60 as the target subject in step S109. Then, the imaging control device 22 performs the processing in steps S106 and S107.
[0368] As described above, according to the third embodiment, by performing scene determination based on whether the hand 61 of the object introducer 60 is in the object identification state, scene determination based on the object position relationship between the hand 61 of the object introducer 60 and the product 70 to be introduced is performed.
[0369] Note that the third embodiment may take the following form. Fig. 20 A modification of the third embodiment is described.
[0370] This example is an example of performing scene determination based on the object position relationship using the state of the hand 61 which is a part of the body of the object introducer 60 and the distance from the imaging device 1 to the product 70 based on the object position relationship.
[0371] First, in step S401, the imaging control device 22 determines the state of the hand 61 of the object introducer 60. That is, the imaging control device 22 determines the state of the hand 61 of the object introducer 60 by using Fig.14 The image data frame acquired in step S101 performs image analysis processing to determine whether the hand 61 is in an object identification state such as holding the product 70.
[0372] When the imaging control device 22 determines in step S402 that the hand 61 is not in the object identification state, it determines in step S403 that the current scene is a performance scene.
[0373] Afterwards, the imaging control device 22 completes Fig. 20 In the processing, according to Fig.14 The imaging control device 22 sequentially advances the processing of steps S103 and S108 in step 106, selects the subject introducer 60 identified in step S101 as the target subject, and performs focus control on the area of the face 62 of the subject introducer 60 as the target subject in step S109. Then, the imaging control device 22 performs the processing in steps S106 and S107.
[0374] When the imaging control device 22 is Fig. 20When it is determined in step S402 that the hand 61 is in the object identification state, it advances the processing to step S404 and calculates the distance Loc from the imaging device 1 to the product 70 to be introduced.
[0375] In step S405 , the imaging control device 22 determines whether the calculated distance Loc is smaller than a predetermined value.
[0376] When the imaging control device 22 determines in step S405 that the distance Loc is smaller than the predetermined value Lth1, it determines in step S406 that the current scene is a product introduction scene.
[0377] Afterwards, the imaging control device 22 completes Fig. 20 In the processing, according to Fig.14 The process of steps S103 and S104 in the sequence of steps S103 and S104 is advanced, the product 70 identified in step S101 is selected as the target subject, and focus control is performed on the product 70 as the target subject in step S105. Then, the imaging control device 22 performs the processes in steps S106 and S107.
[0378] When the imaging control device 22 determines in step S405 that the distance Loc is greater than or equal to the predetermined value Lth1, it determines in step S403 that the current scene is a performance scene.
[0379] Afterwards, the imaging control device 22 completes Fig. 20 In the processing, according to Fig.14 The imaging control device 22 sequentially advances the processing of steps S103 and S108, selects the object introducer 60 identified in step S101 as the target subject, and performs focus control on the face 62 of the object introducer 60 as the target subject in step S109. Then, the imaging control device 22 performs the processing in steps S106 and S107.
[0380] As described above, by performing scene determination based on the state of the hand 61 of the object introducer 60 and the distance Loc from the product 70 to the imaging device 1, scene determination based on the object position relationship between the object introducer 60 and the product to be introduced 70 is performed.
[0381] For example, in a product review video, the object introducer 60 may hold the product 70 with the hand 61 (i.e., it may be in the object identification state) even if the product 70 is not shown and introduced. In this case, if the focus control is performed on the hand 61 based on the state of the hand 61 of the object introducer 60 (e.g., held), the focus control can be performed on the product 70 even if it is actually a performance scene.
[0382] Therefore, by performing scene determination while considering the state of the distance Loc from the imaging device 1 to the product 70 in addition to the state of the hand 61 of the object introducer 60, it is possible to perform imaging control such as focus control in which the intention of the object introducer is more reflected. That is, the accuracy of scene determination performed by the imaging control device 22 can be improved.
[0383] Note that in scene determination, the imaging control device 22 in this example calculates the distance Loc from the imaging device 1 to the product 70 to be introduced in step S404, but, when the hand 61 is in an object identification state such as holding the product 70, it can calculate the distance from the imaging device 1 to the hand 61 of the object introducer 60.
[0384] This is because the distance from the imaging device 1 to the product 70 can be estimated based on the distance from the imaging device 1 to the hand 61 in a state where the hand 61 is in contact with the product 70 (eg, the hand 61 holds the product 70 ).
[0385] Furthermore, at this time, the imaging control device 22 may perform focus control on the hand 61 in contact with the product 70 in step S105. This also allows video shooting focused on the product 70.
[0386] In addition, although not described in this example, when the distance Loc from the imaging device 1 to the product 70 is less than the shortest imaging distance, the imaging control device 22 can set the error flag indicating the determination impossible state to on and perform an error display on the display unit of the imaging device 1.
[0387] Will refer to Fig.21 A scene determination process according to a fourth embodiment is described. The fourth embodiment is an example in which scene determination is performed based on the ratio of the area of the product 70 to be introduced to the entire frame of the captured image. The ratio in this case corresponds to the object position relationship.
[0388] First, in step S501, the imaging control device 22 calculates the ratio of the product 70 to be introduced to the entire frame of the captured image. That is, the imaging control device 22 calculates the ratio P of the area of the product 70 to the area of the entire frame of the captured image.
[0389] Thereafter, in step S502, the imaging control device 22 determines whether the ratio P of the area of the product 70 to the area of the entire frame of the captured image is greater than a predetermined value thP. The predetermined value thP mentioned here is a reference value for determining whether the current scene is a performance scene or a product introduction scene. Based on the fact that the ratio P is greater than the predetermined value thP, it is determined to be a product introduction scene.
[0390] If the ratio P of the area of the product 70 to the area of the entire frame of the captured image is greater than the predetermined value thP, the imaging control device 22 determines in step S503 that the current scene is a product introduction scene.
[0391] It can be said that an increase in the ratio P of the area of the product 70 to the area of the entire frame of the captured image indicates a decrease in the distance between the product 70 and the imaging device 1. Therefore, it can be estimated that the object introducer 60 is about to introduce the product 70, thereby bringing the product 70 closer to the imaging device 1.
[0392] Therefore, the scene determination based on the area ratio P of the product 70 in this embodiment can also be said to be indirectly based on the scene determination based on the distance relationship between the product 70 and the imaging device 1. That is, in this embodiment, the positional relationship between the product 70 and the imaging device 1 is detected by a physical quantity different from the distance.
[0393] Afterwards, the imaging control device 22 performs the steps S503 and Fig.14 The process of steps S103 and S104 in the sequence of steps S103 and S104 is advanced, the product 70 identified in step S101 is selected as the target subject, and focus control is performed on the product 70 as the target subject in step S105. Then, the imaging control device 22 performs the processes in steps S106 and S107.
[0394] If in Fig.21 If, in step S502, the ratio P of the area of the product 70 to the area of the entire frame of the captured image is less than or equal to the predetermined value thP, the imaging control device 22 determines in step S504 that the current scene is a performance scene.
[0395] Afterwards, the imaging control device 22 performs the steps S504 and Fig.14 The imaging control device 22 sequentially advances the processing of steps S103 and S108 in step S106, selects the object introducer 60 identified in step S101 as the target subject, and performs focus control on the face 62 of the object introducer 60 as the target subject in step S109. Then, the imaging control device 22 performs the processing in steps S106 and S107.
[0396] According to the fourth embodiment described above, scene determination is performed based on the ratio of the product 70 to be introduced to the entire frame of the captured image, so that scene determination is performed based on the object position relationship between the hand 61 of the object introducer 60 and the product 70 to be introduced.
[0397] Note that in this embodiment, the imaging control device 22 performs scene determination based on the ratio of the product 70 to be introduced to the entire frame of the captured image, but may perform scene determination based on the ratio of the area of the object introducer 60 to the entire frame of the captured image.
[0398] For example, in step S501, the imaging control device 22 calculates the ratio P1 of the area of the region of the object introducer 60 to the area of the captured image, and in step S502, determines whether the ratio P1 is less than a predetermined value thP'. The predetermined value thP' mentioned here is a reference value for determining whether the current scene is a performance scene or a product introduction scene. Based on the fact that the ratio P1 is less than the predetermined value thP', it is determined to be a product introduction scene.
[0399] If the ratio P1 of the area of the object introducer 60 to the area of the captured image is smaller than the predetermined value thP′, the imaging control device 22 determines in step S503 that the current scene is a product introduction scene.
[0400] Furthermore, if the ratio P1 is greater than or equal to the predetermined value thP′, the imaging control device 22 determines in step S504 that the current scene is a performance scene.
[0401] This is because, from the fact that the ratio P1 of the area of the object introducer 60 to the area of the captured image becomes greater than or equal to the predetermined value thP′, it can be estimated that the object introducer 60 is approaching the imaging device 1 to provide a performance.
[0402] Furthermore, the imaging control device 22 may perform scene determination based on both the ratio P of the product 70 to be introduced on the captured image and the ratio P1 of the object introducer 60 on the captured image.
[0403] For example, if the ratio P of the area of the product 70 to the area of the captured image becomes greater than the predetermined value thP, and the ratio P1 of the area of the object introducer 60 to the area of the captured image becomes less than the predetermined value thP′, the imaging control device 22 can determine that the current scene is a product introduction scene; otherwise, it can determine that the current scene is a performance scene.
[0404] Furthermore, the imaging control device 22 may perform scene determination based not only on the ratio of the product 70 or the object introducer 60 on the captured image but also on the area ratio between the product 70 and the object introducer 60 .
[0405] <5. Summary and revision>
[0406] The imaging control device 22 installed in the imaging device 1 according to the above-mentioned embodiment includes: an identification unit 22a, which identifies an introduction object (product 70) as a subject in captured image data obtained by the imaging unit 13 of the imaging device 1 and an object introducer 60 who introduces the introduction object (product 70); a selection unit 22b, which selects one of the introduction object (product 70) and the object introducer 60 as a target subject based on a positional relationship between the introduction object (product 70), the object introducer 60, and at least any two of the imaging device 1; and an imaging control unit 22c (see FIG. 1 ) that performs imaging control on the subject selected as the target subject by the selection unit 22b. Fig.14 and Figures 17 to 21 ).
[0407] Therefore, for example, even if the object introducer 60 is in a state where he cannot operate the imaging device 1, such as performing a performance for a product review video or the like at a certain distance from the imaging device 1 on which the imaging control device 22 is installed, the imaging control device 22 automatically performs imaging control suitable for the scene according to the movement (e.g., moving, holding, or pointing to a product) of the hand 61. This allows the object introducer 60 to shoot a video that has undergone imaging control reflecting the intention of the object introducer 60 without operating the imaging device 1 during performance shooting.
[0408] Note that the positional relationship between at least any two of the introduction object, the object introducer, and the imaging device 1 includes a positional relationship between the object introducer 60 and the introduction object as in the second and third embodiments, and a positional relationship between the introduction object and the imaging device 1 as in the first and fourth embodiments, and a positional relationship between the imaging device 1 and the object introducer 60. Examples corresponding to the positional relationship between the imaging device 1 and the object introducer 60 include, for example, the following example: scene determination (selection of the target subject) is performed based on whether the ratio P1 of the area of the object introducer 60 to the area of the captured image is less than the predetermined value thP' described in the fourth embodiment. Of course, the scene determination (selection of the target subject) can be performed by setting Figure 6 This is performed by comparing the distance Lhc in with an appropriate threshold.
[0409] Furthermore, an example of the positional relationship may be the positional relationship among the object introducer, the introduction object, and the imaging device.
[0410] In the imaging control device 22 according to each embodiment, the selection unit 22b performs scene determination processing on the captured image based on the object position relationship, and can select one of the to-be-introduced product 70 and the object introducer 60 as the target subject according to the scene determined in the scene determination processing (see Fig.14 . , step S102 in FIG.
[0411] This allows imaging control such as focus control to be appropriately performed on a target subject suitable for each scene. Therefore, during video shooting, video shooting reflecting the intention of the subject introducer 60 can be performed without directly operating the imaging device 1.
[0412] For example, in a product review video, focus control etc. can be performed with the product 70 as the target subject in the product introduction scene which is a scene for introducing the product 70, and focus control etc. can be performed with the object introducer 60 as the target subject in the performance scene which is a scene for performing by the object introducer 60. Therefore, focus control etc. suitable for the subject that should be paid attention to in the current scene can be performed.
[0413] In the imaging control device 22 according to the first embodiment, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the positional relationship between the introduction object (product 70) and the imaging device 1. That is, the imaging control device 22 performs scene determination processing based on the positional relationship between the product 70 and the imaging device 1 (see Fig.14 and Fig.17 (S102 in).
[0414] For example, the selection unit 22 b selects one of the introduction object (product 70 ) and the object introducer 60 as the target subject based on the positional relationship of the product 70 relative to the imaging device 1 .
[0415] This allows the object introducer 60 to shoot a video subject to imaging control reflecting the intention of the object introducer 60 without operating the imaging device 1 during shooting.
[0416] In particular, in the first embodiment, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the distance Loc from the imaging device 1 to the introduction object (product 70) obtained from the positional relationship between the introduction object (product 70) and the imaging device 1.
[0417] For example, in a product review video, the object introducer 60 can bring the product 70 close to the imaging lens of the imaging device 1 to draw the audience's attention to the product 70 to be introduced. In this case, for example, based on the fact that the distance from the product 70 to be introduced to the imaging device 1 becomes smaller than a predetermined value, the current scene is determined to be a product introduction scene, allowing imaging control such as focus control suitable for the area of the product 70 selected as the target subject to be performed in the product introduction scene. In particular, distance measurement allows scene determination and appropriate control, facilitating control.
[0418] In the imaging control device 22 according to the second embodiment, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the positional relationship between the introduction object (product 70) and the object introducer 60. That is, the imaging control device 22 performs scene determination processing based on the positional relationship between the product 70 and the object introducer 60 (see Fig.14 and Fig.17 . , step S102 in FIG.
[0419] The positional relationship between the product 70 and the object introducer 60 may be determined based on the respective distances of the product 70 and the object introducer 60 to the imaging device 1 .
[0420] Therefore, selecting the target subject based on the positional relationship between the product 70 and the object introducer 60 allows the object introducer 60 to shoot a video through imaging control reflecting the intention of the object introducer 60 without operating the imaging device 1 during shooting.
[0421] In addition, scene determination and target subject selection based on the positional relationship between the introduction object (product 70) and the object introducer 60 are less affected by the forward and backward movement of the object introducer 60, so that the accuracy of scene determination and target subject selection can be maintained.
[0422] In particular, in the second embodiment, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the distance Lho from the object introducer 60 to the introduction object (product 70) obtained from the positional relationship between the introduction object (product 70) and the imaging device 1.
[0423] This allows determination of the scene by easily grasping the positional relationship between the object introducer 60 and the product 70, and allows control by setting a target subject suitable for the scene.
[0424] In the imaging control device 22 according to the third embodiment, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the state of a part of the body of the object introducer 60, and performs imaging control suitable for the selected target subject, such as focus control (see Fig.19 ). In particular, based on the positional relationship between the introduction object (product 70) and the object introducer 60 depending on the state of the hand 61 of the object introducer 60, one of the introduction object and the object introducer is selected as the target subject.
[0425] For example, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the state of contact between the hand 61 of the object introducer 60 and the introduction object (product 70), and performs imaging control, such as focusing control, of an area suitable for the selected target subject.
[0426] The state where the hand 61 of the object introducer 60 is not in contact with the product 70 is considered to be a performance scene in which the object introducer 60 is performing a performance such as an opening speech. In this case, focus control is performed on the face 62 with the object introducer 60 as the target subject. This can make the object introducer 60 who is giving the performance stand out in the performance scene.
[0427] On the other hand, in a state where the hand 61 of the object introducer 60 is in contact with the product 70 (e.g., holding the product 70), the imaged scene is considered to be a product introduction scene for introducing the product 70. Therefore, focus control is performed on the product 70 to be introduced so that the product 70 to be introduced can be highlighted.
[0428] Therefore, by determining one of the product 70 and the object introducer 60 as the target subject based on the state of the hand 61 of the object introducer 60, imaging control suitable for each subject can be performed according to each scene. Therefore, video shooting that more reflects the intention of the object introducer 60 can be performed.
[0429] Furthermore, in the third embodiment, the selection unit 22 b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on the state in which the hand 61 of the object introducer 60 points to the introduction object (product 70), and performs imaging control such as focus control on the selected target subject (see Fig. 20 ).
[0430] For example, a state in which the hand 61 of the object introducer 60 does not make a gesture such as pointing is considered to be a performance scene in which the object introducer 60 is performing a performance. In this case, focus control is performed on the face 62 with the object introducer 60 as the target subject. This can make the object introducer 60 who is giving the performance stand out.
[0431] On the other hand, a state in which the hand 61 of the object introducer 60 is making a gesture such as pointing at the product 70 is considered as a product introduction scene in which the object introducer 60 introduces the product 70. Therefore, by performing focus control on the product 70 to be introduced, the product 70 to be introduced can be highlighted.
[0432] As described above, scene determination can also be performed based on the state of gestures such as pointing by the hand 61 of the object introducer 60, and imaging control suitable for each subject can be performed according to each scene. Therefore, video shooting that more reflects the intention of the object introducer 60 can be performed.
[0433] In the imaging control device 22 according to the fourth embodiment, the selection unit 22b selects one of the introduction object (product 70) and the object introducer 60 as the target subject based on one or both of the ratio of the area of the introduction object (product 70) to the entire frame of the captured image and the ratio of the area of the object introducer 60 to the entire frame of the captured image as the positional relationship (see Fig.21 ).
[0434] For example, in a product review video, the object introducer 60 may bring the product 70 close to the imaging lens of the imaging device 1 to draw the audience's attention to the introduced product 70. At this time, due to the change in the object positional relationship, the ratio of the introduced product 70 on the captured image increases.
[0435] Then, for example, based on the fact that the ratio of the product 70 to be introduced on the captured image has become greater than a predetermined value, the current scene is determined to be a product introduction scene, and imaging control such as focus control can be performed on the product 70 selected as the target subject in the product introduction scene.
[0436] The first and second embodiments describe examples in which the positional relationship between the object introducer 60, the introduction object (product 70), and the imaging device 1 is regarded as a distance relationship. Determination of the distance relationship allows relatively easy determination of the scene and selection of the target subject by comparison between distances or comparison between the distance and a predetermined value (threshold value).
[0437] In addition, the distance relationship may be the distance between the object introducer 60, the introduction object (product 70), and the imaging device 1. That is, scene determination and target subject selection may be performed based on two or all of the distance Lho between the object introducer 60 and the introduction object, the distance Loc between the introduction object and the imaging device 1, and the distance Lhc between the imaging device 1 and the object introducer 60.
[0438] As a possible example of scene determination based on the respective distances between the three, if the distance Lhc between the object introducer 60 and the product 70 is too long, even when the distance Loc between the product 70 and the imaging device 1 is less than or equal to the predetermined value Lth1, it is determined that it is not an introduction scene (for example, the object introducer 60 will obtain another product).
[0439] Alternatively, for a performance scene, it may be further set as a condition that the distance Lhc between the object introducer 60 and the imaging device 1 is within a specific range.
[0440] In the imaging control device 22 according to each embodiment, the identification unit 22a identifies the introduction object (product 70) based on the captured image data (see Fig.14 That is, the identification unit 22a performs, for example, image analysis processing or the like on the captured image data acquired from the imaging unit 13 to identify the product 70 to be introduced from the subject appearing in the image data. Thus, the product 70 corresponding to the imaged subject is identified.
[0441] In each embodiment, the imaging control device 22 can detect the hand 61 of the object introducer 60 based on the captured image data, and identify the introduction object (product 70) based on the position of the detected hand 61. This allows the product 70 to be introduced to be identified by estimating the position of the product 70 from the position of the hand 61 even in a case where the product 70 cannot be detected based on the image data.
[0442] In the imaging control device 22 according to each embodiment, an example has been described in which the identification unit 22a identifies the introduction object (product 70) based on the state of the body part (hand 61) of the object introducer 60 (see Fig.14 (S101 in ).
[0443] This allows the product 70 to be introduced to be identified from, for example, the state of the hand 61 as a part of the body of the object introducer 60 (e.g., holding, grasping, pinching, or pointing to the product 70). Therefore, even in the case where a plurality of products 70 appear in the captured image data, the product 70 to be introduced can be identified based on the state of the hand 61.
[0444] In addition, in the imaging control device 22 according to each embodiment, the identification unit 22a can virtually identify the hand 61 as an introduction object as a substitute for the real object to be introduced. By identifying the hand as a substitute for the introduction object when identifying the product 70 to be introduced, the identification process is facilitated.
[0445] In imaging, various subjects appear on the imaging screen, so it is difficult to determine which product 70 is the subject to be introduced in, for example, a product review video. Therefore, by identifying the product 70 to be introduced from the state of the hand 61 of the subject introducer 60 which is easily detected by image analysis processing or the like, the introduction subject can be easily identified from the product 70 appearing in the captured image.
[0446] In the imaging control device 22 according to each embodiment, an example has been described in which the imaging control is the imaging operation control (i.e., the control of the imaging operation of the optical system for focusing the subject light onto the image sensor 14 of the imaging unit 13 and the imaging unit 13) (see Fig.14 . , steps S105 and S109 in FIG.
[0447] For example, auto focus control suitable for the target subject, AE control (aperture control, SS control, and gain control), etc. Therefore, an imaging operation reflecting the intention of the subject introducer 60 can be performed without directly operating the imaging device 1 during video shooting.
[0448] For example, by performing automatic focus control on a target subject suitable for the current scene, imaging focused on the product 70 in the product introduction scene and the object introducer 60 in the performance scene can be performed in the product review video.
[0449] In the imaging control device 22 according to the embodiment, an example has been described in which the imaging control is the captured image processing control (ie, the control of the image processing of the captured image data) (see Fig.12 For example, white balance processing control, contrast adjustment processing control, image effect processing control, etc. suitable for the area of the target subject are performed on the captured image data.
[0450] Therefore, image signal processing suitable for the current scene is performed, and signal processing reflecting the intention of the object introducer 60 is performed without directly operating the imaging device 1 during video shooting.
[0451] The imaging control device 22 according to each embodiment associates metadata related to the result of the selection by the selection unit 22b with the captured image data (see Fig.14 ).
[0452] This facilitates extraction of performance scenes and extraction of product introduction scenes as videos from captured image data during reproduction and editing.
[0453] In addition, the information associated with the captured image data as metadata also includes information about an error flag indicating that the selection unit 22b is in a determination impossible state. Therefore, for example, by deleting a frame to which an error flag is attached after recording, a portion in which proper video shooting cannot be performed can be efficiently deleted.
[0454] That is, metadata associated with the result of selection by the selection unit 22b improves the efficiency of editing captured image data into a video and facilitates reproduction work for inspection.
[0455] Furthermore, the imaging apparatus 1 according to the embodiment is provided as an imaging apparatus that achieves the above-described effects by including the above-described imaging control device 22 .
[0456] The imaging device 1 includes a presenting unit 18, and the presenting unit 18 presents the fact that the selection unit 22b is in a determination impossible state (see Fig.14 Step S106 in Fig.17 Steps S202, S203, S204, etc. in the above description).
[0457] Therefore, the determination impossible state is displayed on the display unit of the presentation unit 18. Furthermore, a sound providing notification of the determination impossible state may be emitted from the audio output of the presentation unit 18.
[0458] Therefore, the object introducer 60 can notice that the imaging device 1 is in a determination impossible state. For example, in the case where the object introducer 60 does not notice that the imaging device 1 is already in a determination impossible state during filming, the video filming may not be subject to the imaging control that reflects the intention of the object introducer 60. In this case, the object introducer 60 needs to shoot the video from the beginning, and consumes extra work and time. Therefore, by notifying the object introducer 60 of the determination impossible state so that he or she notices it in the middle, the convenience of the object introducer 60 can be improved.
[0459] Note that by notifying the determination impossible state by sound from the speaker of the presentation unit 18 , the object introducer 60 can be made aware of the error even when the display unit of the imaging device 1 does not face the object introducer 60 side.
[0460] More specifically, when the distance Loc from the imaging device 1 to the introduction object (product 70) is smaller than the shortest imaging process, the presentation unit 18 presents the control impossible state.
[0461] For example, when the product 70 to be introduced is at a distance shorter than the shortest imaging distance, the focusing lens cannot move to focus on the product 70, and the product 70 experiences blur. Then, by notifying the object introducer 60 of the determination impossible state so that he or she notices it in the middle, the convenience of the object introducer 60 is improved.
[0462] Specifically, the control impossible state is displayed to the object introducer 60 (see Fig.14 Step S106 in the above example, etc.) allows the object introducer 60 to visually recognize it to notice the determination impossible state. In addition, notification by display on the display unit of the imaging device 1 rather than notification by sound can prevent unnecessary sound from being recorded during shooting.
[0463] The program according to the embodiment is for causing a CPU, a DSP, or the like, or a device including them to execute Fig.14 and Figures 17 to 21 The processing procedure in .
[0464] That is, the program according to the embodiment is a program for causing the imaging control device to perform the following processing: identification processing, identifying the introduction object (for example, product 70) and the object introducer 60 who introduces the introduction object as subjects, respectively, based on the captured image data obtained by the imaging unit 13 of the imaging device 1; selection processing, selecting one of the introduction object and the object introducer as the target subject based on the positional relationship between at least any two of the introduction object (product 70), the object introducer 60 and the imaging device 1; and imaging control processing, performing imaging control suitable for the subject selected as the target subject in the selection processing.
[0465] The program as described above allows the above-described imaging control device 22 to be implemented in the imaging apparatus 1 such as the digital video camera 1A, the digital still camera 1B having a video shooting function, or the mobile terminal 1C such as a smartphone.
[0466] The program as described above may be recorded in advance in a HDD as a recording medium built in a device such as a computer device, a ROM in a microcomputer including a CPU, or the like.
[0467] Alternatively, the program may be temporarily or permanently stored (recorded) on a removable recording medium such as a floppy disk, a compact disk read only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium may be provided as a so-called software package.
[0468] Furthermore, the program as described above can be installed from a removable recording medium to a personal computer or the like, or downloaded from a download site via a network such as a local area network (LAN) or the Internet.
[0469] In addition, the program as described above is suitable for widely providing the imaging control device according to the embodiment. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., a personal computer, etc. can be used as the imaging control device of the present disclosure.
[0470] Note that the effects described in this specification are merely examples and non-limiting, and other effects may be included.
[0471] In addition, the description of the embodiment described in this description is only an example, and the present technology is not limited to the above-mentioned embodiment. Therefore, it goes without saying that in addition to the above-mentioned embodiment, various changes can be made according to design, etc. without departing from the technical idea of the present technology.
[0472] The present technology may also have the following configurations.
[0473] (1) An imaging control device comprising:
[0474] an identification unit that identifies an introduction object and an object introducer who introduces the introduction object as photographed objects, respectively, based on captured image data obtained by an imaging unit of an imaging device;
[0475] A selection unit that selects one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; and
[0476] An imaging control unit performs imaging control suitable for the subject selected as the target subject by the selection unit.
[0477] (2) The imaging control device according to (1) above, wherein
[0478] The selection unit selects one of the introduction object and the object introducer as a target subject based on the positional relationship between the introduction object and the object introducer.
[0479] (3) The imaging control device according to (1) above, wherein
[0480] The selection unit selects one of the introduction object and the object introducer as a target subject based on the positional relationship between the introduction object and the imaging device.
[0481] (4) The imaging control device according to any one of (1) to (3) above, wherein
[0482] The identification unit identifies the introduction object by recognizing the introduction object based on the captured image data.
[0483] (5) The imaging control device according to any one of (1) to (4) above, wherein
[0484] The identification unit identifies a hand of an introducer of the object based on the captured image data, and identifies the introduced object based on the identification result of the hand.
[0485] (6) The imaging control device according to (5) above, wherein
[0486] The identification unit imaginarily identifies the hand as an introduction object as a substitute for the real object to be introduced.
[0487] (7) The imaging control device according to (5) or (6) above, wherein
[0488] The identification unit identifies the introduced object based on the state of the hand.
[0489] (8) The imaging control device according to any one of (5) to (7) above, wherein
[0490] The selection unit selects one of the introduction object and the object introducer as a target subject based on the introduction object and a positional relationship between the introduction object and the object introducer depending on a state of a hand of the object introducer.
[0491] (9) The imaging control device according to (7) or (8) above, wherein
[0492] The hand state is the state in which the hand of the object introducer is in contact with the introduction object.
[0493] (10) The imaging control device according to (7) or (8) above, wherein
[0494] The hand state is a state in which the hand of the object introducer points to the introduced object.
[0495] (11) The imaging control device according to (1) above, wherein
[0496] The selection unit selects one of the introduction object and the object introducer as a target subject based on a distance relationship as a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device.
[0497] (12) The imaging control device according to (11) above, wherein
[0498] The distance relationship describes the distance between the object and the imaging device.
[0499] (13) An imaging control device according to (11) above, wherein
[0500] The distance relationship is the distance between the object introducer and the introduced object.
[0501] (14) An imaging control device according to (11) above, wherein
[0502] The distance relationship is the distance between the object introducer, the introduced object, and the imaging device.
[0503] (15) An imaging control device according to (11) above, wherein
[0504] The selection unit detects the distance relationship based on a ratio of an area of at least one of the introduction object or the object introducer to an entire frame of the captured image data.
[0505] (16) The imaging control device according to any one of (1) to (15) above, further comprising:
[0506] A presentation control unit performs presentation control for presenting a difficult control state in which imaging control is difficult to an introducer of the object in a case where the distance between the imaging device and the introduction object is smaller than a predetermined value.
[0507] (17) The imaging control device according to any one of (1) to (16) above, further comprising:
[0508] The association control unit performs association control for associating metadata related to the selection result of the selection unit with the captured image data.
[0509] (18) An imaging device comprising:
[0510] Imaging unit;
[0511] an identification unit that identifies an introduction object and an object introducer who introduces the introduction object based on the captured image data obtained by the imaging unit;
[0512] A selection unit that selects one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; and
[0513] An imaging control unit performs imaging control on the subject selected as the target subject by the selection unit.
[0514] (19) An imaging control method, comprising:
[0515] identification processing of identifying an introduction object and an object introducer who introduces the introduction object as photographed objects, respectively, based on captured image data obtained by an imaging unit of an imaging device;
[0516] A selection process of selecting one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; and
[0517] The imaging control process performs imaging control suitable for the subject selected as the target subject in the selection unit.
[0518] (20) A program for causing an imaging control device to execute the following processing:
[0519] identification processing of identifying an introduction object and an object introducer who introduces the introduction object as photographed objects, respectively, based on captured image data obtained by an imaging unit of an imaging device;
[0520] A selection process of selecting one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; and
[0521] The imaging control process performs imaging control suitable for the subject selected as the target subject in the selection unit.
[0522] Reference designation list
[0523] 1 Imaging equipment
[0524] 11Optical System
[0525] 13 Imaging Unit
[0526] 14 Image Sensor
[0527] 18 Presentation Units
[0528] 22 Imaging control device
[0529] 22a Identification unit
[0530] 22b Select Unit
[0531] 22c Imaging Control Unit
[0532] 22d Presentation Control Unit
[0533] 22e associated control unit
[0534] 61 lots
[0535] 62 Face
[0536] 70 Products
Claims
1. An imaging control device, comprising: an identification unit that identifies an introduction object and an object introducer who introduces the introduction object as photographed objects, respectively, based on captured image data obtained by an imaging unit of an imaging device; A selection unit, which selects one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; as well as an imaging control unit that performs imaging control suitable for the subject selected as the target subject by the selection unit, wherein the selection unit selects one of the introduction object and the object introducer as the target subject based on a distance relationship as a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device, and The distance relationship is the distance between at least any two of the object introducer, the introduced object, and the imaging device.
2. The imaging control device according to claim 1, wherein The identification unit identifies the introduction object by recognizing the introduction object based on the captured image data.
3. The imaging control device according to claim 1, wherein The identification unit identifies a hand of an introducer of the object based on the captured image data, and identifies the introduced object based on the identification result of the hand.
4. The imaging control device according to claim 3, wherein The identification unit imaginarily identifies the hand as an introduction object as a substitute for the real object to be introduced.
5. The imaging control device according to claim 3, wherein The identification unit identifies the introduced object based on the state of the hand.
6. The imaging control device according to claim 5, wherein The hand state is the state in which the hand of the object introducer is in contact with the introduction object.
7. The imaging control device according to claim 5, wherein The hand state is a state in which the hand of the object introducer points to the introduced object.
8. The imaging control device according to claim 1, wherein The selection unit detects the distance relationship based on a ratio of an area of at least one of the introduction object or the object introducer to an entire frame of the captured image data.
9. The imaging control device according to claim 1, further comprising: A presentation control unit performs presentation control for presenting a difficult control state in which imaging control is difficult to an introducer of the object in a case where the distance between the imaging device and the introduction object is smaller than a predetermined value.
10. The imaging control device according to claim 1, further comprising: The association control unit performs association control for associating metadata related to the selection result of the selection unit with the captured image data.
11. An imaging device comprising: Imaging unit; an identification unit that identifies an introduction object and an object introducer who introduces the introduction object based on the captured image data obtained by the imaging unit; A selection unit, which selects one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; as well as an imaging control unit that performs imaging control on the subject selected as the target subject by the selection unit, wherein the selection unit selects one of the introduction object and the object introducer as the target subject based on a distance relationship as a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device, and The distance relationship is the distance between at least any two of the object introducer, the introduced object, and the imaging device.
12. An imaging control method, comprising: identification processing of identifying an introduction object and an object introducer who introduces the introduction object as photographed objects, respectively, based on captured image data obtained by an imaging unit of an imaging device; A selection process of selecting one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; as well as an imaging control process of performing imaging control suitable for the subject selected as the target subject by the selection process, wherein the selection process includes selecting one of the introduction object and the object introducer as the target subject based on a distance relationship as a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device, and The distance relationship is the distance between at least any two of the object introducer, the introduced object, and the imaging device.
13. A program product for causing an imaging control device to execute the following processing: identification processing of identifying an introduction object and an object introducer who introduces the introduction object as photographed objects, respectively, based on captured image data obtained by an imaging unit of an imaging device; A selection process of selecting one of the introduction object and the object introducer as a target subject based on a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device; as well as an imaging control process of performing imaging control suitable for the subject selected as the target subject by the selection process, wherein the selection process includes selecting one of the introduction object and the object introducer as the target subject based on a distance relationship as a positional relationship between at least any two of the introduction object, the object introducer, and the imaging device, and The distance relationship is the distance between at least any two of the object introducer, the introduced object, and the imaging device.
