Image processing apparatus, control method of image processing apparatus, program and storage medium

Abstract

To enable setting an optimum focusing object.SOLUTION: A control method of an image processing apparatus comprises: a recognition step of recognizing a person from an image signal obtained by an imaging device; a detection step of detecting a first operation and a second operation or first audio and second audio; a marker detection step of detecting a marker from an image signal obtained by the imaging device; and a selection step of selecting either one of a person recognized by the recognition step or a marker detected by the marker detection step as an object to adjust a focal state. In the selection step, when the first operation or the first audio is detected, the person recognized by the recognition step is selected, and when the second operation or the second audio is detected, the marker detected by the marker detection step is selected.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, a program, and a storage medium. [Background technology]

[0002] In recent years, video distribution by individuals has become common on video distribution sites. In video distribution by individuals, the distributor who distributes the video often acts as both the performer who appears on the screen to introduce a product, etc., and the person who films it. In such cases, it is difficult for the distributor to operate the camera while filming, and there is a problem that it is difficult to focus on the target intended by the distributor. Therefore, it is conceivable to place a marker near the intended target in advance, and when filming, focus is adjusted based on detection information of the marker in the image.

[0003] In the cited document 1, the actual shape and size of the marker are compared with the shape and size of the marker in the real-time captured image to calculate the actual distance between the marker and the imaging unit, and the focus position for focusing the imaging unit on the object is determined. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO2015 / 141185 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in video distribution by an individual, for example in a product introduction video, it is often desirable to focus not only on the position of the marker but also on people such as the performers (distributors).

[0006] The conventional technology disclosed in the above-mentioned patent documents does not describe the priority order when both a marker and a person are in the video. Patent Document 1 did not consider the situation where both a marker and a person are in the image. In video distribution by individuals, such as videos introducing products, it is often desired to focus not only on the position of the marker but also on the performers.

[0007] Therefore, an object of the present invention is to make it possible to set an optimal focus target. [Means for solving the problem]

[0008] A technical feature of the present invention includes a recognition step of detecting a person from an image signal obtained by an imaging element, a motion detection step of detecting first and second motions of the person recognized by the recognition step, a marker detection step of detecting a marker from the image signal obtained by the imaging element, and a selection step of selecting either the person recognized by the recognition step or the marker detected by the marker detection step as a target for adjusting a focus state, wherein the selection step selects the person recognized by the recognition step when a first motion is detected by the motion detection step, and selects the marker detected by the marker detection step when a second motion is detected by the motion detection step.

[0009] As another technical feature, the system includes a recognition step of recognizing a person from an image signal obtained by an imaging element, a sound detection step of detecting first and second sounds, a marker detection step of detecting a marker from the image signal obtained by the imaging element, and a selection step of selecting either the person recognized by the recognition step or the marker detected by the marker detection step as a target for adjusting the focus state, wherein the selection step selects the person recognized by the recognition step when a first sound is detected by the sound detection step, and selects the marker detected by the marker detection step when a second sound is detected by the sound detection step. Effect of the Invention

[0010] According to the present invention, an optimal focus target can be set. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an imaging device. [Diagram 2] FIG. 2 is a diagram illustrating a processing configuration of an imaging apparatus according to the first embodiment. [Diagram 3] 5 is a flowchart showing an AF operation in the first embodiment. [Figure 4] 10 is a flowchart showing an AF operation in the second embodiment. [Diagram 5] 10 is a flowchart showing a focus mode setting operation in the second embodiment. [Figure 6] 10 is a flowchart showing a focus mode process in the second embodiment. [Figure 7] 10 is a flowchart showing a subject detection process in the second embodiment. [Figure 8] FIG. 13 is a diagram illustrating a processing configuration of an imaging apparatus according to a third embodiment. [Figure 9] 10 is a flowchart showing an AF operation in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] (First embodiment) Imaging device configuration 1 shows a configuration diagram of an imaging device 1 as an example of an image processing device to which the present invention can be applied. The imaging device 1 is a digital camera having a lens unit 100 and a camera body 200, the lens unit 100 being configured to be detachable from the camera body 200, or a digital camera in which the lens unit 100 and the camera body 200 are configured as an integral unit. The camera body 200 can be connected to a server device 300 on a network by wireless communication or wired communication. The server device 300 is a video distribution server having a video distribution function.

[0014] The lens unit 100 constitutes the photographing optical system of the imaging device 1. The lens unit 100 includes an aperture 11, an image stabilization lens group 12, a focus / zoom lens group 13, and the like, and can guide an optical image of a subject to the camera body 200.

[0015] Camera body 200 includes an image sensor 21 that photoelectrically converts an optical image formed by lens unit 100 to generate an image signal, and a mechanical shutter 22 that adjusts the exposure time for exposing image sensor 21. Camera body 200 controls aperture 11 and lens groups 12, 13 of lens unit 100 based on the setting values ​​of a plurality of setting items, and also controls the drive timing of image sensor 21 and the shutter speed of mechanical shutter 22 to capture an image with appropriate exposure.

[0016] Camera body 200 is equipped with rear display unit 23 capable of displaying images captured by imaging element 21, various settings for shooting with the camera, and the like. Rear display unit 23 is configured with a display device such as a liquid crystal panel or an organic EL, and is provided on the rear part of camera body 200 on the side opposite lens unit 100.

[0017] It should be noted that if the image sensor 21 has an electronic shutter function capable of adjusting the exposure time by controlling the signal accumulation time and the signal readout time, there is no need for the mechanical shutter 22. In addition, if the image sensor 21 has both the mechanical shutter 22 and the electronic shutter function, the mechanical shutter 22 is in a fully open state when adjusting the exposure time with the electronic shutter.

[0018] The camera body 200 includes an electric circuit 20. The electric circuit 20 includes an arithmetic processing circuit 20a, a memory circuit 20b, an image processing circuit 20c, an image compression circuit 20d, a drive control circuit 20g, and the like.

[0019] The arithmetic processing circuit 20a includes a processor such as a CPU or MPU that performs various arithmetic processing for controlling the operations of the lens unit 100 and the camera body 200. The arithmetic processing circuit 20a controls each part of the lens unit 100 and the camera body 200 by executing a program stored in the storage unit 29. The program here includes a program that performs the control processing of this embodiment.

[0020] The memory circuit 20b is used as a work memory for expanding a program read from the storage unit 29, a buffer memory for temporarily storing image data captured by the imaging element 21, and an image display memory for the rear display unit 23.

[0021] The image processing circuit 20c converts the image signal generated by the imaging element 21 into digital data and performs various image processing. The image data output from the image processing circuit 20c is output to the rear display unit 23, or compressed into a predetermined data format by the image compression circuit 20d and output to the storage unit 29 for recording.

[0022] The image compression circuit 20d compresses and encodes the image data output from the image processing circuit 20c into a predetermined data format to generate an image file.

[0023] The drive control circuit 20g controls the drive circuits and actuators (not shown) based on the results of the arithmetic processing by the arithmetic processing circuit 20a, and controls the aperture 11 of the lens unit 100, the lens groups 12 and 13, and the mechanical shutter 22 of the camera body 200.

[0024] The camera body 200 includes an operation input unit 28 such as a switch, a button, or a touch panel that accepts user operations. In this embodiment, the operation input unit 28 includes a shutter switch that instructs preparation for shooting or the start of shooting. By lightly pressing the shutter switch to the first step, a so-called "half press," operations such as autofocus processing, autoexposure processing, and autowhite balance processing are started. Furthermore, by pressing the shutter switch deeply from the half press to the second step, a so-called "full press," the mechanical shutter 22 or the electronic shutter function of the image sensor 21 is activated. Then, a series of shooting processing operations from reading a signal from the image sensor 21 to writing image data to the storage unit 29 is started. As the operation input unit 28, a switch that allows the user to turn on or off a video distribution function by the server device 300 described later may be provided.

[0025] The camera body 200 includes a communication unit 25. The communication unit 25 includes an interface circuit for communicatively connecting the camera body 200 to an external device via a network such as the Internet. The camera body 200 can transmit and receive data to and from an external device connected to a wired or wireless network via the communication unit 25. The camera body 200 can control the communication unit 25 to output image data processed by an image processing circuit to a server device 300 on the network.

[0026] The camera body 200 includes an audio input unit 27. The audio input unit 27 includes a microphone or the like, converts input audio into an electrical signal, and outputs it to the electric circuit 20 as audio data. The audio data output to the electric circuit 20 is added to image data and output to the storage unit 29 for recording. In this embodiment, the audio input unit 27 inputs audio uttered by the user and outputs the audio data to the electric circuit 20. The audio input unit 27 may be built into the camera body 200, or may be connected to an external terminal (not shown).

[0027] The camera body 200 includes a storage unit 29 such as a memory card or a hard disk. The storage unit 29 stores programs executed by the arithmetic processing circuit 20a. The storage unit 29 also records image files compressed into a predetermined format by the image compression circuit 20d, or reads out image files that have already been recorded. The storage unit 29 may be detachable from the camera body 200, or may be built into the camera body 200.

[0028] Next, the configuration and functions of the video distribution server 300 of this embodiment will be described.

[0029] The video distribution server 300 includes a control unit 30, a communication unit 31, and a streaming processing unit 32.

[0030] The control unit 30 includes a processor such as a CPU or an MPU that performs various arithmetic processing to control the operation of the video distribution server 300. The control unit 30 controls each part of the video distribution server 300 by executing a predetermined program. The program here includes a program that performs the video distribution processing of this embodiment.

[0031] The communication unit 31 is connected to the communication unit 25 of the camera body 200 via a network, and is capable of transmitting and receiving data to and from the camera body 200 and external devices. The communication unit 31 outputs image data transmitted from the communication unit 25 of the camera body 200 to the streaming processing unit 32.

[0032] The streaming processing unit 32 creates an image for distribution based on the image data transmitted from the communication unit 25, and transmits it to the communication unit 31. The image data that has been subjected to the streaming process is transmitted via the communication unit 31 to a device on the viewer side (not shown in FIG. 1).

[0033] Processing configuration of the imaging device in the first embodiment FIG. 2 is a diagram conceptually illustrating an example of the processing configuration of the electric circuit 20 in the first embodiment.

[0034] As shown in FIG. 2, the electrical circuit 20 includes an image acquisition unit 101, a marker detection unit 102, a first distance calculation unit 103, a person detection unit 104, a second distance calculation unit 105, a priority determination unit 106, a parameter determination unit 107, and a focus setting unit 108.

[0035] Each of these processing units is stored as a program in the memory circuit 20b, and is realized by the arithmetic processing circuit 20a executing the program.

[0036] The image acquisition unit 101 sequentially acquires images from the image sensor 21 in real time.

[0037] The marker detection unit 102 detects a marker from a real-time captured image acquired by the image acquisition unit 101. Here, the marker is a sticker on which a predetermined pattern, shape, and color are printed. The marker detection unit 102 stores the shape, color, information, and the like of the marker in advance, and detects the marker from the captured image based on this information. For this marker detection, a known image recognition method such as the technology disclosed in JP 2021-27544 A is used. The marker of this embodiment may have a shape and color that can be detected from the captured image, and the specific form is not limited. In addition, if there are multiple markers in the captured image, multiple markers may be detected.

[0038] The first distance detection unit 103 detects the distance between the marker and the image sensor 21 based on image information related to the marker detected by the marker detection unit 102. The image information related to the marker indicates the shape, size, etc. in the real-time image. The first distance detection unit 103 can calculate the distance between the marker and the image sensor 21 by comparing the shape and size information related to the marker stored in advance with the detected image information. The method of calculating the distance by the first distance detection unit 103 is not particularly limited as long as at least one of the shape and size information and the image information related to the marker is used.

[0039] The person detection unit 104 recognizes people from the real-time image acquired by the image acquisition unit 101. The person detection unit 104 stores shape information, color information, etc. of people's faces and bodies in advance, and recognizes people from the captured image based on this stored information. A known image recognition method is used for this person detection.

[0040] The second distance detection unit 105 detects the distance between the person and the image sensor 21, based on the real-world information and image information about the person recognized by the person detection unit 102. A known technique, such as the technique disclosed in JP-A-2007-329784, is used to detect the distance between the person and the image sensor 21.

[0041] The priority order determination unit 106 determines which of the people or markers the imaging unit will focus on. Which object is given priority will be described later.

[0042] The parameter determination unit 107 determines focus parameters for adjusting the focus state of the target using the distance information calculated by the first distance calculation unit 103. The focus parameters are an aperture value for adjusting the focus position and the depth of field, etc. The focus position can be expressed by, for example, the distance between the lens and the image sensor, the focal length, etc., depending on the configuration for adjusting the focus state in the imaging unit 101. A known method can be used as the parameter determination method by the parameter determination unit 150.

[0043] The focus setting unit 108 sets the focus parameters determined by the parameter determining unit 107 in the drive control circuit 20g.

[0044] The voice detection unit 109 detects a specific voice from the voice inputted by the voice input unit 27 .

[0045] The action detection unit 110 detects a specific gesture (action) from the real-time captured image acquired by the image acquisition unit 101.

[0046] Note that the first distance detection unit 103 and the second distance detection unit 105 may obtain defocus information using an image plane phase difference AF method and detect the distance between the target to be focused on and the image sensor 21 based on the defocus information. Also, the focus parameters may be set using the defocus information instead of the distance.

[0047] AF Operation in the First Embodiment The imaging control method in the first embodiment will be described below with reference to FIG.

[0048] The imaging control method in the first embodiment includes a plurality of steps shown in FIG. 3, from step S301 to step S315.

[0049] FIG. 3 is a flowchart showing an example of the operation of the electric circuit 20 in the first embodiment.

[0050] In the following description, each step is performed by a processing circuit 20a which is part of the electrical circuit.

[0051] When the product introduction mode is started by an input operation of the operation unit input unit 28, first, in step S301, the image acquisition unit 11 of the electric circuit 20 successively acquires real-time images from the imaging element 21. Then, in step S302, the person detection unit 104 recognizes people from the acquired real-time captured images.

[0052] Next, in step S303, the calculation processing circuit 20a determines whether or not the marker detection mode has been instructed in advance by an input operation of the operation unit input section 28. If it has been instructed, the process proceeds to S304, and if it has not been instructed, the process proceeds to S312.

[0053] In step S304, the marker detection unit 102 of the electric circuit 20 detects a marker from the real-time image acquired in step S301. Step S304 may be executed every time a real-time image is acquired, or may be executed at a predetermined interval.

[0054] In step S305, it is determined whether a marker is detected in the real-time image. If a marker is detected, the process proceeds to step S306, and if not, the process proceeds to step S312.

[0055] In step S306, the arithmetic processing circuit 20a determines whether the marker detection mode is the marker priority mode, the voice mode, or the gesture mode. The marker detection mode is specified in advance by the operation unit input unit 28. If the marker priority mode is selected, the process proceeds to step S311. If the voice mode is selected, the process proceeds to step S307. If the gesture mode is selected, the process proceeds to step S309.

[0056] In step S307, the priority order determination unit 106 determines whether the voice detection unit 109 has detected a first voice from the voice input by the voice input unit 27. The first voice is, for example, a voice indicating the product to be introduced or a voice by the distributor indicating the start of the product introduction. If the first voice is detected, the process proceeds to step S311, and if not, the process proceeds to step S308.

[0057] In step S308, the priority order determination unit 106 determines whether the voice detection unit 109 has detected a second voice from the voice input by the voice input unit 27. The second voice is, for example, a voice by the distributor indicating the end of the product introduction. If the second voice has been detected, the process proceeds to step S311, and if not, the process proceeds to step S312.

[0058] In step S309, the priority order determination unit 106 determines whether or not a first gesture (second action) has been detected from the image by the action detection unit 110. The first gesture is, for example, a pointing gesture or a hand-out gesture that is made at the start of a product introduction. If the first gesture has been detected, the process proceeds to step S311, and if not, the process proceeds to step S309.

[0059] In step S310, the priority order determination unit 106 determines whether or not a second gesture (first action) has been detected from the image by the action detection unit 110. The second gesture is, for example, a gesture of stopping pointing or a gesture of moving the hand back when the product introduction ends. If the second gesture has been detected, the process proceeds to step S312, and if not, the process proceeds to step S311.

[0060] In step S311, first distance detection unit 103 calculates the distance between the marker detected in step S305 and image sensor 21 based on the image information of the marker. If there are multiple markers, priority order determination unit 106 determines the marker for which the distance from image sensor 21 is to be calculated, according to the content of the gesture detected in step S309. For example, a marker near the location pointed to by the finger in the gesture is determined to be given priority over other markers.

[0061] In step S312, it is determined whether or not a person has been recognized from the image by the person detection unit 104. If a person has been detected, the process proceeds to step S313, and if not, the process proceeds to step S314.

[0062] In step S313, second distance detection unit 103 calculates the distance between the person detected in step S313 and image sensor 21 based on image information about the person. In step S314, arithmetic processing circuit 20a detects a subject other than the markers and the person, and sets it as a main subject. Then, based on image information about the main subject, calculates the distance between the main subject and image sensor 21. The main subject is a subject other than the markers and the face, for example, a subject located near the center of the image.

[0063] In step S315, the parameter determination unit 107 determines a focus parameter for adjusting the focus state using the distance calculated in any one of the series of processes. Then, the arithmetic processing circuit 20a of the electric circuit 20 sets the determined focus parameter in the drive control circuit 20g.

[0064] By processing in this manner, when a first sound is detected, the focus is on a marker, and when a second sound is detected, the focus is on a person, allowing the broadcaster to freely switch the subject on which they wish to focus.

[0065] After focusing on the marker, the area where the marker exists is interpolated using the surrounding image data, and the image is retouched to make it look like the marker does not exist. The retouched image is sent to the video distribution server via the communication unit. Meanwhile, the display unit 23 displays the image before retouching, or displays the image after retouching with a focus frame inserted as an indicator of the target for adjusting the focus state at the retouched area. In this way, the video viewer at the video distribution destination can see a natural image without stickers, and the distributor can easily see the point where the focus is set, that is, the location of the marker. In addition, information on the timing of switching can be recorded in the recording unit and used later to determine the timing at which the product is introduced when editing the video.

[0066] Second Embodiment The second embodiment of the present invention will be described in detail below. The same parts as those in the first embodiment are indicated by the same reference numerals, and the description thereof will be omitted.

[0067] AF Operation in the Second Embodiment FIG. 4 is a flowchart showing an example of the operation of the electric circuit 20 in the second embodiment.

[0068] In step S2101, the arithmetic processing circuit 20a performs a focus mode setting process, the details of which will be described with reference to FIG.

[0069] In step S2102, the arithmetic processing circuit 20a performs focus mode processing, the details of which will be described with reference to FIG.

[0070] In step S2103, the arithmetic processing circuit 20a performs subject detection processing, the details of which will be described with reference to FIG.

[0071] Focus Mode Setting Operation in the Second Embodiment FIG. 5 is a flowchart showing an operation example of the focus mode setting process in the second embodiment.

[0072] In step S2201, the arithmetic processing circuit 20a determines whether or not the focus mode has been instructed by the user. If so, the process proceeds to step S2202, and if not, the process proceeds to step S2207.

[0073] In step S2202, the calculation processing circuit 20a judges whether or not the user has instructed to perform marker shape identification processing. If so, the process proceeds to step S2203, and if not, the process proceeds to step S2204.

[0074] In step S2203, the arithmetic processing circuit 20a determines the shape and color of the marker detected by the marker detection unit 102. If the shape of the marker indicates the front position mode, proceed to step S2205. If the shape of the marker indicates the rear position mode, proceed to step S2206. If the shape of the marker indicates the intermediate position mode, proceed to step S2207. The front position mode is a first mode in which the focus is adjusted to a distance closer to the imaging device 1 than the detected marker (front focus). The rear position mode is a second mode in which the focus is adjusted to a distance farther from the imaging device 1 than the detected marker (back focus). The intermediate position mode is a third mode in which the focus is adjusted to the distance of the detected marker (in-focus).

[0075] In step S2204, the first, second, and third modes can be set by user instructions, and the arithmetic processing circuit 20a determines the focus mode setting instructed by the user. If the user instruction is the front position mode, the process proceeds to step S2205. If the user instruction is the rear position mode, the process proceeds to step S2206. If the user instruction is the intermediate position mode, the process proceeds to step S2207.

[0076] In step S2205, the calculation processing circuit 20a sets the focus mode to the front position mode.

[0077] In step S2206, the calculation processing circuit 20a sets the focus mode to the rear position mode.

[0078] In step S2207, the calculation processing circuit 20a sets the focus mode to the intermediate position mode.

[0079] Focus Mode Processing in the Second Embodiment FIG. 6 is a flowchart showing an operation example of the focus mode setting process in the second embodiment.

[0080] In step S2301, the arithmetic processing circuit 20a determines the focus mode set in S2101. If the focus mode is the front position mode, the process proceeds to step S2302. If the focus mode is the rear position mode, the process proceeds to step S2303. If the focus mode is the intermediate position mode, the focus mode process ends.

[0081] In step S2302, the calculation processing circuit 20a stores the distance calculated in step S311 in the memory circuit 20b as the pre-change distance, and then changes the distance in a direction closer to the imaging device 1. The amount of change may be calculated based on a numerical value designated by the user.

[0082] In step S2303, the calculation processing circuit 20a stores the distance calculated in step S311 in the memory circuit 20b as the pre-change distance, and then changes the distance in a direction away from the imaging device 1. The amount of change may be calculated based on a numerical value designated by the user.

[0083] Subject detection processing in the second embodiment FIG. 7 is a flowchart showing an operation example of the focus mode setting process in the second embodiment.

[0084] In step S2401, the arithmetic processing circuit 20a searches for the main subject from the image acquired from the image acquisition unit 101.

[0085] In step S2402, the calculation processing circuit 20a determines whether or not a main subject is present. If present, the subject detection process ends. If not present, the process proceeds to step S2403.

[0086] In step S2403, the parameter determination unit 107 determines the focus parameters based on the pre-change distance stored in the memory circuit 20b in steps S2302 and S2303. After that, the calculation processing circuit 20a sets the determined focus parameters in the drive control circuit 20g.

[0087] As described above, the imaging device 1 of this embodiment can adjust the focus to a predetermined distance in front of or behind the marker even if the marker cannot be directly attached to the product, making it possible to reliably focus on the product.

[0088] (Third embodiment) The third embodiment of the present invention will be described in detail below. The same parts as those in the first embodiment are indicated by the same reference numerals, and the description thereof will be omitted.

[0089] Processing configuration of an imaging device according to the third embodiment FIG. 8 is a diagram conceptually illustrating an example of the processing configuration of an electric circuit 3101 in the third embodiment.

[0090] The electric circuit 3101 differs from the electric circuit 20 in the first embodiment in the configurations of the marker detection unit 3102 and the priority order determination unit 3103. The marker detection unit 3102 detects markers from real-time captured images acquired by the image acquisition unit 101. In this embodiment, the number of detected markers is further calculated. The priority order determination unit 3103 determines which of the people or markers the imaging unit will focus on. Which object is given priority will be described later.

[0091] AF operation in the third embodiment Hereinafter, an imaging control method according to the third embodiment will be described with reference to FIG.

[0092] The same configurations, operations, and processes as those in the first embodiment are denoted by the same reference numerals in the drawings, and the description thereof will be omitted.

[0093] In step S3201, a predetermined number to be compared with the number of detected markers is set. The user sets the predetermined number by operating operation input unit 28. In step S3202, marker detection unit 3102 calculates the number of markers detected in the real-time image. If the number of detected markers is equal to or greater than the predetermined number, the process proceeds to step S306, and if it is less than the predetermined number, the process proceeds to step S312.

[0094] In the present embodiment, an example has been shown in which the predetermined number is set based on a user instruction in step S3201. However, the predetermined number may also be automatically set by a predetermined marker number setting unit (not shown) based on the size and number of markers, the size of a subject to be focused on, and the shooting scene.

[0095] The predetermined number is set to a value smaller than the number of markers placed. When setting the predetermined number according to the number of markers, for example, if there are many markers, the predetermined number is made larger. If there are few markers, the predetermined number is made smaller. The predetermined number is also set according to how easily the markers can be detected. For example, if the markers or the subject to be focused on are large and many markers can be placed, the predetermined number is made larger. If the markers or the subject to be focused on are small, the predetermined number is made smaller. Also, in a shooting scene with multiple people, the markers may be hidden, so the predetermined number is made smaller.

[0096] In this embodiment, an example in which one type of predetermined number is set has been shown, but two or more types of predetermined numbers may be set and the priority order determination unit 3103 may make a determination according to the number of detected markers. Depending on the number of detected markers, it is determined whether to determine the priority order based on the detection result of the motion detection unit 110. Alternatively, it is determined whether to determine the priority order based on the detection result of the voice detection unit 109.

[0097] For example, a first predetermined number, a second predetermined number, and a third predetermined number are set. If the number of detected markers is greater than the first predetermined number, the second predetermined number, and the third predetermined number, the priority is determined based on the detection result of the marker detection unit 3102. If the number of detected markers is greater than the first predetermined number, the second predetermined number, and less than the third predetermined number, the priority is determined based on the detection result of the motion detection unit 110. If the number of detected markers is greater than the first predetermined number and less than the second predetermined number and the third predetermined number, the priority is determined based on the detection result of the voice detection unit 109. If the number of detected markers is less than the first predetermined number, the second predetermined number, and the third predetermined number, the priority is determined based on the detection result of the person detection unit 104. By setting a plurality of predetermined numbers as described above and determining the priority according to the number of detected markers, the detection mode can be selected by hiding or revealing the markers with the user's hand.

[0098] In addition, although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0099] In the above-mentioned embodiment, the present invention has been described as being applied to the digital camera 100, but the present invention is not limited to this example and can be applied to any display control device that can control image processing. In other words, the present invention can be applied to mobile phone terminals, portable image viewers, PCs, printer devices equipped with viewfinders, home appliances with display units, digital photo frames, projectors, tablet PCs, music players, game consoles, electronic book readers, and the like.

[0100] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program codes. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0101] 100 Lens section 11 Aperture 12 Image stabilization lenses 13 Focus / zoom lenses 200 Camera body 20 Electrical Circuits 21 Image sensor 22 Mechanical Shutter 23 Rear display 25 Camera body communication section 27 Audio input section 28 Operation input section 29 Memory section 300 Server device 30 Control section 31 Server device side communication unit 32 Streaming Processing Unit

Claims

1. recognition means for recognizing a person from an image signal obtained by the imaging element; a motion detection means for detecting first and second motions of the person recognized by the recognition means; a marker detection means for detecting a marker from an image signal obtained by the imaging element; a selection means for selecting either the person recognized by the recognition means or the marker detected by the marker detection means as an object for adjusting the focus state, wherein the selection means selects the person recognized by the recognition means when a first movement is detected by the movement detection means, and selects the marker detected by the marker detection means when a second movement is detected by the movement detection means.

2. The image processing device described in Claim 1, characterized in that the selection means selects the person when a gesture indicating the end of product introduction is detected as the first action, and selects the marker when a gesture indicating the start of product introduction is detected as the second action.

3. The image processing device described in Claim 2, characterized in that the selection means selects a marker to be the subject of focus adjustment from among a plurality of markers based on the position of the gesture detected by the movement detection means.

4. recognition means for recognizing a person from an image signal obtained by the imaging element; a voice detection means for detecting first and second voices; a marker detection means for detecting a marker from an image signal obtained by the imaging element; a selection means for selecting either the person recognized by the recognition means or the marker detected by the marker detection means as an object for adjusting the focus state, wherein the selection means selects the person recognized by the recognition means when a first voice is detected by the voice detection means, and selects the marker detected by the marker detection means when a second voice is detected by the voice detection means.

5. The image processing device described in Claim 4, characterized in that the selection means selects the person when a sound indicating the start of a product introduction is detected as the first sound, and selects the marker when a sound indicating the end of the product introduction is detected as the second sound.

6. 5. The image processing apparatus according to claim 1, further comprising a recording unit for recording, as information, the timing at which the selecting unit selects the target.

7. a processing means for retouching the image so that the marker detected by the marker detection means does not appear to exist; 5. The image processing apparatus according to claim 1, further comprising a transmitting unit for transmitting the retouched image signal output by said processing unit to a video distribution server.

8. 8. The image processing device according to claim 7, further comprising a display control means for controlling the display of an indicator indicating a target for adjusting the focus state at a position where a marker of the retouched image signal output by the processing means was present.

9. 2. The image processing device according to claim 1, further comprising a mode setting means capable of setting a first mode for adjusting the focus state so that the marker is in front focus, a second mode for adjusting the focus state so that the marker is in back focus, and a third mode for adjusting the focus state so that the marker is in focus.

10. 10. The image processing apparatus according to claim 9, wherein a specific mode can be set from the first, second, and third modes in response to a user operation.

11. the marker detection means detects at least one of the shape and color of the marker; 10. The image processing apparatus according to claim 9, wherein the mode setting means sets the first, second, or third mode in accordance with at least one of the shape and color of the marker.

12. 10. The image processing device according to claim 9, wherein, in the first mode, if there is no object to be focused on at a predetermined distance from the marker that is in front focus, the focus state is adjusted so that the marker is in focus.

13. 10. The image processing device according to claim 9, wherein, in the second mode, if there is no object to be focused on at a predetermined distance from the marker such that the object is in back focus, the focus state is adjusted so that the marker is in focus.

14. 5. The image processing apparatus according to claim 1, wherein the selection means selects the person recognized by the recognition means when the marker detection means does not detect a predetermined number of markers or more.

15. 15. The image processing device according to claim 14, wherein the predetermined number is set according to the size of the subject and the photographing scene.

16. a recognition step of detecting a person from an image signal obtained by the imaging element; a motion detection step of detecting first and second motions of the person recognized by the recognition step; a marker detection step of detecting a marker from an image signal obtained by the imaging element; a selection step of selecting, as a target for adjusting a focus state, either the person recognized in the recognition step or the marker detected in the marker detection step, the selection step selects the person recognized by the recognition step when a first movement is detected by the movement detection step, and selects the marker detected by the marker detection step when a second movement is detected by the movement detection step.

17. a recognition step of recognizing a person from an image signal obtained by the imaging element; a sound detection step of detecting first and second sounds; a marker detection step of detecting a marker from an image signal obtained by the imaging element; a selection step of selecting, as a target for adjusting a focus state, either the person recognized in the recognition step or the marker detected in the marker detection step, an image processing method, characterized in that in the selection step, if a first voice is detected in the voice detection step, a person recognized in the recognition step is selected, and if a second voice is detected in the voice detection step, a marker detected in the marker detection step is selected.

18. A program for causing a computer to function as each of the means of the image processing device according to claim 1 or 4.

19. 10. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1 or 4.

Images