Information processing apparatus, imaging device, control method, and program

Abstract

To suitably specify a detection result of detection of a subject.SOLUTION: An information processing apparatus has: first acquisition means that acquires a picked-up image; first detection means that detects an area corresponding to a first subject in the picked-up image as a detection area candidate; second detection means that detects an area corresponding to a second subject different from the first subject in the picked-up image; second acquisition means that acquires supplementary information related to the distribution of the first subject relative to the second subject in the picked-up image; and determination means that determines a detection area of the first subject from the detection area candidate on the basis of the supplementary information and a result of detection performed by the second detection means.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an information processing device, an imaging device, a control method and a program, and more particularly to a technique for detecting a specific subject. [Background technology]

[0002] Some imaging devices such as digital cameras have a function to detect a specific subject captured in the angle of view (subject detection function). The subject detection function can be used, for example, in live sports broadcasts, and the detection results are used to control focus on the ball and players around the ball, and to control framing to keep these subjects captured within the imaging range (Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-066889 A [Patent Document 2] JP 2020-036168 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the subject detection function may output an erroneous detection result when a subject with a similar appearance is included within the angle of view. For example, in a ball sport scene, a detector that detects a ball may erroneously detect an object with a nearly circular shape, such as a player's head (especially a shaved head) or a uniform pattern, as a ball. As a result, the focus control and framing of Patent Documents 1 and 2 may not operate properly.

[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an information processing device, an imaging device, a control method, and a program that can specify a detection result in which a subject is preferably detected. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the information processing device of the present invention is characterized by having a first acquisition means for acquiring an captured image, a first detection means for detecting an area in the captured image corresponding to a first subject as a detection area candidate, a second detection means for detecting an area in the captured image corresponding to a second subject different from the first subject, a second acquisition means for acquiring auxiliary information regarding the distribution of the first subject relative to the second subject in the captured image, and a determination means for determining the detection area of ​​the first subject from the detection area candidates based on the detection result of the second detection means and the auxiliary information. Effect of the Invention

[0007] According to the present invention, with such a configuration, it is possible to specify a detection result in which a subject is preferably detected. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a hardware configuration of a digital camera system according to an embodiment and a modification of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining the configuration of an image sensor 22 according to an embodiment and a modification of the present invention; [Diagram 3] FIG. 1 is a block diagram illustrating a functional configuration related to subject detection according to an embodiment and a modified example of the present invention. [Figure 4] FIG. 13 is a diagram for explaining erroneous detection of a ball region according to an embodiment and a modified example of the present invention; [Diagram 5] FIG. 13 is a diagram for explaining a difference in detection results between a first detection unit and a second detection unit according to an embodiment and a modification of the present invention. [Figure 6] FIG. 1 is a diagram illustrating adjustment of an exclusion region according to the first embodiment of the present invention; [Figure 7] FIG. 1 is a diagram for explaining determination of a detection area candidate that is an erroneous detection according to the first embodiment of the present invention; [Figure 8] 1 is a flowchart illustrating an output process executed by the main body 100 according to the first embodiment of the present invention. [Figure 9]FIG. 11 is a diagram illustrating posture information according to a second embodiment of the present invention; [Figure 10] 11 is a flowchart illustrating an output process executed by the main body 100 according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] In the embodiment described below, an example in which the present invention is applied to a lens-interchangeable digital camera (digital camera system) capable of detecting two types of subjects in a captured image as an example of an imaging device will be described. However, the present invention can be applied to any device capable of outputting detection results of two types of subjects in a captured image. Such devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. In addition, such devices are not limited to those having an imaging function, and also include devices that obtain captured images from devices having an imaging function and output detection results based on the captured images. That is, in the following description, the information processing device that realizes the present invention is not limited to a lens-interchangeable imaging device, and is not necessarily an imaging device.

[0010] <Hardware configuration of digital camera system> 1 is a block diagram showing the hardware configuration of a digital camera system according to this embodiment. The digital camera system is made up of a body 100 of a digital camera with interchangeable lenses, and a lens unit 150 that is detachable from the body 100.

[0011] The lens unit 150 has a communication terminal 6 that comes into contact with and electrically connects to a communication terminal 10 provided on the main body 100 when the lens unit 150 is attached to the main body 100. Power is supplied from the main body 100 to the lens unit 150 through the communication terminal 10 and the communication terminal 6. In addition, the lens system control circuit 4 of the lens unit 150 and the system control unit 50 of the main body 100 can communicate bidirectionally through the communication terminal 10 and the communication terminal 6.

[0012] The lens group 103 of the lens unit 150 is an imaging optical system composed of a plurality of lenses including a movable lens. The movable lens includes at least a focus lens. Depending on the lens unit 150, the lens group 103 may further include one or more of a variable magnification lens, a blur correction lens, etc. The AF drive circuit 3 includes a motor, an actuator, etc. that drive the focus lens. The focus lens is driven by the lens system control circuit 4 controlling the AF drive circuit 3. The aperture drive circuit 2 includes a motor actuator, etc. that drives the aperture 102. The opening amount of the aperture 102 is adjusted by the lens system control circuit 4 controlling the aperture drive circuit 2.

[0013] The mechanical shutter 101 is driven by the system control unit 50 and adjusts the exposure time of the image sensor 22 during image capture. The mechanical shutter 101 is kept fully open during video capture.

[0014] The imaging element 22 is a sensor that captures an optical image incident through a lens unit 150 such as a CCD or CMOS sensor. The imaging element 22 outputs an analog image signal by photoelectrically converting the optical image formed on the imaging surface. The imaging element 22 has a plurality of pixels arranged two-dimensionally, and each pixel is provided with one microlens, one color filter, and one or more photoelectric conversion units. In this embodiment, each pixel is provided with a plurality of photoelectric conversion units, and each pixel is configured to be able to read out a signal for each photoelectric conversion unit. By configuring the pixels in this way, it is possible to generate a captured image, a parallax image pair, and an image signal for phase difference AF from a signal read out from the imaging element 22.

[0015] FIG. 2(a) is a diagram showing a schematic diagram of the correspondence between the exit pupil of the lens unit 150 and each photoelectric conversion unit when a pixel of the image sensor 22 has two photoelectric conversion units.

[0016] Two photoelectric conversion units 201a and 201b provided in a pixel share one color filter 252 and one microlens 251. Light that has passed through a partial region 253a of the exit pupil (region 253) is incident on the photoelectric conversion unit 201a, and light that has passed through a partial region 253b of the exit pupil is incident on the photoelectric conversion unit 201b.

[0017] Therefore, for a pixel included in a certain pixel region, an image formed by a signal read from the photoelectric conversion unit 201a and an image formed by a signal read from the photoelectric conversion unit 201b form a parallax image pair. The parallax image pair can be used as image signals (A image signal and B image signal) for phase difference AF. Furthermore, a normal image signal (captured image) can be obtained by adding the signal read from the photoelectric conversion unit 201a and the signal read from the photoelectric conversion unit 201b for each pixel.

[0018] In this embodiment, each pixel of the image sensor 22 functions as a pixel for generating a signal for phase difference AF (focus detection pixel) and as a pixel for generating a normal image signal (imaging pixel). However, some pixels of the image sensor 22 may be configured as focus detection pixels, and other pixels may be configured as imaging pixels. FIG. 2B shows an example of a correspondence relationship between the focus detection pixels and an area 253 of the exit pupil through which incident light passes. As shown in the figure, an opening 254 is provided in the focus detection pixels provided as some pixels, so that the photoelectric conversion unit 201 functions in the same manner as the photoelectric conversion unit 201b in FIG. 2A. Therefore, by distributing the focus detection pixels shown in FIG. 2B and another type of focus detection pixels that functions in the same manner as the photoelectric conversion unit 201a in FIG. 2A over the entire image sensor 22, a focus detection area of ​​substantially any location and size can be set. 2(a) and (b) is a configuration in which an image sensor for obtaining an image for recording is used as a sensor for phase difference AF, but the present invention can be implemented regardless of the AF method, such as other AF that can set a focus detection area of ​​any size and position. For example, the present invention can be implemented even with a configuration that uses contrast AF. When only contrast AF is used, each pixel has one photoelectric conversion unit.

[0019] 1, the A / D converter 23 is used to convert the analog image signal output from the imaging element 22 into a digital image signal (captured image data). The A / D converter 23 may be configured integrally with the imaging element 22. In the following description, the image data captured by the imaging element 22 and converted by the A / D converter 23 may be simply referred to as a captured image.

[0020] The image data (RAW image data) output by the A / D converter 23 is processed by the image processing unit 24 as necessary, and then stored in the memory 32 via the memory control unit 15. The memory 32 is used as a buffer memory for temporarily storing image data and audio data, and as a video memory for the display unit 28.

[0021] The image processing unit 24 applies predetermined image processing to the image data, generates signals and image data, and acquires and / or generates various information. The image processing unit 24 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function, or may be configured to realize a specific function by a processor such as a DSP executing software.

[0022] Here, the image processing applied by the image processing unit 24 includes pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, etc. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is processing for interpolating values ​​of color components not included in image data, and is also called demosaic processing. Correction processing includes white balance adjustment, processing for correcting image luminance, processing for correcting optical aberration of the lens unit 150, color correction processing, etc. Detection processing includes detection and tracking processing of characteristic regions (for example, face regions and human body regions), person recognition processing, etc. Data processing processing includes scaling processing, encoding and decoding processing, header information generation processing, etc. Evaluation value calculation processing includes calculation processing of a pair of image signals for phase difference AF, evaluation values ​​for contrast AF, evaluation values ​​used for automatic exposure control, etc. Note that these are examples of image processing that the image processing unit 24 can perform, and do not limit the image processing performed by the image processing unit 24. Moreover, the evaluation value calculation process may be performed by the system control unit 50.

[0023] The D / A converter 19 generates an analog signal suitable for display on the display unit 28 from the image data for display stored in the memory 32, and supplies the generated analog signal to the display unit 28. The display unit 28 has, for example, a liquid crystal display device, and performs display based on the analog signal from the D / A converter 19 on its display surface.

[0024] By continuously capturing video (capture control) and displaying the captured video (display control), the display unit 28 can function as an electronic viewfinder (EVF). A video displayed to make the display unit 28 function as an EVF is called a live view image. The display unit 28 may be provided inside the main body 100 so that it can be observed through an eyepiece, or it may be provided on the surface of the housing of the main body 100 so that it can be observed without using an eyepiece. The display unit 28 may be provided both inside the main body 100 and on the surface of the housing.

[0025] The system control unit 50 is, for example, a CPU (also called an MPU or microprocessor). The system control unit 50 loads a program stored in a non-volatile memory 56 into a system memory 52 and executes it to control the operation of the main body 100 and the lens unit 150 and realize the functions of the camera system. The system control unit 50 controls the operation of the lens unit 150 by transmitting various commands to the lens system control circuit 4 by communication through the communication terminals 10 and 6.

[0026] The non-volatile memory 56 stores programs executed by the system control unit 50, various setting values ​​of the camera system, image data of a GUI (Graphical User Interface), etc. The system memory 52 is a main memory used when the system control unit 50 executes programs. The data (information) stored in the non-volatile memory 56 may be rewritable.

[0027] As part of its operation, the system control unit 50 performs automatic exposure control (AE) processing based on evaluation values ​​generated by the image processing unit 24 or by itself, and determines the shooting conditions. For example, the shooting conditions for still image shooting are shutter speed, aperture value, and sensitivity. The system control unit 50 determines one or more of the shutter speed, aperture value, and sensitivity according to the AE mode that is set. The system control unit 50 controls the aperture value (aperture) of the aperture mechanism of the lens unit 150. The system control unit 50 also controls the operation of the mechanical shutter 101.

[0028] In addition, the system control unit 50 drives the focus lens of the lens unit 150 based on the evaluation value or defocus amount generated by the image processing unit 24 or itself, and performs autofocus detection (AF) processing to focus the lens group 103 on a subject within the focus detection area.

[0029] The system timer 53 is a built-in clock and is used by the system control unit 50 .

[0030] The operation unit 70 has a plurality of input devices (buttons, switches, dials, etc.) that can be operated by the user. Some of the input devices of the operation unit 70 have names according to the functions assigned to them. For convenience, the shutter button 61, the mode change switch 60, and the power switch 72 are illustrated separately from the operation unit 70, but are included in the operation unit 70. If the display unit 28 is a touch display equipped with a touch panel, the touch panel is also included in the operation unit 70. The operation of the input devices included in the operation unit 70 is monitored by the system control unit 50. When the system control unit 50 detects an operation of an input device, it executes processing according to the detected operation.

[0031] The shutter button 61 has a first shutter switch 62 that is turned on when pressed halfway and outputs a signal SW1, and a second shutter switch 64 that is turned on when pressed all the way and outputs a signal SW2. When the system control unit 50 detects the signal SW1 (ON of the first shutter switch 62), it executes a preparatory operation for still image shooting. The preparatory operation includes AE processing and AF processing. When the system control unit 50 detects the signal SW2 (ON of the second shutter switch 64), it executes a still image shooting operation (image capturing and recording operation) according to the shooting conditions determined by the AE processing.

[0032] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage to each unit including the recording medium 200 for the necessary period.

[0033] The power supply unit 30 is composed of a battery, an AC adapter, or the like. The I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. Data files such as captured images and sounds are recorded on the recording medium 200. The data files recorded on the recording medium 200 can be read out via the I / F 18 and played back via the image processing unit 24 and the system control unit 50.

[0034] The communication unit 54 realizes communication with an external device by at least one of wireless communication and wired communication. Images captured by the imaging element 22 (captured images; including live view images) and images recorded on the recording medium 200 can be transmitted to an external device via the communication unit 54. In addition, image data and various other information can be received from an external device via the communication unit 54.

[0035] The body orientation detection unit 55 detects the orientation of the body 100 with respect to the direction of gravity. The body orientation detection unit 55 may be an acceleration sensor or an angular velocity sensor. The system control unit 50 can record orientation information corresponding to the body orientation detected by the body orientation detection unit 55 during shooting in a data file that stores image data obtained during the shooting. The orientation information can be used, for example, to display a recorded image in the same orientation as when it was shot.

[0036] <Functional Configuration> FIG. 3A is a block diagram showing the functional configuration related to specific subject detection, which is realized by the main body 100 of this embodiment.

[0037] As described above, the image processing unit 24 is configured to be able to detect a specific subject from a captured image, and the main body 100 of this embodiment is configured to be able to perform various operational controls based on the detection result of the subject. More specifically, the main body 100 performs various controls so that the area (characteristic area) where the subject is detected becomes an appropriate image.

[0038] Such operational control can include, for example, automatic focus control (AF) for focusing on the characteristic region, automatic exposure control (AE) for properly exposing the characteristic region, etc. Operational control can also include processes such as automatic white balance for properly adjusting the white balance of the characteristic region, and automatic flash light amount adjustment for properly adjusting the brightness of the characteristic region.

[0039] Furthermore, the operation control based on the detection result is not limited to such image quality related control, but can also include automatic tracking control to keep the subject of the characteristic region within the angle of view, for example, in a mode in which the camera platform to which the main body 100 is attached can be controlled. In addition, the information on the characteristic region can also be used to specify the region for detecting the state of the subject. For example, when the face region of the main subject is detected as the characteristic region, the image of the region can be analyzed to obtain the state of the subject, such as whether or not red-eye is occurring, whether or not the eyes are closed, and what facial expression the subject is making (such as smiling).

[0040] In this way, the operational control performed based on the subject detection result may be any one of them, and is not limited to the present invention, so is excluded from the functional configuration.

[0041] As shown in the figure, the functional configuration related to subject detection in this embodiment includes an acquisition unit 301, a first detection unit 302, a second detection unit 303, an information acquisition unit 304, and a detection area selection unit 305. These functional configurations can be realized by the image processing unit 24 and the system control unit 50. Fig. 3(a) shows an aspect in which the image processing unit 24 realizes the functional configurations of the acquisition unit 301, the first detection unit 302, and the second detection unit 303, and the lens unit 150 realizes the functional configurations of the information acquisition unit 304 and the detection area selection unit 305.

[0042] The acquisition unit 301 acquires a captured image to be subjected to subject detection. In this embodiment, the acquisition unit 301 acquires a captured image captured by the image sensor 22 and subjected to A / D conversion by the A / D converter 23. The captured image may be input as a live view image. Upon acquiring the captured image, the acquisition unit 301 transmits the captured image to the first detection unit 302 and the second detection unit 303.

[0043] The first detection unit 302 and the second detection unit 303 perform a detection process to detect different subjects from the input captured image. Hereinafter, the subject to be detected by the first detection unit 302 may be referred to as the first subject, and the subject to be detected by the second detection unit 303 may be referred to as the second subject. The detection process performed by the first detection unit 302 and the second detection unit 303 may employ any known method. Since any detection method may be employed by each detection unit, it will not be described in this specification, but the detection process may be, for example, a pattern matching process for the captured image. Alternatively, in an aspect in which each detection unit is a learning model that is machine-learned from the image of the subject to be detected by the detection unit, the detection process may be a process for obtaining an inference result inferred by the learning model.

[0044] Each detection unit detects a region determined to exhibit a predetermined characteristic as a feature region, and outputs information about each detected feature region, such as its position in the captured image, its size, and the reliability of the detection result (the likelihood that the image of the feature region is an image of the subject to be detected).

[0045] In the following, for ease of understanding of the invention, it is assumed that the first detection unit 302 detects an area corresponding to one or more first subjects in the captured image and outputs information about the area as the detection result. Also, it is assumed that the second detection unit 303 detects an area corresponding to one or more second subjects in the captured image and outputs information about the area as the detection result. However, the present invention does not exclude cases where each detection unit does not detect any characteristic area depending on the captured image.

[0046] In this embodiment, the scene captured by the digital camera system is a ball game scene, and the operation control of the main body 100 based on the detection result is performed based on the ball used in the game. Therefore, the first detection unit 302 is configured to detect an area corresponding to the ball from the input captured image by assuming that the first object to be detected is the ball. On the other hand, objects having a circular shape like a ball may exist other than the ball in the game scene. That is, the detection result by the first detection unit 302 may include objects other than the ball that are erroneously detected as the ball. For example, the heads of players, etc., and numbers and logos on the uniforms and bibs worn by players may appear as circles in the captured image, and therefore may be erroneously detected by the first detection unit 302 as areas corresponding to the ball.

[0047] For this reason, in the digital camera system of this embodiment, in order to detect the human body of a player or the like, the second detection unit 303 is configured to detect the head of a person as the second subject to be detected from the input captured image. That is, in order to exclude erroneously detected areas from the areas corresponding to the ball detected by the first detection unit 302, the detection result of the second detection unit 303 is used in this embodiment.

[0048] Here, in this embodiment, the second detection unit 303 has a relatively lower false detection rate than the first detection unit 302, and the detection result of the second detection unit 303 is regarded as positive to determine whether the area corresponding to the ball detected by the first detection unit 302 is a false detection. Therefore, the area corresponding to the ball detected by the first detection unit 302 is temporarily treated as a candidate (detection area candidate) of the area to be detected as a ball (detection area). Then, the detection area selection unit 305 described later determines the detection area (ball area) to be finally used for the operation control of the main body 100 from the detection area candidates based on the detection result of the area corresponding to the head detected by the second detection unit 303 (head area). In other words, in the main body 100 of this embodiment, when determining the ball area, the detection result of the second detection unit 303, which has a relatively low false detection rate, is trusted, and some of the detection area candidates detected by the first detection unit 302 are excluded, thereby improving the detection performance of the ball area.

[0049] In particular, in an aspect in which the first detection unit 302 and the second detection unit 303 are configured to enhance their detection performance by being individually designed and trained, it is difficult to make the false detection rates of these detectors approximately equal. Therefore, as in this embodiment, by using the detection result of one detector (the second detection unit 303) whose false detection rate is relatively low, it is possible to indirectly reduce the false detection rate of the other detector (the first detection unit 302) whose false detection rate is relatively high.

[0050] The information acquisition unit 304 acquires information to assist the detection area selection unit 305 in determining the detection area (ball area) from the detection area candidates. Although details will be described later, in this embodiment, the information acquisition unit 304 acquires, as auxiliary information, competition information related to the competition (sport event) captured using a digital camera system. The competition information may be prepared in advance for each competition, for example, and stored in the non-volatile memory 56 or the like. The information acquisition unit 304 acquires competition information related to the competition based on, for example, information on the recognition result obtained by performing scene recognition based on the captured image. Note that the competition does not need to be identified using the recognition result based on the captured image, and may be identified based on, for example, an operation input by the user specifying the competition.

[0051] The detection area selection unit 305 determines and outputs a detection area (ball area) from the detection area candidates output from the first detection unit 302 based on the detection result of the second detection unit 303 and the auxiliary information (game information) acquired by the information acquisition unit 304. The information on the detection area determined by the detection area selection unit 305 is used for controlling the operation of the main body 100 as described above.

[0052] <<Outline of ball area determination>> Below, an overview of the process of determining the ball region performed by detection region selection unit 305 will be described. Below, the process of determining the ball region will be described using as an example a state in which a captured image is obtained in which a player 401 and a ball 402 are captured within the angle of view, as shown in Fig. 4. Note that the captured image in Fig. 4 is an example of an image obtained by capturing an image of a soccer game scene.

[0053] 4, detection area candidates 411 to 413 detected in the captured image by the first detection unit 302 are indicated by dashed frames. As shown in the figure, only the detection area candidate 411 is a region that truly corresponds to the ball, but the first detection unit 302 erroneously detects the head of the player 401 and the number on the bib worn by the player 401 as detection area candidates 412 and 413, respectively.

[0054] The first detection unit 302 also outputs information on the reliability (evaluation value indicating the likelihood that it is a ball) for each of the detection area candidates. Here, it is assumed that the reliability output for the detection area candidates 411, 412, and 413 are 100, 105, and 60, respectively. In this case, in a conventional method in which a reliability threshold is set and a detection area candidate exceeding the threshold is detected as a ball area, not only the detection area candidate 411 corresponding to the ball 402 but also the detection area candidate 412 corresponding to the head of the player 401 is determined as a ball area. In a mode in which the precondition that there is only one ball in a game is met, the detection area candidate with the highest reliability is determined as the ball area, and therefore the detection area candidate 412 is determined as the ball area. In such a case, the detection area candidate 411 (only) corresponding to the ball 402 is not determined as the ball area, and therefore the preferable operation control of the main body 100 is not realized.

[0055] In contrast, Fig. 5 shows the same captured image with the detection results by the second detection unit 303. In Fig. 5, in order to clearly show the difference between the detection results of each detection unit, the detection results by the first detection unit 302 (detection area candidate group shown in Fig. 4) are shown in addition to the detection results by the second detection unit 303. The second detection unit 303 may output multiple detection results, but in the example of Fig. 5, it is assumed that the head area 501 with the highest reliability is output. The head area 501 is shown in the figure by a dashed line frame, and the area corresponding to the head of the player 401 is detected.

[0056] In this way, the area corresponding to the head of the player 401 that was erroneously detected by the first detection unit 302 is determined to be the head area 501 by the second detection unit 303, and there is a discrepancy between the detection results of the first detection unit 302 and the second detection unit 303. That is, the first detection unit 302 and the second detection unit 303 detect the same area (the area corresponding to the head of the player 401) as an area corresponding to the subject of the detection target, but one detects the subject correctly and the other erroneously detects the subject. Therefore, for an area that both detection units detect as the subject of the detection target, the detection result of the second detection unit 303, which has a relatively low erroneous detection rate, is treated as correct, so that the detection area candidate detected for the same area by the first detection unit 302 can be excluded.

[0057] <Exclusion area setting> On the other hand, since the first detection unit 302 and the second detection unit 303 each detect an area corresponding to a different subject, the two detection units do not necessarily detect an area in the exact same position or size. In addition, since the detection area candidate 413 that detected the bib number is a detection of an object different from the head area 501, even if it is detected with high reliability, it cannot be excluded by the detection result of the second detection unit 303. For this reason, the detection area selection unit 305 of this embodiment sets an area (human body area) corresponding to the human body of the player 401 by expanding the head area 501 detected by the second detection unit 303, and determines the detection area candidate that overlaps with the area as a false detection. In other words, the human body area is used to exclude the detection area candidate detected by the first detection unit 302 from the options when the detection area is finally determined. Hereinafter, the human body area will be referred to as an excluded area. Hereinafter, the setting of the excluded area performed by the detection area selection unit 305 will be described in more detail.

[0058] When the head region 501 is detected as shown in FIG. 5, the detection region selection unit 305 assumes that a body region exists below the head region 501 in the imaging device, and sets an exclusion region 601 including the head region 501 and the body region as shown in FIG. 6(a). Here, the position and size of the body region are determined in advance relative to the head region 501. The exclusion region may be determined by determining the head region 501 and the body region and then connecting them, but in the example of FIG. 6(a), it is determined directly from the head region 501. That is, the detection region selection unit 305 determines the exclusion region 601 by expanding the head region 501 by a certain magnification in both the horizontal and vertical directions with a predetermined position as a reference. In the example of FIG. 6(a), the exclusion region is set to include at least the entire head region 501.

[0059] By defining an exclusion region that integrates the head region and the torso region in this way, it is possible to reduce the number of times to determine whether each detection region candidate described below overlaps with the exclusion region. In other words, if the head region and the torso region are defined as independent exclusion regions, it is necessary to determine whether each of them overlaps with each detection region candidate, but by defining the body region of the same person as a single exclusion region, the determination can be made only once. As a result, it is possible to reduce the amount of calculations performed by the detection region selection unit 305 in relation to the determination of the detection region.

[0060] <Adjustment of exclusion area> Incidentally, in a captured image of a competitive scene, the area corresponding to the ball does not always appear as an area not continuous with the human body area. For example, in soccer, the ball mainly comes into contact with the legs of the human body at the timing of dribbling, shooting, etc. In addition, in volleyball, the ball mainly comes into contact with the arms and palms of the human body at the timing of serving, tossing, spiking, etc. In other words, in an exclusion area determined at a fixed ratio with respect to the head area detected by the second detection unit 303, there is a possibility that the ball area cannot be appropriately set at the timing when the ball comes into contact with the human body.

[0061] That is, by uniformly setting the exclusion area based on the detected head area, in a captured image in which the area corresponding to the ball is discontinuous with the human body area, it is possible to suitably exclude detection area candidates that are erroneously detected. On the other hand, in a captured image in which the area corresponding to the ball is continuous with the human body area, the detection area candidates that truly detect the ball are excluded by overlapping with the exclusion area, so it is preferable to adjust the exclusion area so as not to include a place where the ball and the human body may come into contact. However, as described above, the part of the human body that the ball may come into contact with differs depending on the sport, so the detection area selection unit 305 of this embodiment adjusts the fixed exclusion area set based on the detection result of the second detection unit 303 according to which sport is being played. Therefore, the detection area selection unit 305 adjusts the exclusion area based on the sport information acquired by the information acquisition unit 304. That is, the detection area selection unit 305 changes the setting conditions of the exclusion area for the relevant head area according to the sport information.

[0062] The sport information referred to here may simply be information identifying the sport, or may be information indicating the position of the ball in the sport relative to the human body, or may be information indicating which part of the human body the exclusion region should be included in.

[0063] For example, in a soccer game scene as shown in Fig. 6(a), if the ball and a human body overlap within the angle of view, the ball may be located mainly at the position of the human body's legs. If the exclusion region 601 is set based on the head region and the body region as described above, the leg region is not included in the exclusion region 601, and therefore the detection region selection unit 305 controls not to make adjustments based on the game information. This is because, if the exclusion region 601 does not include the leg region, even in a captured image where the region corresponding to the ball is continuous with the human body region, the region corresponding to the ball does not overlap with the exclusion region 601, and the corresponding detection region candidate is not determined to be a false detection.

[0064] Also, for example, in a volleyball game scene as shown in Fig. 6(b), if the ball and a human body overlap within the angle of view, the ball may be located at the position of the human body's chest. Therefore, if the exclusion region 601 is set based on the head region and the body region as described above, the chest region is included in the exclusion region 601, and the detection region selection unit 305 adjusts it based on the game information so that the body region is not included. This is because if the exclusion region 601 is set to include the body region, as in soccer, in a captured image in which the region corresponding to the ball is continuous with the human body region, the region corresponding to the ball will overlap with the exclusion region 601, and the corresponding detection region candidate will be determined to be a false detection.

[0065] In this way, the detection area selection unit 305 can avoid excluding a detection area candidate that actually detects the ball by adjusting the exclusion area set based on the head area in accordance with the game information input from the information acquisition unit 304. In the example of Fig. 6(b), the adjusted exclusion area 602 is set so as not to include the body area.

[0066] (Exclusion of False Positive Detection Area Candidates) Hereinafter, an outline of the process of excluding a detection area candidate determined to be a false detection, which is performed by the detection area selection unit 305 based on the exclusion area, will be described. As described above, whether or not to exclude a detection area candidate is determined based on whether the detection area candidate overlaps with any of the exclusion areas. FIG. 7 illustrates an example of the relationship between the detection area candidate detected by the first detection unit 302 related to the determination and the exclusion area. FIG. 7(a) illustrates an example in which an exclusion area 710 is set with a size equivalent to the head area, like a volleyball shown in FIG. 6(b), and FIG. 7(b) illustrates an example in which an exclusion area 720 is set including the head area and the body area, like a soccer ball shown in FIG. 6(a). In both figures, a detection area candidate 701 in which a head is falsely detected as a ball is shown, and an area where the detection area candidate 701 overlaps with each exclusion area is shown by a thick line as an overlap area 711 or an overlap area 721.

[0067] In a mode in which the detection region candidate 701 and the exclusion region 710 are of equal size as in Fig. 7(a), the overlapping region 711 is evaluated, for example, by IoU (Intersection of Union). The detection region selection unit 305 can determine that there is overlap when the obtained evaluation value exceeds a threshold. In the case of Fig. 7(a), the IoU can be derived by dividing the area of ​​the overlapping region 711 by the area of ​​the region obtained by combining the detection region candidate 701 and the exclusion region 710.

[0068] On the other hand, in a mode in which the sizes of the detection area candidate 701 and the exclusion area 720 are not equal as in FIG. 7(b), the area of ​​the area obtained by combining the detection area candidate 701 and the exclusion area 720 becomes too large compared to the overlap area 721, and therefore the overlap cannot be appropriately evaluated by IoU. That is, in a mode in which the size of the exclusion area is changed according to the competition information, the value by which the area of ​​the overlap area 721 is divided may vary, and therefore IoU may not be able to correctly determine whether the detection area candidate overlaps with the exclusion area. Therefore, the detection area selection unit 305 of this embodiment uses an evaluation value obtained by dividing the area of ​​the overlap area by the area of ​​the detection area candidate 701 to determine whether the detection area candidate 701 overlaps with the exclusion area. More specifically, the detection area selection unit 305 determines that the detection area candidate overlaps with the exclusion area when the evaluation value exceeds a threshold value.

[0069] That is, the detection area selection unit 305 determines whether each of the detection area candidates obtained for the captured image overlaps with the exclusion area, and if so, excludes the detection area candidate from the candidates for the ball area. Then, the detection area selection unit 305 determines the most reliable detection area candidate as the detection area (ball area) from among the detection area candidates remaining after excluding the detection area candidates overlapping with the exclusion area. In this way, it is possible to reduce the number of detection area candidates that are erroneous detections being output as the detection result for the ball area, and it is possible to cause the main body 100 to perform suitable operation control.

[0070] Output Processing The output process executed to output the detection result of the ball region in the main body 100 of this embodiment having such a configuration will be described in detail with reference to the flowchart in Fig. 8. The process corresponding to the flowchart can be realized by the system control unit 50 reading out the corresponding processing program stored in, for example, the non-volatile memory 56, and developing and executing it in the system memory 52. ​​This output process will be described as being started, for example, when the main body 100 is set to an imaging mode in which the ball is automatically focused on as the main subject and started up. In the following description, each step of this output process will be described as being mainly performed by the functional configuration shown in Fig. 3, but as described above, each functional configuration is realized by the system control unit 50 or the image processing unit 24.

[0071] In S801, the acquisition unit 301 acquires a captured image of a detection target of a ball region. Hereinafter, the captured image acquired in this step will be referred to as a target image.

[0072] In S802, the second detection unit 303 executes a detection process to detect a region corresponding to a human head in the target image. The second detection unit 303 also outputs information on each region detected by the detection process to the detection region selection unit 305.

[0073] In S803, the first detection unit 302 executes a detection process to detect an area corresponding to the ball in the target image. The first detection unit 302 also outputs information about each area detected by the detection process to the detection area selection unit 305 as a detection area candidate.

[0074] In S804, the information acquisition unit 304 acquires competition information related to the sport captured in the target image and outputs it to the detection area selection unit 305. The processing in this step may be performed based on the result of scene recognition performed on the target image, or may acquire the relevant competition information based on the information of the sport designated by the user.

[0075] In S805, the detection area selection unit 305 selects a head area from the area corresponding to the head detected in S802, based on the reliability.

[0076] In S806, the detection region selection unit 305 sets an exclusion region for each of the head regions selected in S805. The setting of the exclusion region in this step is performed by expanding the head region by a predetermined magnification in the horizontal and vertical directions.

[0077] In S807, the detection area selection unit 305 adjusts the exclusion area set in S806 based on the competition information.

[0078] In S808, the detection area selection unit 305 determines whether each of the detection area candidates overlaps with any of the exclusion areas.

[0079] In S809, the detection area selection unit 305 excludes the detection area candidates that were determined to overlap with any of the excluded areas in S808. The detection area selection unit 305 then determines a detection area (ball area) from the remaining detection area candidates, outputs the information, and completes this output process. For example, in the case of a competition scene in which only one ball is present, the ball area is determined to be the detection area candidate whose reliability exceeds a threshold and has the highest reliability among the remaining detection area candidates.

[0080] As described above, the information processing device of this embodiment can specify a detection result that preferably detects a desired subject. In other words, it is possible to reduce the rate at which an area corresponding to a human body is erroneously detected as a ball area. In addition, since the excluded area is adjusted according to the captured competition scene, it is possible to stably detect the ball area while narrowing down the detection area candidates.

[0081] [Variation 1] In the above embodiment, it is assumed that a body region exists below the head region, and an exclusion region corresponding to each head region is set based on the detection result of the head region, but the implementation of the present invention is not limited to this. The second detection unit 303 may further detect a region corresponding to the body so that an exclusion region that is more suited to the state of the person (player) is set. In this case, the detection region selection unit 305 can combine the head region and body region estimated to belong to the same person output from the second detection unit 303, and set the exclusion region to include these.

[0082] [Embodiment 2] In the above-described embodiment, a mode has been described in which the detection result of the second detection unit 303 is prioritized and detection area candidates detected by the first detection unit 302 are excluded, assuming that the second detection unit 303 has a lower false detection rate than the first detection unit 302. However, the present invention is not limited to this embodiment, and the difference between the false detection rates of the first detection unit 302 and the second detection unit 303 does not characterize the present invention.

[0083] <Function configuration> 3(b) is a block diagram showing a functional configuration related to specific subject detection realized in the main body 100 of this embodiment. As shown in the figure, the functional configuration related to subject detection of this embodiment further includes a posture estimation unit 311 in addition to the functional configuration shown in the first embodiment.

[0084] In this embodiment, the second detection unit 303 is configured to be able to detect regions corresponding to predetermined parts of the human body other than the head, such as the hands and feet, in addition to the region corresponding to the head of the human body, as candidates for regions corresponding to the human body (human body region candidates) from the captured image. That is, the second detection unit 303 is configured to be able to output, as the detection result, information on the head region candidate which is the region corresponding to the head of the human body, and information on the part region candidate which is the region corresponding to the predetermined part, as information on the human body region candidate.

[0085] The posture estimation unit 311 generates posture information by estimating the posture of the person corresponding to each head region candidate detected by the second detection unit 303. Here, when a part region candidate estimated to belong to the same person is detected around the head region candidate, the posture estimation unit 311 generates posture information about the human body region candidate related to the person using the information. On the other hand, when a part region candidate estimated to belong to the same person is not detected around the head region candidate, the second detection unit 303 generates posture information about the human body region candidate related to the person using only the information about the head region candidate. Therefore, the posture information generated by the posture estimation unit 311 is associated with at least one of the head region candidates, and there may be two types of information that includes information about the part region candidate and does not include information about the part region candidate.

[0086] FIG. 9 is a diagram illustrating posture information generated by posture estimation unit 311 superimposed on a captured image for comparison. In the diagram, posture information 900 distributed on the left side is posture information obtained about a ball erroneously detected as a region corresponding to a head by second detection unit 303. As shown in the diagram, posture information 900 includes only head region candidate 901 erroneously detected as a head because no part region candidate was detected in the periphery. In contrast, posture information 910 distributed on the right side is posture information obtained about a player detected by second detection unit 303. As shown in the diagram, posture information 910 includes joint information 912 corresponding to the part region candidate in addition to head region candidate 911 because a part region candidate was detected in the periphery. Joint information 912 is configured by binding the head region candidate and the joint points of the human body (including the joint points corresponding to the head) estimated based on the part region candidate. In this embodiment, the posture estimation unit 311 is described as estimating each joint point based on information about the candidate part area, but it is also possible for the second detection unit 303 to estimate the joint points, and for the posture estimation unit 311 to generate posture information based on the joint point estimation results.

[0087] The information acquisition unit 304 acquires the posture information generated in this manner as auxiliary information and outputs it to the detection area selection unit 305, and the detection area selection unit 305 performs processing related to determining the ball area by referring to the input posture information.

[0088] <<Outline of ball area determination>> An outline of the process of determining the ball region performed by the detection region selection unit 305 of this embodiment will be described below.

[0089] <Exclusion area setting> In this embodiment, since the second detection unit 303 is not necessarily determined to have a lower false detection rate than the first detection unit 302, a candidate that is likely to be a person's head is identified from the head region candidates detected by the second detection unit 303, and an exclusion region is set for that candidate. That is, the detection region selection unit 305 extracts a head region candidate that is likely to be a person's head from the head region candidates detected by the second detection unit 303, and sets an exclusion region. In this embodiment, the detection region selection unit 305 determines that a head region candidate indicates a person's head if the following two conditions are satisfied, and sets an exclusion region based on that head region candidate.

[0090] The first condition is that the pose information includes joint information. In other words, the head region candidate included in the pose information is configured as part of the joint information. This condition makes it possible to exclude head region candidates that do not have any part region candidates around them, such as a ball.

[0091] The second condition is that the posture of the person indicated by the posture information is appropriate. A posture is appropriate means, for example, that it is not an unnatural posture for a person, or that it is not a posture that the subject person (athlete) cannot take in the sport being imaged. Such posture evaluation can be performed, for example, by preparing dictionary data of appropriate postures, and based on whether or not the posture of the joint information is within a predetermined range from the state of each joint indicated by the posture registered in the dictionary data.

[0092] Therefore, detection region selection unit 305 identifies head region candidates whose corresponding posture information satisfies both of the above two conditions from among the head region candidates, and sets the candidates as head regions. Detection region selection unit 305 also excludes head region candidates whose corresponding posture information does not satisfy at least one of the above two conditions, and performs control so as not to set them as head regions.

[0093] When the head region is set in this manner, the detection region selection unit 305 sets an exclusion region based on the head region and posture information. In this embodiment, since the information of the torso region corresponding to the head region has already been obtained as posture information, the exclusion region including the region corresponding to the human body can be set in a manner that matches the image of the player more than in the first embodiment. At this time, the detection region selection unit 305 can determine which part of the human body to include in the exclusion region by referring to, for example, competition information. As described above, the region in which the ball may be present relative to the human body differs depending on the competition. Therefore, for example, when the competition is soccer, since the ball may be present mainly at the position of the legs of the human body, the detection region selection unit 305 may set the exclusion region based on the joint information excluding the legs and the head region based on the competition information. That is, the detection region selection unit 305 sets the exclusion region based on the information of the position where the ball may be present relative to the human body in the competition so as not to include the position.

[0094] <Adjustment of exclusion area> Next, a method of adjusting the exclusion region thus set according to this embodiment will be described.

[0095] In this embodiment, posture information is obtained as auxiliary information, but the posture information is not only referenced to the reliability of the head region candidate and to the setting of the excluded region. In a competition scene, a player may perform a specific preparatory movement before performing a specific action, or may adopt a specific posture while performing the specific action. In other words, by referring to the posture information, it is possible to estimate what is currently happening in the competition and where the ball is located relative to the player.

[0096] Therefore, the detection area selection unit 305 adjusts the exclusion area set to include a part of the human body according to the sport based on the sport information, based on the posture information. That is, the detection area selection unit 305 changes the setting condition of the exclusion area for the relevant head area according to the posture information. For example, if the sport is volleyball, and it is found from the posture information that the player is in a posture with both hands raised up, the premise that the ball may be present at the player's chest can be excluded. That is, as shown in FIG. 6(b), in the case of volleyball, the exclusion area is set so as not to include the position of the chest (torso area), but if it is possible to specify from the posture information that the ball is not present at the chest position, this can be expanded to include the torso area. In this way, the exclusion area can be adjusted according to the situation of the subject, so that it becomes possible to more appropriately exclude detection area candidates that have been erroneously detected.

[0097] Output Processing Hereinafter, the output process executed by the main body 100 of this embodiment to output the detection result of the ball region will be specifically described with reference to the flowchart in Fig. 10. The process corresponding to this flowchart can be realized by the system control unit 50 reading out a corresponding processing program stored in, for example, the non-volatile memory 56, and expanding and executing the program in the system memory 52. ​​This output process will be described as being started when, for example, the main body 100 is set to an imaging mode in which the ball is automatically focused on as the main subject and started up.

[0098] In the following description, each step of this output process will be described assuming that it is the functional configuration shown in Fig. 3, but as described above, each functional configuration is realized by the system control unit 50 or the image processing unit 24. In the description of the output process of this embodiment, steps that perform the same processes as those in the output process of Fig. 8 are given the same reference numbers and their description will be omitted, and only steps specific to this embodiment will be described below.

[0099] When a target image is acquired in S801, the second detection unit 303 executes a detection process in S1001 to detect a region corresponding to a human body's head and a region corresponding to a predetermined part in the target image. The second detection unit 303 outputs information on each human body region candidate detected by the detection process (information on the head region candidate and information on the part region candidate) to the posture estimation unit 311 and the detection region selection unit 305.

[0100] In S1002, posture estimation section 311 generates posture information for each head region candidate. At this time, posture estimation section 311 generates one piece of posture information by integrating head region candidates and part region candidates estimated to correspond to the same person. Furthermore, when there is no part region candidate estimated to correspond to the same person for the head region candidate, posture estimation section 311 generates posture information based only on the head region candidate.

[0101] When the detection area candidates are output in S803, the information acquisition unit 304 acquires, as auxiliary information in S1003, competition information related to the competition captured in the target image and posture information related to each head area candidate generated in S1002. Then, the information acquisition unit 304 outputs the acquired auxiliary information to the detection area selection unit 305.

[0102] In S1004, the detection area selection unit 305 refers to the posture information corresponding to each of the head area candidates, and excludes the head area candidate if the head area candidate is not configured as part of the joint information.

[0103] In S1005, detection area selection unit 305 refers to the posture information corresponding to each of the head area candidates (after the processing of S1004), and if the posture represented by the posture information is not appropriate, excludes the head area candidate.

[0104] In S1006, the detection area selection unit 305 selects the head area candidate (after the processing of S1005) as the head area.

[0105] In S1007, the detection area selection unit 305 sets an exclusion area for each head area selected in S1006. The setting of the exclusion area in this step can be performed using, for example, the competition information and the posture information corresponding to the head area.

[0106] In S1008, the detection area selection unit 305 adjusts the exclusion area set in S1007 based on the posture information corresponding to the head area included in the exclusion area, and moves the process to S808.

[0107] In this way, the information processing device of this embodiment can specify a detection result that preferably detects a desired subject. In other words, it is possible to reduce the rate at which an area corresponding to a human body is erroneously detected as a ball area. In addition, since the excluded area is adjusted according to the captured competition scene and the posture of the human body, it is possible to stably detect the ball area while narrowing down the detection area candidates.

[0108] [Variation 2] In the above-mentioned second embodiment, the posture information based on the detection result of the region corresponding to a predetermined part of the human body is used to select the head region candidate and to set and adjust the excluded region, but the implementation of the present invention is not limited to this. For example, it goes without saying that the excluded region may be adjusted by referring to the posture information together with the competition information in S807 of the output process of the first embodiment. That is, if the region in which the ball is present, which is estimated based on the posture information, is not included in the human body region, the already set excluded region can be expanded to preferably exclude erroneously detected detection region candidates such as bib numbers. Alternatively, the posture information may be used similarly when selecting the head region in S805 of the output process of the first embodiment.

[0109] [Embodiment 3] In the above-mentioned second embodiment, the position of the ball is estimated based on the posture information according to the game, but the present invention is not limited to this. For some postures, the position of the ball can be estimated regardless of the game. Therefore, even if the setting and adjustment of the excluded area are simply performed based on the detection result of the second detection unit 303 and the posture information, it is possible to preferably exclude a detection area candidate that is erroneously detected. That is, in a mode in which the detection result of the head area and the posture information of the human body corresponding to the head area are obtained, it is possible to control which part of the human body is included in the setting of the excluded area based on the position of the ball area estimated based on the posture information. In this case, the detection area selection unit 305 can exclude a detection area candidate that erroneously detects a region corresponding to the human body as a region corresponding to the ball without referring to the game information.

[0110] [Variation 3] In the above-described embodiment and modified example, a method has been described in which an exclusion area is set based on the detection result of an area corresponding to a human body from a captured image, and a ball area is determined from detection area candidates excluding detection area candidates that overlap with the exclusion area. However, the implementation of the present invention is not limited to this, and it is not necessary to immediately exclude a detection area candidate just because the detection area candidate overlaps with the exclusion area.

[0111] In captured images of competitive scenes, the ball region may often overlap with the human body region. When the human body comes into contact with the ball, such as when trapping or heading the ball in soccer or receiving the ball in volleyball, the ball region may overlap with the exclusion region. In other words, if a detection region candidate is excluded because it overlaps with the exclusion region, the ball region cannot be detected properly.

[0112] On the other hand, the operation control of the imaging device based on the detection result of the ball area in the competition scene is performed continuously, and the scene is also captured continuously. That is, in a mode in which the acquisition unit 301 sequentially acquires the captured images as frames of a moving image, it is possible to determine whether the detection area candidate should be excluded by taking correspondence between each detection area candidate between frames. The detection area selection unit 305 of this modified example manages the detection area candidates detected for the same subject between frames, and when the detection area candidate overlaps with the excluded area for a predetermined number of consecutive frames, it excludes the detection area candidate as a false detection. In this way, it is possible to avoid the exclusion of the detection area candidate that truly indicates the ball in a frame in which the ball momentarily comes into contact with a player, or the selection of an incorrect ball area, which prevents the main body 100 from performing a suitable operation control.

[0113] [Variation 4] In the above-described embodiment and modified example, the main body 100 of the digital camera system, which is an imaging device, has been described as executing an output process that outputs the detection result of the ball region, but the implementation of the present invention is not limited to this. The present invention can be realized by any information processing device that acquires a captured image, detects two types of subjects in the captured image, and determines the detection region of one of the subjects based on the detection result of the other subject and auxiliary information.

[0114] [Variation 5] In the above-described embodiment and modified example, the first object to be detected by the first detection unit 302 is a ball used in a competitive scene, but it goes without saying that the present invention is not limited to this. The first object to be detected in a competitive scene can be any equipment used in the sport, such as a shuttlecock in badminton, a puck in ice hockey, or a stone in curling.

[0115] In addition, the second subject to be detected by the second detection unit 303 has been described as a specific part such as the head, limbs, etc. of a person, but it goes without saying that the embodiment of the present invention is not limited to this. That is, the second subject may be at least a part of a human body, such as a face. Which part of the human body is to be detected may be determined, for example, for each sport.

[0116] Of course, the first subject is not limited to a human body and may be anything defined as long as the area in the captured image is a target for reference in the operational control of the main body 100, etc. Similarly, the second subject is detected to exclude subjects that may be erroneously detected as the first subject, and therefore is not limited to a part of the human body and may be, for example, clothing, an arbitrary pattern, a specific color area, etc.

[0117] [Variation 6] In the above-mentioned embodiment and modified example, the exclusion area is set mainly based on the detection result of the second detection unit 303, and then the exclusion area is further adjusted based on various auxiliary information, but the implementation of the present invention is not limited to this. That is, when setting the exclusion area, it is not essential to implement the present invention to intermediately set the area before adjustment. Therefore, the detection area selection unit 305 may set the exclusion area corresponding to the adjustment based on the detection result of the second detection unit 303 and the auxiliary information.

[0118] [Embodiment 4] In the above-described embodiment and modified example, an exclusion area is set based on the detection result and auxiliary information of the second detection unit 303, and a detection area is determined from the remaining detection area candidates after excluding detection area candidates that overlap with the exclusion area. However, the implementation of the present invention is not limited to this, and the method of excluding detection area candidates that are estimated to be erroneous detections from among the detection area candidates is not limited to this. In other words, the elimination of a detection area candidate that is estimated to be erroneous detection does not need to be performed via a determination of whether the candidate overlaps with the exclusion area, and it may be determined directly whether or not to validate the detection area candidate based on the detection result of the second subject and auxiliary information.

[0119] Therefore, the present invention only needs to detect a first subject and a second subject from an imaging device, and determine an area to be the detection area of ​​the first subject from among the detection area candidates of the first subject based on the detection result of the second subject and the auxiliary information. Therefore, the auxiliary information can be, for example, information that simply indicates the relative position of the area corresponding to the first subject with respect to the area corresponding to the second subject. The auxiliary information is information that supplements elements that may be insufficient to determine whether the detection area candidate is valid / invalid based on the detection result of the second subject, and can include information other than the above-mentioned competition information and posture information.

[0120] [Summary of the embodiment and modifications] The disclosure of this specification includes the following information processing device, imaging device, control method, and program. (Item 1) A first acquisition means for acquiring a captured image; a first detection means for detecting an area corresponding to a first object in the captured image as a detection area candidate; a second detection means for detecting an area of ​​the captured image corresponding to a second subject different from the first subject; a second acquisition means for acquiring auxiliary information regarding a distribution of the first subjects relative to the second subjects in the captured image; a determining means for determining a detection area of ​​the first subject from the detection area candidates based on a detection result of the second detection means and the auxiliary information; 13. An information processing device comprising: (Item 2) The information processing device described in item 1, characterized in that the determination means sets an exclusion area for the first subject in the captured image based on the detection result of the second detection means and the auxiliary information, and determines the detection area of ​​the first subject from the detection area candidates that do not overlap with the exclusion area. (Item 3) 3. The information processing device according to item 2, wherein the determining means sets the exclusion region to include at least a region corresponding to the second subject detected by the second detecting means. (Item 4) The auxiliary information includes competition information related to the competition being imaged, The determining means changes a setting condition of the excluded area in accordance with the competition information. 4. The information processing device according to item 2 or 3. (Item 5) the first subject is equipment used in the sport; the second subject includes at least a part of a human body; the sport information includes information indicating a position where the equipment may be present relative to a human body in the sport; The determining means sets the exclusion area so as not to include a position where the tool may be present in a region corresponding to the second subject detected by the second detecting means. 5. The information processing device according to item 4, (Item 6) the second acquisition means further acquires posture information of the human body as the auxiliary information based on a detection result of the second detection means; The information processing device further includes an adjustment unit that adjusts the exclusion area set by the determination unit in accordance with the posture information. 6. The information processing device according to item 5, (Item 7) The information processing device described in item 6, characterized in that the adjustment means estimates the area in which the tool exists in the captured image based on the posture information, and expands the exclusion area if the estimated area is not included in the area corresponding to the second subject detected by the second detection means. (Item 8) 8. The information processing device according to any one of items 5 to 7, wherein the first subject is a ball used in the game. (Item 9) the second subject includes at least a part of a human body; the second acquisition means acquires posture information of the human body as the auxiliary information based on a detection result of the second detection means; The determining means changes a setting condition of the exclusion region in accordance with the posture information. 4. The information processing device according to item 2 or 3. (Item 10) the second subject includes a head and a body part other than the head of a human body; the second detection means detects a region in the captured image corresponding to the second subject as a candidate human body region for setting the exclusion region; the second acquisition means acquires, for each of the human body region candidates, posture information of the human body based on a detection result of the second detection means as the auxiliary information; The determining means is determining whether each of the human body region candidates is a region corresponding to a human body based on the corresponding posture information; The exclusion region is set based on the human body region candidate determined to be a region corresponding to the human body. 4. The information processing device according to item 2 or 3. (Item 11) the first subject is equipment used in the sport being imaged, The second acquisition means further acquires competition information related to the competition as the auxiliary information; the sport information includes information indicating a position where the equipment may be present relative to a human body in the sport; The determining means sets the exclusion region so as not to include a position where the tool may be present in a region corresponding to the human body. 11. The information processing device according to item 10, (Item 12) 12. The information processing device according to item 11, further comprising an adjustment unit that adjusts the exclusion area set by the determination unit in accordance with the posture information. (Item 13) The information processing device described in item 12, characterized in that the adjustment means estimates the area in which the tool exists in the captured image based on the posture information, and expands the exclusion area if the area is not included in the area corresponding to the human body. (Item 14) 14. The information processing device according to any one of items 11 to 13, wherein the first subject is a ball used in the game. (Item 15) the captured image is a frame of a video; the first acquisition means sequentially acquires frames of the moving image as the captured images; When the exclusion region and the detection region candidate overlap for a predetermined number of consecutive frames, the determination means determines the detection region of the first subject by excluding the detection region candidate. 15. The information processing device according to any one of items 2 to 14. (Item 16) An imaging means; An information processing device according to any one of items 1 to 15, a control means for performing an operation control in accordance with the detection area of ​​the first subject determined by the determination means; An imaging device comprising: (Item 17) A first acquisition step of acquiring a captured image; a first detection step of detecting an area corresponding to a first object in the captured image as a detection area candidate; a second detection step of detecting an area corresponding to a second subject different from the first subject in the captured image; a second acquisition step of acquiring auxiliary information regarding a distribution of the first subjects relative to the second subjects in the captured image; a determination step of determining a detection area of ​​the first subject from among the detection area candidates based on a detection result of the second detection step and the auxiliary information; 13. A method for controlling an information processing apparatus comprising the steps of: (Item 18) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 15.

[0121] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0122] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0123] 100: main body, 24: image processing unit, 50: system control unit, 301: acquisition unit, 302: first detection unit, 303: second detection unit, 304: information acquisition unit, 305: detection area selection unit, 311: posture estimation unit

Claims

1. A first acquisition means for acquiring a captured image; a first detection means for detecting an area corresponding to a first object in the captured image as a detection area candidate; a second detection means for detecting an area of ​​the captured image corresponding to a second subject different from the first subject; a second acquisition means for acquiring auxiliary information regarding a distribution of the first subjects relative to the second subjects in the captured image; a determination means for determining a detection area of ​​the first subject from the detection area candidates based on a detection result of the second detection means and the auxiliary information; 13. An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the determination means sets an exclusion area for the first subject in the captured image based on the detection result of the second detection means and the auxiliary information, and determines the detection area of ​​the first subject from the detection area candidates that do not overlap with the exclusion area.

3. The information processing apparatus according to claim 2 , wherein the determining means sets the exclusion region so as to include at least a region corresponding to the second object detected by the second detecting means.

4. The auxiliary information includes competition information related to the competition being imaged, The determining means changes a setting condition of the excluded area in accordance with the competition information.

3. The information processing apparatus according to claim 2.

5. the first subject is equipment used in the sport, the second subject includes at least a part of a human body; the sport information includes information indicating a position where the equipment may be present relative to a human body in the sport; The determining means sets the exclusion area so as not to include a position where the tool may be present in a region corresponding to the second subject detected by the second detecting means.

5. The information processing apparatus according to claim 4.

6. the second acquisition means further acquires posture information of the human body as the auxiliary information based on a detection result of the second detection means; The information processing device further includes an adjustment unit that adjusts the exclusion area set by the determination unit in accordance with the posture information.

6. The information processing apparatus according to claim 5,

7. The information processing device according to claim 6, characterized in that the adjustment means estimates the area in which the tool exists in the captured image based on the posture information, and expands the exclusion area when the estimated area is not included in the area corresponding to the second subject detected by the second detection means.

8. The information processing apparatus according to claim 5 , wherein the first object is a ball used in the game.

9. the second subject includes at least a part of a human body; the second acquisition means acquires posture information of a human body as the auxiliary information based on a detection result of the second detection means; The determining means changes a setting condition of the exclusion region in accordance with the posture information.

3. The information processing apparatus according to claim 2.

10. the second subject includes a head and a body part other than the head of a human body; the second detection means detects a region in the captured image corresponding to the second subject as a candidate human body region for setting the exclusion region; the second acquisition means acquires, for each of the human body region candidates, posture information of the human body based on a detection result of the second detection means as the auxiliary information; The determining means is determining whether each of the human body region candidates is a region corresponding to a human body based on the corresponding posture information; The exclusion region is set based on the human body region candidate determined to be a region corresponding to the human body.

3. The information processing apparatus according to claim 2.

11. the first subject is equipment used in the sport being imaged, The second acquisition means further acquires competition information related to the competition as the auxiliary information; the sport information includes information indicating a position where the equipment may be present relative to a human body in the sport; The determining means sets the exclusion region so as not to include a position where the tool may be present in a region corresponding to the human body.

11. The information processing apparatus according to claim 10,

12. 12. The information processing apparatus according to claim 11, further comprising an adjustment unit that adjusts the exclusion area set by the determination unit in accordance with the posture information.

13. The information processing device according to claim 12 , wherein the adjustment means estimates a region in the captured image in which the tool exists based on the posture information, and expands the exclusion region when the estimated region is not included in a region corresponding to the human body.

14. The information processing device according to claim 11 , wherein the first object is a ball used in the game.

15. the captured image is a frame of a video; the first acquisition means sequentially acquires frames of the moving image as the captured images; The determining means, when the exclusion region and the detection region candidate overlap for a predetermined number of consecutive frames, determines the detection region of the first subject by excluding the detection region candidate.

3. The information processing apparatus according to claim 2.

16. An imaging means; An information processing device according to any one of claims 1 to 15; a control means for performing an operation control in accordance with the detection area of ​​the first subject determined by the determination means; An imaging device comprising:

17. A first acquisition step of acquiring a captured image; a first detection step of detecting an area corresponding to a first subject in the captured image as a detection area candidate; a second detection step of detecting an area of ​​the captured image corresponding to a second subject different from the first subject; a second acquisition step of acquiring auxiliary information regarding a distribution of the first subjects relative to the second subjects in the captured image; a determination step of determining a detection area of ​​the first subject from among the detection area candidates based on a detection result of the second detection step and the auxiliary information; 13. A method for controlling an information processing apparatus comprising the steps of:

18. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 15.

Images