Subject detection device, subject detection method, imaging device, and program

Abstract

To provide a technique for continuously focusing on a subject aimed by a photographer even if a specific object disappears when photographing a game dealing with the specific object.SOLUTION: An object detection apparatus includes an object detection unit configured to detect an object from a captured image of a game in which a specific object is dealt with, an object detection unit configured to detect the specific object from the captured image, and a setting unit configured to set a main object to be a target of an autofocus operation from a plurality of objects detected by the object detection unit. The setting unit determines whether to switch a current main object to a newly detected main object candidate when the specific object is no longer detected from the captured image even if the main object candidate is newly detected.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for determining a main subject of a tracking target. [Background technology]

[0002] In video shooting and continuous still image shooting, a tracking function is known that determines a main subject from multiple moving subjects detected by the imaging device and keeps the main subject in focus. In particular, when shooting a scene of a sports game (such as a ball) involving a specific object (such as a ball), multiple subjects may overlap, and the camera may determine that a subject not intended by the photographer is the main subject.

[0003] Patent Document 1 describes a technology that estimates the posture of a player from a video of a sport using a moving object (ball), and controls camerawork based on the player's movement identified from the posture and the speed and position of the moving object. Patent Document 2 describes a technology that determines whether a player has taken an action on the ball based on changes in the ball's trajectory, and recognizes the player who has taken the action. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-145527 [Patent Document 2] Patent No. 7289080 Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Documents 1 and 2 do not take into consideration situations where a specific object such as a ball falls outside the angle of view of the image or disappears behind another subject.

[0006] When photographing a sport involving a specific object, one condition for determining the main subject is that the subject must be near the specific object. However, if the main subject is determined based on the detection of the specific object, if the specific object disappears, the main subject cannot be determined, and the focus may be on a subject that the photographer was not aiming for.

[0007] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that allows a photographer to continue to focus on a subject he or she is aiming at when photographing a competition involving a specific object, even if the specific object disappears. [Means for solving the problem]

[0008] In order to solve the above problems and achieve the object, the subject detection device of the present invention comprises a subject detection means for detecting a subject from an image captured of a competition involving a specific object, an object detection means for detecting the specific object from the captured image, and a setting means for setting a main subject to be the target of autofocus operation from multiple subjects detected by the subject detection means, and the setting means determines whether to switch the current main subject to the newly detected main subject candidate if the specific object is no longer detected from the captured image, even if a new main subject candidate is detected. [Effects of the Invention]

[0009] According to the present invention, when photographing a competition involving a specific object, it is possible for the photographer to continue to focus on the subject he or she is aiming at even if the specific object disappears. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating the configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating the configuration of a focus detection apparatus according to an embodiment of the present invention. [Figure 3] 6 is a flowchart illustrating a main subject determination process according to the first embodiment. [Figure 4]4A to 4C are diagrams illustrating an example of posture information and object information of a subject according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating the structure of a neural network according to the first embodiment. [Figure 6] 10 is a flowchart illustrating a main subject setting process according to a ball disappearance state according to the first embodiment. [Figure 7] 10 is a flowchart illustrating a main subject setting process according to a ball disappearance state according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] Below, we will explain an example in which the subject detection device and imaging device of the present invention are applied to a digital camera with an integrated lens, but this is not limited to this example and the invention can also be applied to, for example, a film camera, a digital camera with interchangeable lenses, a digital video camera, a smartphone with a camera function, a tablet computer, a web camera such as a surveillance camera, etc.

[0013] In this embodiment, an example will be described in which an imaging device is equipped with a subject detection device, and when photographing a scene of a sport (such as a ball game) in which multiple subjects handle a specific object (such as a ball), the imaging device determines the subject (main subject) that is the target (tracking target) of the autofocus operation (AF control).

[0014] A sport is a competition in which opposing groups of players compete to score the most points by getting a ball or similar object to reach a goal or area in a set target space, and the competition space can be on land, water, or ice. Sports can also be played on either land or ice. Sports can be played using the body directly (basketball, water polo, etc.) or on a stick (hockey, lacrosse, etc.). Land-based sports include basketball, handball, hockey, polo, lacrosse, and football (soccer, rugby, American football). Water-based sports include water polo, and ice-based sports include ice hockey. Ice hockey and badminton are sports that use balls or similar objects that are not spherical or ellipsoidal, such as balls.

[0015] <Device configuration> First, the hardware configuration of the imaging device of this embodiment will be described with reference to FIG.

[0016] FIG. 1 is a block diagram illustrating the hardware configuration of an imaging apparatus according to this embodiment.

[0017] The imaging device 100 of this embodiment includes a lens unit 101 controlled by a main control unit 140. The lens unit 101 constitutes an imaging optical system that, under the control of the main control unit 140, causes the imaging unit 131 to form an optical image of the subject, which is light reflected from the subject.

[0018] The lens unit 101 includes a fixed first group lens 102, a zoom lens 103 driven by a zoom lens driving unit 104, an aperture 105 driven by an aperture driving unit 106, a shift lens 107 driven by a shift lens driving unit 108, and a focus lens 109 driven by a focus lens driving unit 110.

[0019] The zoom lens 103 moves in the optical axis direction to change the focal length and perform a zoom operation. The iris 105 changes the aperture diameter to adjust the amount of light of the subject image formed on the imaging surface of the imaging unit 131. The shift lens 107 moves in a direction perpendicular to the optical axis to change the optical axis and perform image stabilization. The focus lens 109 has a focus lens function that corrects the movement of the focal plane that accompanies the zoom operation and a compensator lens function that adjusts the focus state.

[0020] The zoom control unit 121 performs zoom control to change the focal length by controlling the motor of the zoom lens drive unit 104 to drive the zoom lens 103 under the control of the main control unit 140. The iris control unit 122 performs exposure control to adjust the opening diameter of the iris 103 and adjust the amount of light during shooting under the control of the main control unit 140 by controlling the motor of the iris drive unit 106 to drive the iris 105. The focus control unit 124 performs AF control to adjust the focus state of the subject by controlling the motor of the focus lens drive unit 110 to drive the focus lens 109 under the control of the main control unit 140.

[0021] Under the control of the main control unit 140, the image stabilization control unit 123 controls the motor of the shift lens drive unit 108 to drive the shift lens 107 in accordance with the shake of the imaging device 100, thereby performing image stabilization control to reduce the image stabilization. The drive amount of the shift lens 107 is calculated by the main control unit 140 as the image stabilization amount for canceling out the shake of the imaging device 100 detected by the shake detection unit 151. Although detection of shake of the imaging device 100 by the shake detection unit 151 and image stabilization by the shift lens 107 require movement in two axial directions, the yaw direction and the pitch direction, FIG. 1 shows only one axis for simplification. The detection result of the shake detection unit 151 is used not only to calculate the image stabilization amount of the imaging device 100, but also to determine whether the photographer is panning the imaging device 100 in the left-right direction or tilting the imaging device 100 in the up-down direction.

[0022] Each lens of the lens unit 101 is usually made up of a plurality of lenses, but for simplicity, FIG. 1 shows only one lens.

[0023] The subject image formed on the imaging surface of the imaging unit 131 by the lens unit 101 is converted into an electrical signal by the imaging unit 131. The imaging unit 131 is an image sensor equipped with photoelectric conversion elements such as CCD or CMOS that photoelectrically convert the subject image (optical image) into an electrical signal. The imaging unit 131 has photoelectric conversion elements arranged with m pixels in the horizontal direction and n pixels in the vertical direction. The image signal generated by the imaging unit 131 is subjected to predetermined signal processing by the imaging signal processing unit 132 and output as image data. This makes it possible to acquire an image on the imaging surface. For example, in the case of a setting of NTSC, FHD 60p, image data equivalent to 1920 pixels x 1080 pixels is acquired for each frame (1 / 60 seconds).

[0024] The image data processed by the imaging signal processing unit 132 is output to the imaging control unit 133 and temporarily stored in a volatile memory. The image data stored in the volatile memory is subjected to various image processes by the image processing unit 141, compressed by the compression / decompression unit 142, and then recorded on a recording medium 147 such as a memory card.

[0025] The compression / decompression unit 142 compresses and encodes the image data output from the image processing unit 141 using a compression method for moving images or still images, and records the compressed image data as an image file on the recording medium 147, or decodes the image file read from the recording medium 147. The recording medium 147 is a hard disk drive (HDD), a solid state drive (SSD), a memory card, or the like. The recording medium 147 may be configured to be removable from the imaging device 100, or may be configured not to be easily removable from the imaging device 100.

[0026] The image processing unit 141 applies predetermined image processing to the image data stored in the volatile memory. The predetermined image processing includes, but is not limited to, development processing such as white balance processing, color interpolation (demosaic) processing, and gamma correction processing, signal format conversion processing, and scaling processing.

[0027] Furthermore, image processing unit 141 executes subject detection processing on the image data to detect subjects in the image. Image processing unit 141 determines the main subject based on posture information (for example, joint positions) of the detected subject and position information of an object specific to the shooting scene (hereinafter, a specific object). In this embodiment, the subject is a person, and the main subject is a subject that is the subject of the shooting target (AF control target) that the photographer is aiming at. For example, if the main subject is playing a ball game, the accuracy of determining the main subject can be expected to improve by treating the detected ball as a specific object.

[0028] Image processing unit 141 may use the determination result of the main subject for image processing (for example, white balance processing). Image processing unit 141 stores the image data that has been subjected to image processing, posture information of the detected subject, information on the position and size of a specific object, position information of the center of gravity of the main subject, face and eye position information, etc. in a volatile memory.

[0029] Display unit 145 displays images being captured (live view), captured still images, videos being recorded, detected subjects and main subjects in displayed images, and a GUI for interactive operation. Display unit 145 is a display device such as a liquid crystal display or an organic EL display. Display unit 145 may be an integral part of imaging device 100 or an external device connected to imaging device 100.

[0030] The operation unit 146 is an operation member such as a switch, button, ring, or lever that accepts user operation, and outputs an operation signal corresponding to the operation member operated by the user to the main control unit 140. The main control unit 140 outputs a control signal to each component of the image capture device 100, including the lens unit 101, based on the operation signal, to control them. The operation member also includes, for example, a touch panel that is integrally configured with the display unit 145. A user, i.e., a photographer, can perform various operations on the image capture device 100 by operating the operation unit 146. In addition, the photographer can perform various settings on the image capture device 100 by operating a GUI (Graphical User Interface) displayed on the display unit 145 using the operation unit 146.

[0031] The operation unit 146 includes at least a still image capture button, a video capture button, a mode dial, and a power switch. The still image capture button is an operation member for instructing the main control unit 140 to perform still image capture processing. The video capture button is an operation member for instructing the main control unit 140 to perform video capture processing. The mode dial is an operation member for switching the operation mode of the imaging device 100. The mode dial can switch the operation mode of the imaging device 100 between still image capture mode, video capture mode, and playback mode. The power switch is an operation member for switching the power of the imaging device 100 on / off.

[0032] The power supply control unit 148, under the control of the main control unit 140, controls the supply of power from the battery 149 to each component of the imaging device 100 in accordance with the state of the imaging device 100. The battery 149 is a secondary battery or the like that can supply power for operating the imaging device 100.

[0033] In the still image shooting mode, when the still image shooting button is pressed halfway, the main control unit 140 starts automatic exposure (AE) control and AF control. When the still image shooting button is pressed all the way, the main control unit 140 executes a still image shooting process for recording image data captured by the imaging unit 131 onto the recording medium 147.

[0034] In addition, in the video shooting mode, when the video shooting button is pressed for the first time, the main control unit 140 performs AE control and AF control on the image data (frames) captured by the imaging unit 131, continues the video shooting process of recording a video for a predetermined time on the recording medium 147, and stops the video shooting process when the video shooting button is pressed again.

[0035] The volatile memory 143 is, for example, a DRAM, and is used as a buffer memory that temporarily stores image data captured by the imaging unit 131, an image display memory for the display unit 145, a work area for the main control unit 140, etc.

[0036] The nonvolatile memory 144 is, for example, a flash ROM, and stores a control program executed by the main control unit 140. When the power is turned on by a user operation and the imaging device 100 is started up, the control program stored in the nonvolatile memory 144 is read (loaded) into a part of the volatile memory 143. The main control unit 140 controls the operation of the imaging device 100 in accordance with the control program loaded into the volatile memory 143.

[0037] The main control unit 140 performs arithmetic processing for controlling the imaging device 100 including the lens unit 101. The main control unit 140 includes at least one programmable processor such as a CPU that controls the components of the imaging device 100. The main control unit 140 controls the components of the imaging device 100 and realizes the functions of the imaging device 100, for example, by loading a program stored in the non-volatile memory 144 into the volatile memory 143 and executing it. Note that instead of the main control unit 140 controlling the entire imaging device 100, the entire imaging device 100 may be controlled by multiple pieces of hardware sharing the processing.

[0038] The main control unit 140 controls the focus control unit 124 based on the focus detection result obtained by the phase difference detection method or the TV-AF method, and executes AF processing to drive the focus lens 109.

[0039] The main control unit 140 also executes automatic exposure (AE) processing to automatically determine exposure conditions (shutter speed or accumulation time, aperture value, and sensitivity) based on brightness information of the subject. The brightness information of the subject can be acquired, for example, by the image processing unit 141. The main control unit 140 can also determine exposure conditions based on the area of ​​a specific subject, such as a person's face, for example.

[0040] The shake detection unit 151 detects shake of the imaging device 100. The shake detection unit 151 detects the amount of shake in three orthogonal axial directions of the imaging device 100. The shake detection unit 151 is, for example, a gyro sensor that detects angular velocities in the three axial directions of the imaging device 100, namely, the pitch direction, the yaw direction, and the roll direction.

[0041] In the imaging device 100 of this embodiment, the image stabilization control unit 123 performs optical shake correction by controlling the shift lens driving unit 108 to drive the shift lens 107 under the control of the main control unit 140. Note that the imaging control unit 133 may perform optical shake correction by moving the imaging unit 131 based on the amount of shake of the imaging device 100 detected by the shake detection unit 151, or the image processing unit 141 may perform electronic shake correction of the image based on the amount of shake of the imaging device 100 detected by the shake detection unit 151 under the control of the main control unit 140.

[0042] Motion vector detection unit 152 detects inter-frame motion vectors for each frame. For example, if the position of the main subject detected in one frame is shifted in the shooting angle of view from the next frame, the amount of movement of the main subject in the shooting angle of view in each of the horizontal X direction and vertical Y direction can be calculated in units of, for example, 1 / 256 pixels. Motion vector detection unit 152 can detect not only the main subject but also multiple movements simultaneously, such as movements of a specific object or the entire background.

[0043] The motion vector information, which is the detection result of the motion vector detection unit 152, can be used, for example, to further correct the remaining blur after camera shake correction by the shift lens 107 by using the motion information of the entire background. It can also be used for special image processing such as fixing the main subject at a fixed position within the shooting angle of view, or for positioning when combining multiple frames.

[0044] Furthermore, by using the detection results of the shake detection unit 151 and the detection results of the main subject by the motion vector detection unit 152, it is possible to determine with a high degree of probability whether the photographer is chasing the subject or wants to switch to another subject.

[0045] The components of the image capture device 100 are connected via a bus 160 so that data can be exchanged, and are controlled by a main control unit 140 .

[0046] <Main subject determination process> Next, the main subject determination process of this embodiment will be described with reference to FIGS.

[0047] FIG. 2 is a block diagram illustrating the configuration of image processing unit 141 as the subject detection device of this embodiment, and illustrates each functional unit from when image processing unit 141 acquires image data to when it determines the main subject.

[0048] Each function of the image processing unit 141 is realized by hardware and / or software. When each functional unit shown in Fig. 2 is configured by hardware instead of by software, it is sufficient to have a circuit configuration corresponding to each functional unit in Fig. 2.

[0049] The image acquisition unit 201 acquires image data captured at a time of interest from the imaging control unit 133. The subject detection unit 202 detects (one or more) people who are subjects in the image acquired by the image acquisition unit 201. The posture acquisition unit 203 estimates the posture of each of the multiple subjects detected by the subject detection unit 202 and acquires posture information. The content of the posture information to be acquired is determined according to the type of subject. In this embodiment, the subject is a person, so the posture acquisition unit 203 acquires position information of multiple joints of the person.

[0050] The object detection unit 204 detects a specific object from the image acquired by the image acquisition unit 201, and acquires the two-dimensional coordinates and size of the specific object in the image. The type of specific object to be detected is determined based on the photographed scene. In this embodiment, the photographed scene is a ball game, so the object detection unit 204 detects a ball used in the game as the specific object.

[0051] The posture estimation method and object detection method are not limited to a specific direction, and for example, the methods described in the following documents 3 and 4 can be used. (Reference 3) Redmon, Joseph, et al., ``You only look once: Unified, real-time object detection.'', Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. (Reference 4) Cao, Zhe, et al., ``Realtime multi-person 2d pose estimation using part affinity fields.'', Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. FIG. 4 is a diagram illustrating posture information and object information of a subject according to this embodiment.

[0052] FIG. 4(a) illustrates an example of an image that is the target of the main subject determination process of this embodiment. In the example of FIG. 4(a), a subject 401 is about to kick a ball 403. The subject 401 is a subject that is performing a specific important action in a shooting scene. In this embodiment, posture information of the subject and object information of a specific object (ball) are used to determine a main subject that is likely to be intended by the photographer as an imaging target (tracking target). On the other hand, a subject 402 is a non-main subject. A non-main subject is a subject other than the main subject.

[0053] FIG. 4(b) illustrates object information including posture information of subjects 401 and 402 in FIG. 4(a) and the position and size of ball 403. Joint 411 represents each joint of subject 401, and joint 412 represents each joint of subject 402. In the example of FIG. 4(b), position information of the top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles is acquired as joints. However, the joint positions are not limited to this example, and some of these positions may be acquired, or other position information may be acquired. Furthermore, information on axes connecting joints may be used in addition to joint positions, and any information representing the posture of the subject may be used as posture information. The following describes the case where joint positions are acquired as posture information.

[0054] Orientation acquisition unit 203 acquires the two-dimensional coordinates (x, y) of joints 411 and 412 in the image. The coordinates (x, y) are expressed in pixels. Center of gravity position 413 represents the center of gravity position of ball 403, and arrow 414 represents the size of ball 403 in the image. Object detection unit 204 acquires the two-dimensional coordinates (x, y) of the center of gravity position of ball 403 in the image and the number of pixels indicating the width of ball 403 in the image.

[0055] The reliability calculation unit 205 calculates a reliability (probability value) representing the likelihood of each subject being a main subject, based on the coordinates of the joint positions estimated by the posture acquisition unit 203 and at least one of the coordinates and size of the specific object acquired by the object detection unit 204. Here, as an example of a method for calculating the reliability, a case where a neural network, which is a machine learning technique, is used will be described.

[0056] FIG. 5 is a diagram illustrating the structure of a neural network.

[0057] The neural network includes an input layer 501, a hidden layer 502, an output layer 503, neurons 504, and lines 505 indicating the connections between the neurons 504. For simplification, FIG. 5 shows only representative neurons and connecting lines with reference numerals. The number of neurons 504 in the input layer 501 is equal to the dimension of the input data, and the number of neurons in the output layer 503 is two. This corresponds to the two-class classification problem of determining whether or not an object is the main subject.

[0058] A weight wij is assigned to the line 505 connecting the i-th neuron 504 in the input layer 501 and the j-th neuron 504 in the hidden layer 502, and the value zj output by the j-th neuron 504 in the hidden layer 502 is given by the following equation 1. (Formula 1) TIFF2026014049000002.tif51158 (Formula 2) TIFF2026014049000003.tif26124In Equation 1, xi represents the value input to the i-th neuron 504 in the input layer 501. The sum is taken for all neurons 504 in the input layer 501 that are connected to the j-th neuron. bj is called the bias and is a parameter that controls the likelihood of firing the j-th neuron 504. The function h defined in Equation 2 is an activation function called ReLU (Rectified Linear Unit). Another function, such as a sigmoid function, can also be used as the activation function. The value yk output by the k-th neuron 504 in the output layer 503 is given by Equation 3 below. (Formula 3) TIFF2026014049000004.tif51158 (Formula 4) TIFF2026014049000005.tif39138In Equation 3, zj represents the value output by the jth neuron 504 in the hidden layer 502, where i, k = 0, 1. 0 corresponds to a non-main subject, and 1 corresponds to the main subject. The sum is taken for all neurons in the hidden layer 502 that are connected to the kth neuron. Furthermore, the function f defined in Equation 4 is called a softmax function, and outputs a probability value that belongs to the kth class. In this embodiment, f(y1) is used as the probability that the image is likely to be the main subject.

[0059] When performing the learning process, the coordinates of the subject's joint positions and the coordinates and size of the specific object are input. Then, all weights and biases are optimized to minimize the loss function that uses the output probability and correct label. Here, the correct label takes two values: "1" for the main subject and "0" for a non-main subject. The loss function L can be the binary cross-entropy expressed in the following equation 5. (Formula 5) TIFF2026014049000006.tif26166In Equation 5, the subscript m represents the index of the subject being trained. ym is the probability value output from neuron 504 with k=1 in output layer 503, and tm is the correct label. In addition to Equation 5, the loss function can be any function that can measure the degree of agreement with the correct label, such as mean squared error. By optimizing based on Equation 5, it is possible to determine the weights and biases so that the correct label and the output probability value are closer.

[0060] The learned weight and bias values ​​are stored in advance in non-volatile memory 144 and read out to volatile memory 143 as needed. A plurality of types of weight and bias values ​​may be prepared depending on the shooting scene. The reliability calculation unit 205 uses the learned weight and bias (the results of machine learning performed in advance) to output the probability value f(y1) based on Equations 1 to 4.

[0061] Note that when performing the learning process, the state of the main subject before moving on to a specific important action can be learned. For example, in the case of kicking a ball, the state of the main subject swinging his / her leg up to kick the ball can be learned as one of the states of the main subject. The reason for doing so is that when the main subject actually performs a specific important action, control must be executed accurately in imaging device 100. For example, if the reliability (probability value) corresponding to the main subject exceeds a preset threshold, control to automatically record a video or still image (recording control) can be initiated, allowing the photographer to capture important moments without missing them. In this case, information on the typical time from the state that is the target of the learning process to the specific important action may be used to control imaging device 100.

[0062] The learning process of this embodiment may be executed by dedicated hardware such as a GPU (Graphics Processing Unit), or may be executed according to a program running on the CPU of the main control unit 140, or may be executed by a combination of these.

[0063] In this embodiment, a method for calculating reliability (probability value) using a neural network has been described, but other machine learning techniques such as a support vector machine or a decision tree may be used as long as it is possible to classify an object as to whether it is the main subject or not. Furthermore, a function that outputs reliability (probability value) based on a certain model may be constructed, without being limited to machine learning.

[0064] In this embodiment, the probability that a subject is the main subject of the image to be processed is used as the reliability indicating the likelihood of the subject being the main subject (reliability corresponding to the degree of possibility that the subject is the main subject of the image to be processed), but a value other than probability may also be used. For example, the reciprocal of the distance between the center of gravity of the subject and the center of gravity of the unique object may be used as the reliability.

[0065] The action detection unit 210 includes a subject detection unit 202, a posture acquisition unit 203, an object detection unit 204, and a reliability calculation unit 205. The action detection unit 210 detects the subject with the highest reliability (probability value) from among the multiple subjects detected by the subject detection unit 202, and outputs a detection result indicating that a subject performing an action has been detected from the acquired image.

[0066] The main subject determination unit 206 determines the main subject based on the detection results of the action detection unit 210 and the detection results of the information acquisition unit 207 .

[0067] Information acquisition section 207 outputs the detection results of shake detection section 151 , the detection results of motion vector detection section 152 , and the detection results of object detection section 204 to main subject determination section 206 .

[0068] Next, the main subject determination process performed by the main subject determination unit 206 of this embodiment will be described with reference to FIG.

[0069] FIG. 3 is a flowchart illustrating the processing of the main subject determination unit 206 in FIG.

[0070] The processing in Fig. 3 is realized by the image processing unit 141 of this embodiment functioning as the main subject determination unit 206 under the control of the main control unit 140. Note that the processing in Fig. 3 is executed when a tracking function that keeps the main subject in focus is enabled in still image shooting mode or video shooting mode, and while the photographer is shooting a ball game (for example, basketball). Note that the type of sport is not limited to basketball.

[0071] In step S301, main subject determination unit 206 determines whether or not a main subject performing an action is being tracked. Since the tracking state does not begin until the main subject is detected for the first time, the process proceeds to step S302.

[0072] In step S302, main subject determination unit 206 determines whether or not a main subject performing an action has been detected by action detection unit 210. If a main subject performing an action has not been detected, the process ends, and if a main subject performing an action has been detected, the process proceeds to step S303.

[0073] In steps S303 and S304, main subject determination unit 206 sets the main subject performing an action as the tracking target and turns on the tracking in progress flag. Setting the tracking target means storing color information, brightness information, etc. of the currently selected main subject as a template. This makes it possible to continue tracking the tracking target subject from the template even if the tracking target can no longer be detected.

[0074] In step S305, main subject determination unit 206 determines whether or not a ball has been detected based on the detection result of object detection unit 204 obtained by information acquisition unit 207. If a ball has not been detected, the process ends, and if a ball has been detected, the process proceeds to step S306.

[0075] In step S306, main subject determination unit 206 turns on the ball detection flag.

[0076] Here, if a main subject performing an action is detected by action detection unit 210 and is set as a tracking target, the process proceeds from step S301 to step S307.

[0077] In step S307, main subject determination unit 206 determines whether a new main subject candidate has been detected by action detection unit 210. A condition for detecting a new main subject candidate is that, among multiple subjects detected by subject detection unit 202, the reliability of the current main subject decreases and the reliability of another subject increases. If a new main subject candidate has been detected by action detection unit 210, the process proceeds to step S313; if a new main subject candidate has not been detected, the process proceeds to step S308.

[0078] In step S308, main subject determination unit 206 determines whether or not the ball is still being detected, similar to step S305. If the ball is not being detected, the process skips step S309 and proceeds to step S310, but if the ball is being detected, the process proceeds to step S309.

[0079] In step S309, the main subject determination unit 206 turns on the ball detection flag.

[0080] In step S310, main subject determination unit 206 determines whether the main subject has been lost. A state in which the main subject has been lost refers to a state in which the main subject has not been detected and there is no target that matches the template set in step S303, meaning that the main subject has moved out of the shooting angle of view. If the subject is in a lost state in step S310, the process proceeds to step S311; if the subject is not in a lost state, the processes of steps S31 and S312 are skipped and tracking of the current main subject continues.

[0081] In steps S311 and S312, main subject determination unit 206 turns off the ball detection flag and the tracking flag.

[0082] If a new subject is detected in step S307, the process proceeds to step S313, where main subject determination unit 206 determines whether the ball detection flag is on. If the ball detection flag is on, the process proceeds to step S314, and if the ball detection flag is off, the process proceeds to step S315.

[0083] In step S314, main subject determination unit 206 determines whether the ball has disappeared. Information indicating whether the ball has disappeared is obtained from information acquisition unit 207. If the ball has disappeared, the process proceeds to step S316, and if the ball has not disappeared, the process proceeds to step S315.

[0084] If the process proceeds from step S313 to S315, this means that the reliability of the current main subject and the reliability of the newly detected main subject candidate have been reversed when the ball is not detected. If the process proceeds from step S314 to S315, this means that the reliability of the current main subject and the reliability of the newly detected main subject candidate have been reversed when the ball is detected. In either case, the main subject performing the action has changed, so in step S315 the current main subject is switched to the newly detected main subject candidate. When the main subject is switched in step S315, the tracking target changes, so the processes of steps S303 to S306 are executed again, and the process ends.

[0085] If a new main subject candidate is detected in step S307, this means that the reliability of the newly detected main subject candidate is higher than the reliability of the current main subject. Therefore, ideally, the current main subject should be switched to the newly detected main subject candidate. However, in this embodiment, when detecting a subject performing a specific action, the detection state of the ball by object detection unit 204 also contributes to the reliability of the main subject. In other words, if the ball disappears from the shooting angle of view, the reliability of the current main subject may suddenly drop. In this case, there is a high possibility that the reliability of the current main subject and the reliability of the newly detected main subject candidate will be reversed, and the main subject may be switched to a newly detected main subject candidate that is different from the main subject being tracked by the photographer. Therefore, in this embodiment, even if a new main subject candidate is detected in step S307, if the ball detected immediately before disappears in step S314, the main subject is not immediately switched, but the process proceeds to step S316 to determine whether to switch the current main subject to the newly detected main subject candidate.

[0086] In step S316, main subject determination unit 206 turns off the ball detection flag.

[0087] In step S317, main subject determination unit 206 performs a process of setting a main subject according to the state of ball disappearance. Details of this process will be described later with reference to FIG.

[0088] In step S318, main subject determination unit 206 proceeds to processing to perform necessary operations based on the main subject setting result of step S317. That is, if tracking of the current main subject is to continue, processing ends, and if the main subject is to be changed, processing proceeds to step S303, where the main subject to be tracked is set.

[0089] FIG. 6 is a flowchart illustrating the main subject setting process in step S317 of FIG.

[0090] In step S601, the main subject determination unit 206 acquires various information according to the ball disappearance situation. This information is obtained from the information acquisition unit 207, and includes ball detection information, the detection results of the shake detection unit 151, the detection results of the motion vector detection unit 152, and time-series information on each of these.

[0091] In step S602, the main subject determination unit 206 determines whether the ball has disappeared near the center of the imaging angle of view based on the ball position according to the ball disappearance situation. If the ball has disappeared near the center of the imaging angle of view, the process proceeds to step S603. If the ball has disappeared at a position other than the center of the imaging angle of view, the process proceeds to step S607.

[0092] In step S603, main subject determination unit 206 determines whether the main subject to be tracked has moved by more than a predetermined amount. Whether the main subject to be tracked has moved by more than a predetermined amount can be determined from the amount of movement of the main subject to be tracked within the shooting angle of view, using the detection result from motion vector detection unit 152. If the main subject to be tracked has not moved by more than a predetermined amount and the movement is small, the process proceeds to step S604, and if the main subject to be tracked has moved by more than a predetermined amount and the movement is large, the process proceeds to step S605.

[0093] In step S605, main subject determination unit 206 determines whether the photographer is shaking image capture device 100 to track the main subject. This can be determined by whether the detection result of shake detection unit 151 matches the direction in which the main subject moved within the shooting angle of view. If the photographer is tracking the main subject, the process proceeds to step S604; if the photographer is not tracking the main subject, the process proceeds to step S608.

[0094] If the ball disappears near the center of the shooting angle of view, it is likely that it has gone behind another player or an obstacle. Examples include a basketball player colliding with another player while dribbling and losing sight of the ball, or a layup or dunk shot where the ball is hidden by the shadow of the hoop. The process proceeds from step S603 to S604 when the main subject is moving slowly and the photographer is tracking the main subject, which is considered to correspond to the scene described above. Therefore, in step S604, the main subject is not changed. On the other hand, the process proceeds from step S602 to S606 when, for example, the photographer makes a pass while feinting (from behind the photographer) and then runs to another location. In this case, the photographer is likely chasing the ball, but the direction of movement of the main subject being tracked and the location of the next candidate main subject for the pass are generally opposite, so the image capture device 100 moves in the opposite direction to the main subject's movement. In other words, since the main subject is changing, the main subject is changed in step S608.

[0095] If the ball disappearance position is not near the center of the shooting angle of view in step S602, it means that the ball has disappeared near the boundary of the shooting angle of view, and in step S606 it is determined whether the ball disappearance position is on the upper side of the shooting angle of view. The direction in which the ball moved can also be determined by checking the motion vector relative to the object. If the ball disappearance position is on the upper side of the shooting angle of view in step S606, the process proceeds to step S607, and if it is not on the upper side, the process proceeds to step S608. Note that if the type of sport is not basketball, the ball disappearance position is not limited to the upper side of the shooting angle of view, and conditions such as front, back, or front are set according to the type of sport.

[0096] In step S607, main subject determination unit 206 determines whether the newly detected main subject candidate has moved more than a predetermined amount. If the newly detected main subject candidate has moved more than the predetermined amount, and the movement is large, processing proceeds to step S608; if the newly detected main subject candidate has not moved more than the predetermined amount, and the movement is small, processing proceeds to step S604. If the ball disappears from a position that is not at the top of the shooting angle of view, there is a high possibility that the ball has been passed to another player, and therefore there is a high possibility that the main subject will be changed. Therefore, the main subject is changed in step S608. If the movement of the newly detected main subject candidate in step S607 is small, this may be due to a free throw, for example, and therefore a determination is made in step S604 that the main subject will not be changed.

[0097] Action detection unit 210 notifies main subject determination unit 206 of the subject (person) detected by subject detection unit 202 that has the highest reliability (probability value) as a main subject candidate. Then, main subject determination unit 206 selects the main subject based on ball detection information from information acquisition unit 207, etc., and stores the coordinates of the joint positions of the main subject and representative coordinates representing the main subject (center of gravity position, face position, etc.) in volatile memory 143. This completes the main subject determination process.

[0098] [Embodiment 2] Next, a second embodiment will be described with reference to FIG.

[0099] In the second embodiment, the photographer can operate the operation unit 146 to select the type of sport to be photographed, and main subject setting processing is performed according to the sport selected by the photographer.

[0100] The configuration of the imaging device 100 and the image processing unit 141 as a subject detection device of the second embodiment is the same as that of the embodiment shown in FIGS. 1 and 2, but the operation of the main subject determination unit 206 is different in that the main subject setting process in accordance with the ball disappearance situation in step S317 of FIG. 3 is different.

[0101] FIG. 7 is a flowchart illustrating the main subject setting process according to the ball disappearance state in step S317 of FIG. 3 according to the second embodiment.

[0102] In FIG. 7, in step S701, main subject determination unit 206 acquires the type of sport selected by the photographer operating operation unit 146.

[0103] In step S702, main subject determination unit 206 proceeds to processing according to the event selected by the photographer in step S701. If event A is set in step S702, processing proceeds to step S703, if event B is set, processing proceeds to step S704, and if event C is set, processing proceeds to step S705.

[0104] In steps S703, S704, and S705, main subject determination unit 206 performs a main subject setting process that is most suitable for each sport.

[0105] In embodiment 2, the photographer selects the sport to be photographed and performs the main subject setting process optimal for the selected sport, thereby significantly reducing the possibility that the main subject intended by the photographer as the subject to be photographed (tracked) will change when the ball disappears in each sport.

[0106] According to each of the above-described embodiments, even if a new main subject candidate is detected by action detection unit 210, it is determined whether the ball has disappeared and whether to switch the main subject depending on the circumstances under which the ball disappeared. This makes it possible to avoid a situation in which the main subject intended as the imaging target (tracking target) by the photographer while filming a sport is inadvertently switched.

[0107] [Other embodiments] This embodiment can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0108] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0109] The disclosure of this specification includes the following subject detection device, subject detection method, imaging device, and program. [Item 1] an object detection means for detecting an object from an image of a game involving a specific object; an object detection means for detecting the specific object from the captured image; a setting means for setting a main subject to be an object of autofocus operation from among the plurality of subjects detected by the subject detection means, The subject detection device is characterized in that the setting means determines whether to switch the current main subject to the newly detected main subject candidate when the specific object is no longer detected from the captured image, even if a new main subject candidate is detected. [Item 2] Item 1. The subject detection device according to item 1, characterized in that, even if the new main subject candidate is detected, if the specific object is no longer detected from the captured image, the setting means determines whether to switch the current main subject to the newly detected main subject candidate depending on the situation in which the specific object is no longer detected. [Item 3] Item 2. The subject detection device according to item 2, wherein the setting means determines whether to switch the current main subject to the newly detected main subject candidate depending on the situation in which the specific object is no longer detected in the captured image, even if the new main subject candidate is detected while the specific object is still detected in the captured image. [Item 4] 4. The subject detection device according to item 2 or 3, wherein the situation in which the specific object is no longer detected includes at least one of the position where the specific object disappeared from the shooting angle of view, the direction in which the specific object moved, the movement of the current main subject, and the movement of the newly detected main subject candidate. [Item 5] 5. The subject detection device according to item 4, wherein the position at which the specific object has disappeared from the imaging angle of view includes a case where the position is near the center of the imaging angle of view or a case where the position is not near the center of the imaging angle of view. [Item 6] 6. The subject detection device according to item 5, wherein the setting means, if the position where the specific object has disappeared from the photographic angle of view is not near the center of the photographic angle of view, determines whether to switch the current main subject to the newly detected main subject candidate based on the direction in which the specific object has moved and the movement of the newly detected main subject candidate. [Item 7] 7. The subject detection device according to item 6, wherein the setting means determines to switch the current main subject to the newly detected main subject candidate if the position where the specific object has disappeared from the photographic angle of view is not near the center of the photographic angle of view, or if the specific object is not moving in a predetermined direction. [Item 8] Item 7. The subject detection device according to item 6, wherein the setting means determines that the specific object has moved in a predetermined direction if the position where the specific object has disappeared from the photographic angle of view is not near the center of the photographic angle of view, and determines that the newly detected main subject candidate has not moved by more than a predetermined amount, and does not switch the current main subject to the newly detected main subject candidate. [Item 9] Item 7. The subject detection device according to item 6, wherein the setting means determines to switch the current main subject to the newly detected main subject candidate if the position where the specific object has disappeared from the photographing angle of view is not near the center of the photographing angle of view, the specific object has moved in a predetermined direction, and the newly detected main subject candidate has moved by more than a predetermined amount. [Item 10] 6. The subject detection device according to item 5, wherein the setting means determines whether to switch the current main subject to the newly detected main subject candidate based on the movement of the current main subject when the position where the specific object has disappeared from the photographic angle of view is near the center of the photographic angle of view. [Item 11] Item 11. The subject detection device according to item 10, characterized in that the setting means determines not to switch the current main subject to the newly detected main subject candidate if the position where the specific object has disappeared from the photographing angle of view is near the center of the photographing angle of view, the current main subject has moved by more than a predetermined amount, and the imaging device is moving towards the current main subject. [Item 12] Item 11. The subject detection device according to item 10, characterized in that the setting means determines that the current main subject has moved by more than a predetermined amount if the position where the specific object has disappeared from the photographing angle of view is near the center of the photographing angle of view, and determines to switch the current main subject to the newly detected main subject candidate if the imaging device has not moved toward the main subject. [Item 13] Item 11. The subject detection device according to item 10, wherein the setting means determines not to switch the current main subject to the newly detected main subject candidate if the position where the specific object has disappeared from the photographic angle of view is near the center of the photographic angle of view and the current main subject has not moved by more than a predetermined amount. [Item 14] 2. The subject detection device according to item 1, wherein the setting means determines to switch the current main subject to the newly detected main subject candidate when the new main subject candidate is detected and the specific object is not detected from the captured image. [Item 15] a posture acquisition means for acquiring posture information of the subject; an action detection means for detecting a subject performing a specific action based on a detection result of the object detection means and a detection result of the posture acquisition means, 15. The subject detection device according to any one of items 1 to 14, wherein the setting means sets the main subject based on the detection result of the action detection means. [Item 16] a calculation means for calculating a reliability indicating a likelihood of each of the plurality of subjects being a main subject based on at least one of the posture of the subject and the position and size of the specific object; Item 16. The subject detection device according to item 15, wherein the action detection means detects the subject having the highest reliability from the plurality of subjects as a main subject candidate. [Item 17] the calculation means calculates the reliability through a learning process; Item 17. The subject detection device according to item 16, wherein the learning process learns the state of the main subject before moving on to the specific action. [Item 18] a selection means for selecting the type of competition; 18. The subject detection device according to any one of items 1 to 17, wherein the setting means executes a process of setting the main subject according to the type of sport. [Item 19] 19. The subject detection device according to any one of items 1 to 18, wherein the sport is a ball game and the specific object is a ball or an object similar to a ball. [Item 20] An imaging means; An object detection device according to any one of items 1 to 19, and a focus control unit that executes the autofocus operation on the main subject. [Item 21] A subject detection method for a subject detection device, comprising: detecting a subject from an image captured of a game involving a specific object; detecting the specific object from the captured image; and setting a main subject to be subjected to autofocus operation from the plurality of detected subjects, In the setting step, even if a new candidate for the main subject is detected, if the specific object is no longer detected from the captured image, it is determined whether or not to switch the current main subject to the newly detected candidate for the main subject. [Item 22] 20. A program for causing a computer to function as the subject detection device according to any one of items 1 to 19. [Explanation of symbols]

[0110] 100...imaging device, 140...main control unit, 141...image processing unit (subject detection device), 202...subject detection unit, 204...object detection unit, 206...main subject determination unit, 210...action detection unit

Claims

1. an object detection means for detecting an object from an image of a game involving a specific object; an object detection means for detecting the specific object from the captured image; a setting means for setting a main subject to be an object of autofocus operation from among the plurality of subjects detected by the subject detection means, The subject detection device is characterized in that the setting means determines whether to switch the current main subject to the newly detected main subject candidate when the specific object is no longer detected from the captured image, even if a new main subject candidate is detected.

2. 2. The subject detection device according to claim 1, wherein, even if the new candidate for the main subject is detected, if the specific object is no longer detected in the captured image, the setting means determines whether to switch the current main subject to the newly detected candidate for the main subject depending on the situation in which the specific object is no longer detected.

3. 3. The subject detection device according to claim 2, wherein even if the new main subject candidate is detected while the specific object has been detected in the captured image, if the specific object is no longer detected in the captured image, the setting means determines whether to switch the current main subject to the newly detected main subject candidate depending on the situation in which the specific object is no longer detected.

4. 3. The subject detection device according to claim 2, wherein the situation in which the specific object is no longer detected includes at least one of the position where the specific object disappeared from the shooting angle of view, the direction in which the specific object moved, the movement of the current main subject, and the movement of the newly detected main subject candidate.

5. The subject detection device according to claim 4 , wherein the position at which the specific object has disappeared from the photographic field of view includes a case where the position is near the center of the photographic field of view or a case where the position is not near the center of the photographic field of view.

6. 6. The subject detection device according to claim 5, wherein, when the position where the specific object has disappeared from the photographic angle of view is not near the center of the photographic angle of view, the setting means determines whether to switch the current main subject to the newly detected main subject candidate based on the direction in which the specific object has moved and the movement of the newly detected main subject candidate.

7. 7. The subject detection device according to claim 6, wherein the setting means determines to switch the current main subject to the newly detected main subject candidate if the position where the specific object has disappeared from the photographic angle of view is not near the center of the photographic angle of view and if the specific object is not moving in a predetermined direction.

8. 7. The subject detection device according to claim 6, wherein the setting means determines that the specific object has moved in a predetermined direction if the position at which the specific object has disappeared from the photographic angle of view is not near the center of the photographic angle of view, and that the newly detected main subject candidate has not moved by more than a predetermined amount, and determines not to switch the current main subject to the newly detected main subject candidate.

9. 7. The subject detection device according to claim 6, wherein the setting means determines to switch the current main subject to the newly detected main subject candidate if the position at which the specific object has disappeared from the photographing angle of view is not near the center of the photographing angle of view, the specific object has moved in a predetermined direction, and the newly detected main subject candidate has moved by more than a predetermined amount.

10. 6. The subject detection device according to claim 5, wherein the setting means determines whether to switch the current main subject to the newly detected main subject candidate based on the movement of the current main subject when the position at which the specific object has disappeared from the photographic angle of view is near the center of the photographic angle of view.

11. 11. The subject detection device according to claim 10, wherein the setting means determines not to switch the current main subject to the newly detected main subject candidate if the position at which the specific object has disappeared from the photographing angle of view is near the center of the photographing angle of view, the current main subject has moved by more than a predetermined amount, and the imaging device is moving toward the current main subject.

12. 11. The subject detection device according to claim 10, wherein the setting means determines that the current main subject has moved by more than a predetermined amount when the position where the specific object has disappeared from the photographing angle of view is near the center of the photographing angle of view, and determines to switch the current main subject to the newly detected main subject candidate when the imaging device is not moving toward the main subject.

13. 11. The subject detection device according to claim 10, wherein the setting means determines not to switch the current main subject to the newly detected main subject candidate if the position at which the specific object has disappeared from the photographic angle of view is near the center of the photographic angle of view and the current main subject has not moved by more than a predetermined amount.

14. 2. The subject detection device according to claim 1, wherein the setting means determines to switch the current main subject to the newly detected main subject candidate when the new main subject candidate is detected and the specific object is not detected from the captured image.

15. a posture acquisition means for acquiring posture information of the subject; an action detection means for detecting a subject performing a specific action based on a detection result of the object detection means and a detection result of the posture acquisition means, 2. The subject detection device according to claim 1, wherein the setting means sets the main subject based on the detection result of the action detection means.

16. a calculation means for calculating a reliability indicating a likelihood of each of the plurality of subjects being a main subject based on at least one of the posture of the subject and the position and size of the specific object; 16. The subject detection device according to claim 15, wherein the action detection means detects the subject having the highest reliability from the plurality of subjects as a main subject candidate.

17. the calculation means calculates the reliability through a learning process; 17. The subject detection device according to claim 16, wherein the learning process learns the state of the main subject before moving on to the specific action.

18. a selection means for selecting the type of competition; 2. The subject detection device according to claim 1, wherein the setting means executes a process of setting the main subject according to the type of the sport.

19. 2. The subject detection device according to claim 1, wherein the game is a ball game, and the specific object is a ball or an object similar to a ball.

20. An imaging means; An object detection device according to any one of claims 1 to 19; and a focus control unit that executes the autofocus operation on the main subject.

21. A subject detection method for a subject detection device, comprising: detecting a subject from an image captured of a game involving a specific object; detecting the specific object from the captured image; and setting a main subject to be subjected to autofocus operation from the plurality of detected subjects, In the setting step, even if a new candidate for the main subject is detected, if the specific object is no longer detected from the captured image, it is determined whether or not to switch the current main subject to the newly detected candidate for the main subject.

22. A program for causing a computer to function as the subject detection device according to any one of claims 1 to 19.

Images