Image processor, method for processing image, and imaging device

Abstract

To provide an image processor, a method for processing an image, and an imaging device which can determine an object desirable for a user even when more than one object exist.SOLUTION: Main object determination processing in an image processing unit includes: detecting a specific object from an acquired image; detecting an attitude of each detected object and acquiring a reliability of being a main object on the basis of the attitude; acquiring information of a priority target; and determining a main object on the basis of the reliability and the information of the priority target.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus, and particularly to a technique for determining a main subject from an image.

Background Art

[0002] Patent Document 1 discloses a technique for estimating the posture of a person in a sports scene and detecting a specific posture to be noted, such as a shoot or a goal, to determine the main subject.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When determining the main subject as described in Patent Document 1, if a subject that the user does not aim at takes a specific posture, that subject may be determined as the main subject.

[0005] In one aspect of the present invention, a technique for determining a desired main subject for a user even when there are a plurality of subjects is provided.

Means for Solving the Problems

[0006] As a technical feature of the present invention, it includes a first detection means for detecting a specific subject from an image, a second detection means for detecting a posture for each of the subjects detected by the first detection means, a first acquisition means for acquiring a reliability of being a main subject based on the posture for each of the subjects detected by the first detection means, a second acquisition means for acquiring information on a priority target, and a determination means for determining a main subject based on the reliability and the information on the priority target.

Advantages of the Invention

[0007] According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an imaging apparatus capable of determining a desired main subject for a user even when there are a plurality of subjects.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] In the following embodiments, the case where the present invention is implemented in an imaging device such as a digital camera will be described. However, the imaging function is not essential for the present invention, and it can be implemented with any electronic device. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented with other electronic devices.

[0011] ● Overall Configuration FIG. 1 is a block diagram showing the configuration of an imaging device 100 including a main subject determination device as an example of an image processing device according to the present embodiment. The imaging device 100 is a digital still camera, a video camera, or the like that captures a subject and records moving image or still image data on various media such as a tape, a solid-state memory, an optical disk, or a magnetic disk, but is not limited thereto. Hereinafter, the case where the subject is a person will be described as an example. Also, the main subject represents the subject that is the target of imaging control intended by the user. Each unit in the imaging device 100 is connected via a bus 160. Also, each unit is controlled by a main control unit 151.

[0012] The lens unit 101 includes a fixed single-group lens 102, a zoom lens 111, a diaphragm 103, a fixed three-group lens 121, and a focus lens 131. The diaphragm control unit 105 adjusts the aperture diameter of the diaphragm 103 by driving the diaphragm 103 via a diaphragm motor 104 (AM) in accordance with a command from the main control unit 151, thereby performing light amount adjustment during imaging. The zoom control unit 113 changes the focal length by driving the zoom lens 111 via a zoom motor 112 (ZM). The focus control unit 133 determines a driving amount for driving a focus motor 132 (FM) based on the amount of deviation in the focus direction of the lens unit 101. In addition, the focus control unit 133 controls the focus adjustment state by driving the focus lens 131 via the focus motor 132 (FM). AF control is realized by the movement control of the focus lens 131 by the focus control unit 133 and the focus motor 132. The focus lens 131 is a focus adjustment lens, which is simply shown as a single lens in FIG. 1, but is usually composed of a plurality of lenses.

[0013] The subject image formed on the imaging element 141 via the lens unit 101 is converted into an electrical signal by the imaging element 141. The imaging element 141 is a photoelectric conversion element that photoelectrically converts a subject image (optical image) into an electrical signal. The imaging element 141 is provided with light receiving elements having m pixels in the horizontal direction and n pixels in the vertical direction. The image formed and photoelectrically converted on the imaging element 141 is arranged as an image signal (image data) by the imaging signal processing unit 142. Thereby, an image of the imaging surface can be acquired.

[0014] The image data output from the imaging signal processing unit 142 is sent to the imaging control unit 143 and temporarily stored in a RAM 154 (random access memory). The image data stored in the RAM 154 is compressed by the image compression / decompression unit 153 and then recorded on the image recording medium 157. In parallel with this, the image data stored in the RAM 154 is sent to the image processing unit 152.

[0015] The image processing unit 152 applies predetermined image processing to the image data stored in the RAM 154. The image processing applied by the image processing unit 152 includes so-called development processing such as white balance adjustment processing, color interpolation (demosaicing) processing, and gamma correction processing, as well as signal format conversion processing, scaling processing, etc., but is not limited thereto. Further, the image processing unit 152 determines the main subject based on the information on the pose of the subject (for example, joint positions) and the position information of an object specific to the scene (hereinafter referred to as a specific object). The image processing unit 152 may use the result of the determination process for other image processing (for example, white balance adjustment processing). The image processing unit 152 stores the processed image data, the information on the pose of each subject, the position and size information of the specific object, the center of gravity, face, and pupil position information of the main subject, etc. in the RAM 154.

[0016] The operation switch 156 is an input interface including a touch panel, buttons, etc., and various operations on the imaging device 100 can be performed by the user performing a selection operation on various function icons displayed on the display unit 150.

[0017] The main control unit 151 has one or more programmable processors such as a CPU or an MPU. Then, the main control unit 151 controls each part of the imaging device 100 by reading and executing a program stored in the flash memory 155 into the RAM 154, and realizes the functions of the imaging device 100. The main control unit 151 also executes an AE process for automatically determining exposure conditions (shutter speed or accumulation time, aperture value, and sensitivity) based on the information on the subject luminance. The information on the subject luminance can be obtained from, for example, the image processing unit 152. The main control unit 151 can also determine the exposure conditions based on a region of a specific subject such as a person's face.

[0018] The focus control unit 133 performs AF control on the position of the main subject stored in the RAM 154. The aperture control unit 105 performs exposure control using the luminance value of a specific subject region.

[0019] The display unit 150 displays images, detection results of the main subject, and the like. The battery 159 is appropriately managed by the power management unit 158 to supply stable power to the entire imaging device 100. The flash memory 155 stores control programs necessary for the operation of the imaging device 100, parameters used for the operation of each unit, and the like. When the imaging device 100 is activated by a user operation (when shifting from the power-off state to the power-on state), the control program and parameters stored in the flash memory 155 are read into a part of the RAM 154. The main control unit 151 controls the operation of the imaging device 100 according to the control program and constants loaded into the RAM 154.

[0020] ● Main subject determination process With reference to FIGS. 2 and 3, the main subject determination process executed by the image processing unit 152 will be described. FIG. 2 is a block diagram showing a part of the detailed configuration of the image processing unit 152. FIG. 3 is a flowchart of the main subject determination process. The processing of each step of this flowchart is realized by the respective units of the image processing unit 152 operating under the control of the main control unit 151, unless otherwise specified. Hereinafter, a ball game played by multiple people will be described as the shooting scene for the main subject determination process, but the shooting scenes applicable to this embodiment are not limited to this.

[0021] In S301, the image acquisition unit 201 acquires an image captured at the time of interest from the imaging control unit 143. In S302, the subject information detection unit 202 detects a specific object (an object of a predetermined type) and a subject (a person) from the image acquired by the image acquisition unit 201. Then, information on the posture of the subject is obtained, and a reliability representing the likelihood of the main subject of the subject information is calculated from the information on the object and the information on the posture of the subject. Details will be described later.

[0022] In S303, the priority target information acquisition unit 203 acquires the information of the priority target registered by the user. The information of the priority target is information necessary for the user to determine the target that the user wants to prioritize as the main subject at the time of shooting. For example, there are the position of the goal of the team to be shot, the position of the court, the color of the uniform, and the like. In the main subject determination process, in order to determine which subject among the plurality of subjects detected by the subject information detection unit 202 is the main subject, it is necessary to determine the main subject using the information of the priority target. The information of the priority target is stored in the flash memory 155 in advance and stored in the RAM 154 as needed.

[0023] In S304, the priority posture determination unit 204 determines the priority posture using the information of the subject detected by the subject information detection unit 202 and the information of the priority target acquired from the priority target information acquisition unit 203. As a method for determining the priority posture, there are a method of determining according to the posture of the team to be shot and a method of determining according to whether the team to be shot is in the posture or not.

[0024] First, a method of determining based on the posture of the team to be photographed will be described. The priority posture determination unit 204 selects a specific posture using the goal position or the court position acquired from the priority target information acquisition unit 203. For example, when the detected postures are classified into an attacking posture and a defensive posture, if the team to be photographed is on the attacking side, the attacking posture is selected as the priority posture, and if it is on the defensive side, the defensive posture is selected as the priority posture. Examples of the attacking posture include shooting and spiking. Examples of the defensive posture include sliding and blocking. Then, in order to determine whether the team to be photographed is on the attacking side or the defensive side, the method of determining the priority posture is changed according to the competition to be photographed. FIG. 6 shows examples of the subject and the goal position in soccer and basketball competitions. FIG. 6(a) is an example of the subject and the goal position in a soccer competition. If the subject 601 is moving significantly towards the registered goal position as shown in FIG. 6(a), it can be determined that the subject of the team to be photographed is on the attacking side. Therefore, by setting the attacking posture as the priority posture, the posture information of the subject 601, which is the subject of the team to be photographed, can be prioritized. Also, when the panning direction of the imaging device 100 is the same as the position of the registered goal, the attacking posture is similarly set as the priority posture. FIG. 6(b) is an example of the subject and the goal position in a basketball competition. If the subjects are concentrated in the direction opposite to the position of the registered goal as shown in FIG. 6(b), it can be determined that the team to be photographed has gathered for defense. Therefore, by setting the defensive posture as the priority posture, the posture information of the subjects 605, 606, and 607, which are the subjects of the team to be photographed, can be prioritized. In volleyball competitions, if the ball is at the position of the registered court, it can be determined that the team to be photographed is on the attacking side, so the attacking posture is set as the priority posture.

[0025] Next, a method for determining whether it is the posture of the team to be photographed or not will be described. The priority posture determination unit 204 compares the color of the uniform worn by the person in the detected posture with the color of the registered uniform acquired from the priority target information acquisition unit 203. As a comparison method, techniques such as template matching and histogram matching can be used based on the feature amount of the uniform area. When it is determined that the color of the registered uniform is the same as the color of the uniform worn by the person in the detected posture, the detected posture is assumed to be the posture taken by the subject of the team to be photographed. Therefore, the detected posture is determined as the priority posture. If there are a plurality of detected postures, each is compared with the color of the registered uniform, and the priority posture is determined. The number of priority postures may be one or more.

[0026] In the above, as methods for determining the priority posture, two methods have been described: a method of determining based on whether the team to be photographed is on the attacking side or the defending side, and a method of determining based on whether it is the posture taken by the subject of the team to be photographed or not. However, either one of the methods may be used, or both methods may be used.

[0027] In S305, the main subject determination unit 205 determines the subject with the highest reliability among the priority postures determined by the priority posture determination unit 204 as the main subject. FIG. 7 is an example of subject information when the main subject determination unit 205 determines the main subject. A subject ID is assigned to each detected subject, and information on the reliability and whether it is a priority posture is attached to each subject. When there is a plurality of subject information as shown in FIG. 7(a), the subject 703 with the highest reliability among the subjects with the priority posture is set as the main subject.

[0028] In order to make it easier to determine the subject in the prioritized pose as the main subject, it is also possible to set a low confidence threshold only for the prioritized pose. For example, if the prioritized pose is an attack pose, the confidence threshold may be set low only for the attack pose. For example, in Fig. 7(b), when the confidence threshold for the prioritized pose is set to 90 and the confidence threshold for poses other than the prioritized pose is set to 100, the subject 710 with the highest confidence among the prioritized poses is set as the main subject. Then, the main subject determination unit 205 stores the coordinates of the joints of the main subject and representative coordinates (such as the center of gravity position and the position of the face) representing the main subject in the RAM 154. Thereby, the main subject determination process is completed.

[0029] ● Subject information detection process Fig. 4 is a block diagram showing a part of the detailed configuration of the subject information detection unit 202. Fig. 5 is a flowchart of the subject information detection process in S302.

[0030] In S501, the object detection unit 403 detects a specific object (an object of a predetermined type) in the image acquired by the image acquisition unit 201, and acquires the two-dimensional coordinates and size of the specific object in the image. The type of the specific object to be detected is determined based on the shooting scene of the image. Here, since the shooting scene is a ball game, the object detection unit 403 is to detect a ball as the specific object. However, depending on the shooting scene, in addition to the ball, it may also be configured to detect objects that move among competitors in sports, such as a pack in ice hockey and a shuttlecock in badminton.

[0031] In S502, the subject detection unit 401 detects a subject (person) in the image acquired by the image acquisition unit 201.

[0032] In S503, the pose acquisition unit 402 performs pose estimation for each of the plurality of subjects detected by the subject detection unit 401 and acquires the pose. The content of the pose information to be acquired is determined according to the type of the subject. Here, since the subject is a person, the pose acquisition unit 402 acquires the positions of a plurality of joints of the person as the pose information for the subject.

[0033] Note that the method for estimating the pose of the subject from the image of the subject area may be any known method. For example, the method described in Cao, Zhe, et al., “Realtime multi-person 2d pose estimation using part affinity fields.”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017 can be used.

[0034] FIG. 8 is a conceptual diagram of the information acquired by the pose acquisition unit 402 and the object detection unit 403. FIG. 8(a) represents the image to be processed, and the subject 801 is in a pose of about to kick the ball 803. The subject 801 is an important subject in the shooting scene. In the present embodiment, by using the pose information of the subject and the information of the specific object, the main subject that is likely to be intended by the user as the target of imaging control is determined. On the other hand, the subject 802 is a non-main subject. Here, the non-main subject represents a subject other than the main subject.

[0035] FIG. 8(b) is a diagram showing examples of the pose information of the subjects 801 and 802, and the position and size of the ball 803. The joint 811 represents each joint of the subject 801, and the joint 812 represents each joint of the subject 802. In FIG. 8(b), an example of acquiring the positions of the top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles as joints is shown, but the joint positions may be a part of these, or other positions may be acquired. Also, not only the joint positions but also information such as the axes connecting the joints may be used, and any information that represents the pose of the subject can be used as the pose information. Hereinafter, the case of acquiring joint positions as the pose information will be described.

[0036] The posture acquisition unit 402 acquires the two-dimensional coordinates (x, y) in the images of the joints 811 and 812. Here, the unit of (x, y) is pixels. The center of gravity position 813 represents the center of gravity position of the ball 803, and the arrow 814 represents the size of the ball 803 in the image. The object detection unit 403 acquires the two-dimensional coordinates (x, y) in the image of the center of gravity position of the ball 803, and the number of pixels indicating the width of the ball 803 in the image.

[0037] Return to the description of FIG. 5. In S504, the probability calculation unit 404 calculates a reliability (probability) representing the main subjectness for each subject based on at least one of the coordinates of the joints estimated by the posture acquisition unit 402 and the coordinates and size of the specific object acquired by the object detection unit 403. The method of calculating the probability will be described later. In the present embodiment, the case where the probability corresponding to the degree of possibility that the subject is the main subject of the processed image is adopted as the reliability (reliability representing the main subjectness) will be described, but values other than probability may be used. For example, as the reliability, the reciprocal of the distance between the center of gravity position of the subject and the center of gravity position of the specific object can be used.

[0038] ● Method of calculating probability A method of calculating the probability representing the main subjectness based on the coordinates of each joint and the coordinates and size of the specific object in the probability calculation unit 404 will be described. Hereinafter, the case of using a neural network, which is a method of machine learning, will be described.

[0039] FIG. 9 is a diagram showing an example of the structure of a neural network. The neural network has an input layer 901, an intermediate layer 902, and an output layer 903. The intermediate layer 902 may have a plurality of layers. Each layer has a plurality of neurons 904, and the neurons 904 in adjacent layers are connected to each other by synapses 905.

[0040] The number of neurons 904 in the input layer 901 is equal to the dimension of the input data. Also, the number of neurons in the output layer 903 is equal to the number of answers. Here, since we are using a neural network to obtain two types of answers as to whether a certain type of subject is the main subject or not, the number of neurons 904 in the output layer is two. Using a neural network that classifies the input into two classes, it is determined whether the first type of subject is the main subject (whether the probability is high or not).

[0041] Let the weight of the synapse 905 connecting the i-th neuron 904 in the input layer 901 and the j-th neuron in the intermediate layer 902 be w ij Then, the output z j of the j-th neuron in the intermediate layer 902 is given by the following formula.

[0042]

Equation

[0043] In Equation (1), x i represents the value input to the i-th neuron in the input layer 901. Since all neurons in the input layer 901 are connected to the j-th neuron in the intermediate layer 902, the input values of each neuron are weighted and added and input to the j-th neuron in the intermediate layer 902.

[0044] The j-th neuron in the intermediate layer 902 outputs the value of the activation function h with the value obtained by adding the bias b j to the input value as an argument. The bias b j is a parameter corresponding to the sensitivity of the neuron. The activation function h is a function that converts the input value into a value representing the excitation state of the neuron. Here, ReLU (Rectified Linear Unit) is used, but another function such as the sigmoid function may also be used.

[0045] Let the weight of the synapse 905 connecting the j-th neuron in the intermediate layer 902 and the k-th neuron in the output layer 903 be w kj, let the bias of the k-th neuron in the output layer 903 be b k . At this time, the value y k output by the k-th neuron 904 in the output layer 903 is given by the following formula.

[0046]

Equation

[0047] In Equation (3), z j is the output value from the j-th neuron in the intermediate layer 902 connected to the k-th neuron in the output layer 903. Also, i and k are the numbers of the neurons in the output layer 903, and i, k = 1 or 2. The output y k of each neuron in the output layer 903 is normalized by the softmax function shown in Equation (4) so that the sum becomes 1. Let y 1 correspond to the classification of the main subject and y 2 correspond to the non-main subject. Then, f(y 1 ) represents the probability that it is the main subject, and f(y 2 ) represents the probability that it is the non-main subject, respectively.

[0048] The input values of the neural network are the coordinates of the joints of a person, and the coordinates and size of the ball. Then, all weights and biases are optimized by learning so as to minimize the loss function using the output probability and the correct label. The correct label is a binary value of 1 for the main subject and 0 for the non-main subject. The loss function may be any function that can measure the degree of agreement with the correct label, such as the mean squared error. Here, as an example, the binary cross-entropy shown below is used as the loss function.

[0049]

Equation

[0050] In Equation (5), m is the index of the subject to be learned. y mis equal to the probability value f(y 1 ) output from the neuron with k = 1 in the output layer 903. t m is the correct label (0 or 1).

[0051] By optimizing so that the value of Equation (5) becomes smaller, the weights and biases can be learned so that the correct label and the output probability value approach each other.

[0052] The learned weight and bias values are stored in the flash memory 155 in advance and stored in the RAM 154 as needed. Multiple types of weight and bias values may be prepared according to the scene. The probability calculation unit 404 uses the learned weights and biases (the results of machine learning performed in advance) to calculate the probability value f(y 1 ) based on Equations (1) to (4) and outputs it.

[0053] During the learning of the neural network, the subject information (here, joint position) of the state immediately before moving to an important action can be used as the state of the main subject. For example, in a sport where a ball is thrown, learning can be performed using the joint position detected from an image in a state where the hand is extended forward to throw the ball as one of the states of the main subject. The reason for performing such learning is to enable the imaging device 100 to execute appropriate control for the subject that should be the main subject and has performed the action. For example, when the reliability (probability value) corresponding to the main subject exceeds a preset threshold, by starting control (recording control) to automatically record an image or video, the user can take a picture without missing an important moment. At this time, information on the typical time from the state of the learning target to the important action may be used for the control of the imaging device 100.

[0054] The method for calculating probability using a neural network has been described above. However, if the class classification of whether it is the main subject can be performed, other machine learning methods such as a support vector machine or a decision tree may be used. Also, not limited to machine learning, a function that outputs a reliability or probability value based on a certain model may be constructed. As described in the explanation of FIG. 5, it is also possible to assume that the closer the distance between the subject and the specific object, the higher the reliability of it being the main subject, and use the value of a monotonically decreasing function with respect to the distance between the subject and the specific object.

[0055] In the explanation of FIG. 5, the determination of the main subject was performed using the information of the specific object. However, it is also possible to perform the determination of the main subject using only the pose information of the subject. In addition, data obtained by performing a predetermined transformation such as a linear transformation on the joint position, the position and size of the specific object may be used as input data.

[0056] As described above, according to the present embodiment, the imaging device 100 detects a plurality of poses from the processing target image. Then, the imaging device 100 acquires the information of the priority target, and determines the pose to be prioritized among the plurality of poses based on the information of the priority target. Thereby, it becomes possible to determine the main subject closer to the user's intention in an image in which a plurality of subjects exist.

[0057] In the description of the above embodiment, a configuration for determining whether the subject of the team to be photographed is in a certain pose has been described. However, it can also be applied to a configuration for determining the prioritized pose when searching for an image in a desired pose from the image after shooting. Also, when editing an image, it can be applied to a configuration for determining the pose to be preferentially detected.

[0058] The above embodiments include the following configurations.

[0059] (Configuration 1) First detection means for detecting a specific subject from an image, Second detection means for detecting a pose for each of the subjects detected by the first detection means, For each of the subjects detected by the first detection means, a first acquisition means for acquiring the reliability of being the main subject based on the posture; A second acquisition means for acquiring information on the priority target; An image processing apparatus, comprising: a determination means for determining a main subject based on the reliability and the information on the priority target.

[0060] (Configuration 2) The determination means selects a specific posture from among the plurality of postures detected by the second detection means as the prioritized posture, and determines the main subject based on the prioritized posture. The image processing apparatus according to Configuration 1.

[0061] (Configuration 3) The plurality of postures detected by the second detection means are an attack posture and a defensive posture. The image processing apparatus according to Configuration 1 or 2.

[0062] (Configuration 4) Determine whether the priority target is on the attack side or the defense side, and select one of the attack posture and the defense posture as the prioritized posture. The image processing apparatus according to Configuration 2.

[0063] (Configuration 5) The determination means determines whether the plurality of postures detected by the second detection means are the postures of the priority target, and sets the posture of the priority target as the prioritized posture. The image processing apparatus according to Configuration 2.

[0064] (Configuration 6) Whether it is the posture of the priority target is determined based on the color of the uniform. The image processing apparatus according to Configuration 5.

[0065] (Configuration 7) The image processing apparatus according to any one of Configurations 1 to 6, further comprising a registration means for registering at least one of a goal position, a court position, and a color of a uniform as the information on the priority target.

[0066] (Configuration 8) An image processing apparatus according to Configuration 2, characterized in that the determination process of the priority posture in the determination means is changed according to the competition.

[0067] (Configuration 9) As the determination process of the priority posture, if there is a goal position in the panning direction of the image processing apparatus, the attacking posture is prioritized; if there is a subject moving towards the goal position, the attacking posture is prioritized; if there are subjects concentrated in the direction opposite to the goal position, the defensive posture is prioritized; if there is a ball at the court position, the attacking posture is prioritized; if there is a subject wearing a uniform of a registered color, the posture of the subject is prioritized. The image processing apparatus according to Configuration 8, characterized by using at least any one of them.

[0068] (Configuration 10) The determination means sets a reliability threshold value for detecting a posture according to the priority posture. The image processing apparatus according to Configuration 2, characterized by this.

[0069] (Configuration 11) If the priority posture is an attacking posture, the reliability threshold value is set lower for the attacking posture than for the defensive posture. The image processing apparatus according to Configuration 10, characterized by this.

[0070] (Configuration 12) An imaging means for imaging an image, An imaging apparatus, characterized by having the image processing apparatus according to any one of Configurations 1 to 11, which detects a specific subject using the image imaged by the imaging means by the first detection means.

[0071] (Method 1) Detecting a specific subject from an image, For each of the detected subjects, detecting a posture, For each of the detected subjects, obtaining a reliability as the main subject based on the posture, Obtaining information on the priority target, An image processing method, comprising: determining a main subject based on the reliability and the information of the priority target.

[0072] (Program 1) A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 11.

[0073] (Other Embodiments) As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Some of the above-described embodiments may be appropriately combined.

[0074] In addition, when a software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium or to a system or apparatus having a computer capable of executing the program using wired / wireless communication and the program is executed, it is also included in the present invention.

[0075] Therefore, in order to realize the functional processing of the present invention by a computer, the program code itself supplied to and installed in the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0076] In that case, as long as it has the functions of the program, the form of the program is not limited, such as object code, a program executed by an interpreter, script data supplied to an OS, etc.

[0077] Examples of the recording medium for supplying the program include magnetic recording media such as hard disks and magnetic tapes, optical / photo-magnetic storage media, and non-volatile semiconductor memories.

[0078] Also, as a method of supplying the program, a method may be considered in which a computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

Explanation of Signs

[0079] 100 Imaging device 141 Image sensor 151 Main control unit 152 Image processing unit 201 Image acquisition unit 202 Subject information detection unit 203 Priority target information acquisition unit 204 Priority pose determination unit 205 Main subject determination unit 401 Subject detection unit 402 Pose acquisition unit 403 Object detection unit 404 Probability calculation unit

Claims

1. First detection means for detecting a specific subject from an image, Second detection means for detecting the posture of each of the subjects detected by the first detection means, First acquisition means for acquiring the reliability of being the main subject for each of the subjects detected by the first detection means based on the posture, Second acquisition means for acquiring information on the priority target, An image processing apparatus comprising: determination means for determining a main subject based on the reliability and the information on the priority target.

2. The determination means selects a specific posture from among a plurality of postures detected by the second detection means as a priority posture, and determines the main subject based on the priority posture. The image processing apparatus according to claim 1.

3. The plurality of postures detected by the second detection means are an attack posture and a defense posture. The image processing apparatus according to claim 1.

4. Determine whether the priority target is on the attack side or the defense side, and select one of the attack posture and the defense posture as the priority posture. The image processing apparatus according to claim 2.

5. The determination means determines whether a plurality of postures detected by the second detection means are the postures of the priority target, and sets the posture of the priority target as the priority posture. The image processing apparatus according to claim 2.

6. Whether it is the posture of the priority target is determined based on the color of the uniform. The image processing apparatus according to claim 5.

7. The image processing apparatus according to claim 1, further comprising registration means for registering at least one of a goal position, a court position, and the color of a uniform as information on the priority target.

8. The image processing apparatus according to claim 2, wherein the determination process of the priority posture by the determination means is changed according to the competition.

9. As the determination process of the priority posture, if there is a goal position in the panning direction of the image processing apparatus, the attack posture is prioritized; if there is a subject moving towards the goal position, the attack posture is prioritized; if the subjects are concentrated in the direction opposite to the goal position, the defense posture is prioritized; if there is a ball at the court position, the attack posture is prioritized; the posture of the subject wearing the registered color uniform is prioritized. The image processing apparatus according to claim 8, characterized by using at least one of them.

10. The image processing apparatus according to claim 2, wherein the determination means sets a reliability threshold value for detecting a posture according to the prioritized posture.

11. The image processing apparatus according to claim 10, wherein if the prioritized posture is an attack posture, the reliability threshold value is set lower for the attack posture than for the defense posture.

12. An imaging device comprising: imaging means for imaging an image; The image processing apparatus according to any one of claims 1 to 11, wherein the first detection means detects a specific subject using the image imaged by the imaging means.

13. Detecting a specific subject from an image; For each of the detected subjects, detecting a posture; For each of the detected subjects, obtaining a reliability of being a main subject based on the posture; Obtaining information on a priority target; An image processing method, comprising: determining a main subject based on the reliability and the information on the priority target.

14. A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 11.

Images