Image processing apparatus, method for controlling image processing apparatus, and storage medium

Abstract

To provide a technique for determining a main subject close to the intention of a photographer in consideration of posture information and information on a registered person when there are a plurality of subjects.SOLUTION: The image processing apparatus includes a detection unit configured to detect an object and a specific posture of the object from an image, an authentication unit configured to authenticate the specific object based on information about the specific object registered in advance, and a determination unit configured to determine a main object from a plurality of objects detected from the image based on a detection result of the specific posture by the detection unit and an authentication result by the authentication unit. The method of claim 1, further comprising: SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, and more particularly to a main subject determination device. [Background technology]

[0002] There are technologies that determine a main subject from multiple subjects and maintain focus on the main subject when taking multiple shots in a continuous shooting sequence or when shooting video. However, particularly in ball game scenes, multiple subjects may overlap, and a subject not intended by the photographer may be determined to be the main subject. Patent Document 1 discloses a technology that acquires posture information of the subject and determines the main subject based on the reliability calculated from the posture information. Patent Document 2 also discloses an imaging device that automatically selects a registered person using face recognition if the person is registered in advance as a tracking target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-71794 A [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-187591 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional techniques disclosed in Patent Documents 1 and 2 mentioned above have not considered determining the main subject when there is a subject having both posture information and registered person information.

[0005] Therefore, an object of the present invention is to provide a technique for determining a main subject that is closest to the photographer's intention based on posture information and registered person information when there are multiple subjects. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present invention is characterized by having a detection means for detecting a subject and a specific posture of the subject from an image, an authentication means for authenticating the specific subject based on information about the specific subject that has been registered in advance, and a determination means for determining a main subject from a plurality of subjects detected from an image based on the detection result of the specific posture by the detection means and the authentication result by the authentication means. [Effects of the Invention]

[0007] According to the present invention, even when there are multiple subjects, it is possible to determine the main subject that is closest to the photographer's intention based on posture information and registered person information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging device 100. [Figure 2] FIG. 2 is a block diagram showing a detailed configuration of a part of an image processing unit 152 according to the first to sixth embodiments. [Figure 3] 4 is a flowchart of a main subject determination process according to the first embodiment. [Figure 4] 10 shows examples of main subject candidates when main subject determination section 212 determines a main subject in the first embodiment. [Figure 5] 10 is a flowchart of a main subject determination process according to the second embodiment. [Figure 6] 10 is a flowchart of a main subject determination process according to the third embodiment. [Figure 7] 10 shows examples of main subject candidates when main subject determination section 212 determines a main subject in the third embodiment. [Figure 8] 10 is a flowchart of a main subject determination process according to a fourth embodiment. [Figure 9] 13 shows examples of main subject candidates when main subject determination section 212 determines a main subject in the fourth embodiment. [Figure 10] 13 is a flowchart of a main subject determination process according to the fifth embodiment. [Figure 11] 13 shows examples of main subject candidates when main subject determination section 212 determines a main subject in the fifth embodiment. [Figure 12] 13 is a flowchart of a main subject determination process according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] First Embodiment ■ Overall structure FIG. 1 is a block diagram showing the configuration of an imaging device 100 including a main subject determination device. The imaging device 100 may be, but is not limited to, a digital still camera or video camera that captures an image of a subject and records video and still image data on various media, such as tape, solid-state memory, optical disk, or magnetic disk. The following description will be given using a person as an example of a specific subject that may be determined as the main subject. Specifically, the imaging device 100 performs a process to determine, from among multiple people detected in an input captured image, a main subject that is the target of imaging control, such as autofocus and automatic exposure control, based on information and calculations described below. Note that the subject is not limited to a person; the present invention can be applied to any subject with distinguishable features, such as animals, other living creatures, cars, or buildings. The configuration shown in FIG. 1 is merely an example of the configuration of the imaging device 100.

[0011] Each unit in the imaging device 100 is connected via a bus 160. Furthermore, each unit is controlled by a main control unit 151.

[0012] The lens unit 101 includes a fixed first-group lens 102, a zoom lens 111, an aperture 103, a fixed third-group lens 121, and a focus lens 131. The aperture control unit 105 adjusts the aperture diameter of the aperture 103 and controls the amount of light during shooting by driving the aperture 103 via an aperture motor 104 (AM) in accordance with commands from the main control unit 151. The zoom control unit 113 changes the focal length by driving the zoom lens 111 via a zoom motor 112 (ZM). The focus control unit 133 determines the drive amount for driving the focus motor 132 (FM) based on the amount of deviation in the focus direction of the lens unit 101. Additionally, the focus control unit 133 controls the focus adjustment state by driving the focus lens 131 via the focus motor 132 (FM). AF (autofocus) control is achieved by controlling the movement of the focus lens 131 by the focus control unit 133 and the focus motor 132. The focus lens 131 is a lens for adjusting focus, and although it is simply shown as a single lens in FIG. 1, it is usually made up of multiple lenses.

[0013] The subject image formed on the image sensor 141 via the lens unit 101 is converted into an electrical signal by the image sensor 141. The image sensor 141 is a photoelectric conversion element that photoelectrically converts the subject image (optical image) into an electrical signal, and is, for example, a CMOS sensor. The image sensor 141 has light receiving elements arranged with m pixels in the horizontal direction and n pixels in the vertical direction. The image formed on the image sensor 141 and photoelectrically converted is arranged as an image signal (image data) by the image signal processing unit 142. This makes it possible to acquire an image on the imaging surface.

[0014] The image data output from the imaging signal processing unit 142 is sent to the imaging control unit 143 and temporarily stored in a RAM (random access memory) 154. The image data stored in the RAM 154 is compressed by an image compression / decompression unit 153 and then recorded on an image recording medium 157. In parallel with this, the image data stored in the RAM 154 is sent to the image processing unit 152.

[0015] Image processing unit 152 applies predetermined image processing to image data stored in RAM 154. The image processing applied by image processing unit 152 includes, but is not limited to, so-called development processing such as white balance adjustment processing, color interpolation (demosaic) processing, and gamma correction processing, as well as signal format conversion processing and scaling processing. Image processing unit 152 also determines a main subject based on posture information of the subject (e.g., joint positions) and position information of objects specific to the scene (hereinafter referred to as specific objects). Image processing unit 152 may use information on the main subject (processing results) obtained by the main subject determination processing for other image processing (e.g., white balance adjustment processing). Image processing unit 152 stores in RAM 154 the image data to which the predetermined image processing has been applied, posture information of each subject, information on the positions and sizes of the specific objects, and position information of the center of gravity, face, and pupils of the main subject.

[0016] The operation unit 156 is an input interface including a touch panel, buttons, and the like, and allows the user to perform various operations on the imaging device 100 by selecting various function icons displayed on the display unit 150 .

[0017] The main control unit 151 has one or more programmable processors, such as a CPU or MPU. The main control unit 151 controls each unit of the imaging device 100 by loading a program stored in, for example, a flash memory 155 into a RAM 154 and executing the program, thereby realizing the functions of the imaging device 100. The main control unit 151 also executes automatic exposure control (AE) processing that automatically determines exposure conditions (shutter speed or accumulation time, aperture value, and sensitivity) based on information about the brightness of the subject. The information about the brightness of the subject can be acquired from, for example, the image processing unit 152. The main control unit 151 can also determine exposure conditions based on the area of ​​a specific subject, such as a person's face.

[0018] A focus control unit 133 performs AF control for the position of the main subject stored in the RAM 154. An aperture control unit 105 performs exposure control using the brightness value of a specific subject area.

[0019] Display unit 150 displays images, main subject detection results, etc. Battery 159 is appropriately managed by power management unit 158 ​​and provides a stable power supply to the entire imaging device 100. Flash memory 155 stores control programs necessary for the operation of imaging device 100, parameters used for the operation of each unit, etc. When imaging device 100 is started up by a user operation (when the imaging device 100 transitions from a power-off state to a power-on state), the control programs and parameters stored in flash memory 155 are loaded into a part of RAM 154. Main control unit 151 controls the operation of imaging device 100 in accordance with the control programs and constants loaded into RAM 154.

[0020] ■ Main subject detection processing The main subject determination process executed by image processing unit 152 will be described with reference to Figures 2 and 3. Figure 2 is a block diagram showing a portion of the detailed configuration of image processing unit 152. Figure 3 is a flowchart of the main subject determination process. Unless otherwise specified, the processing of each step in this flowchart is realized by each unit of image processing unit 152 operating under the control of main control unit 151. In the following, a ball game played by multiple people will be described as a shooting scene that is the target of the main subject determination process, but the shooting scene to which this embodiment can be applied is not limited to this.

[0021] In S301, the image acquisition unit 201 acquires an image captured at a time of interest from the imaging control unit 143. In S302, the subject detection unit 202 detects multiple subjects in the image acquired by the image acquisition unit 201. Here, as an example, a case will be described where the detected subjects are people.

[0022] In S303 to S305, the orientation detection unit 203 detects the orientation of the subjects detected by the subject detection unit 202, and acquires orientation information about each subject.

[0023] First, in S303, the posture acquisition unit 204 uses a first learning model learned by machine learning to acquire a plurality of joint points of the subject and their positions as feature points (key points).

[0024] In S304, object detection unit 206 detects a specific object (a predetermined type of object) in the image acquired by image acquisition unit 201, and acquires the two-dimensional coordinates and size of the specific object in the image. The type of specific object to be detected is determined based on the captured scene of the image. In this case, since the captured scene is a ball game, object detection unit 206 detects a ball as the specific object, but the specific object is not limited to these, and may be anything that contributes to detecting a specific posture, such as a goal or a racket in a racket game.

[0025] In S305, the posture of the subject is estimated based on the acquired information on the joint points and their positions using a second learning model trained by machine learning. The posture estimation method is not limited to this, and rule-based processing alone may be used, or posture detection may be performed in combination with machine learning. Furthermore, any method for detecting the posture of the subject may be used, such as posture detection using multiple frames. The probability calculation unit 205 calculates a posture detection reliability (probability, score) representing the likelihood of a specific posture for each subject based on at least one of the joint coordinates estimated by the posture acquisition unit 204 and the coordinates and size of the unique object acquired by the object detection unit 206. Here, a value other than probability may be used as the posture detection reliability. For example, the reciprocal of the distance between the center of gravity of the subject and the center of gravity of the unique object may be used as the posture detection reliability. In this embodiment, the likelihood of a specific posture is detected using information on the unique object as well, but it is also possible to detect the likelihood of a specific posture using only the posture information of the subject. Alternatively, data obtained by performing a predetermined transformation, such as a linear transformation, on the joint positions and the positions and sizes of the unique objects may be used as input data. The orientation detection reliability calculated in S305 is stored in the RAM 154 for use in the subsequent main subject determination process.

[0026] In S306 to S308, the registration and authentication unit 208 acquires information about registered subjects that have been registered in advance, and performs authentication to determine whether each or some of the subjects detected by the subject detection unit 202 matches the information about the registered subjects. For subjects that are successfully authenticated, the registration and authentication unit 208 acquires the subject priority included in the information about the registered subjects. Here, S303 to S305 and S306 to S308 do not necessarily have to be processed in this order, and the order may be reversed so that S303 to S305 are processed after S306 to S308.

[0027] Here, the information of the registered subject to be registered in advance and the procedure of the registration process will be described. Registration, like the main subject determination process, is performed by the image processing unit 152. The subject detection unit 202 detects subjects from captured image data for registration (hereinafter referred to as the registration image) prepared by the user, and the user selects a person to register from the detected subjects. The feature extraction unit 209 extracts feature information of the selected detected subject from the image portion of the registration image. Algorithms used to extract feature information include a rule-based algorithm that acquires feature information from the coordinates of feature points of facial features such as the eyes, nose, and mouth, and an algorithm that inputs an input image to a neural network and acquires feature information as its output. The registration information output unit 214 associates the extracted feature information of the subject, the subject area image, the subject priority, the header information, and the subject information, and stores them in the RAM 154 as a single piece of registration information. The subject priority includes a priority among multiple registered subjects. This priority may be specified by the user, or the priority may be set on the camera side based on the order of registration, the frequency of authentication after registration, etc.

[0028] 3, in S306, the feature extraction unit 209 extracts feature information of the detected subject from the image portion of the detected subject. The feature extraction unit 209 inputs the extracted feature information to the authentication unit 210.

[0029] In S307 , the registration information acquisition unit 211 acquires the feature information and subject priority of the registered subject from the RAM 154 and inputs them to the authentication unit 210 .

[0030] In S308, the authentication unit 210 calculates the similarity between the acquired feature information of the registered subject and the feature information of the detected subject input from the feature extraction unit 209, and generates an authentication reliability that represents this similarity. If the authentication reliability is equal to or greater than a predetermined threshold, the authentication unit 210 determines that the authentication is successful (the detected subject is a registered subject), and if it is less than the predetermined threshold, it determines that the authentication is unsuccessful, and stores the determination result in RAM 154 together with the authentication reliability and the subject priority.

[0031] In S309, the main subject determination unit 212 performs a main subject determination process based on the orientation detection reliability included in the orientation information acquired in S305 and the subject priority acquired in S308. The orientation detection reliability and subject priority are added together for each subject to calculate a score of main subject likelihood (hereinafter referred to as "main subject score"), and the subject with the highest main subject score is determined to be the main subject. The orientation detection reliability and subject priority scores may be tuned (normalized, etc.) in advance so that they can be evaluated by simple addition. Alternatively, a weight corresponding to which is given more importance may be adjusted by user settings, and the total main subject likelihood score may be calculated by weighted addition.

[0032] 4 shows an example of subject information when the main subject determination unit 212 determines a main subject. An ID is assigned to each detected subject, and information on posture detection reliability and subject priority is attached to each subject. When there is multiple pieces of subject information as in FIG. 4(a), the subject 401 with the highest main subject score is determined to be the main subject.

[0033] As described above, the main subject is determined based on the total score. Therefore, even if a subject has a posture detection reliability below a predetermined reliability threshold, it may be determined to be the main subject if it has the highest main subject score. In Figure 4(b), if the posture reliability threshold is set to 90, subject 409 has a posture reliability below the threshold. However, subject 409 has the highest main subject score, which is the sum of posture reliability and subject priority. In this case, subject 409 is determined to be the main subject because it is most likely the main subject, even if its posture detection reliability is below the threshold. By increasing the subject priority in this way, a subject that should have been determined to be the main subject but was not detected because its posture reliability was lower than the standard can be determined to be the main subject. Furthermore, for subjects for which no subject priority is registered, the posture detection reliability is used as the main subject score without using the subject priority, and the person with the highest score is determined to be the main subject.

[0034] Then, main subject determination section 212 stores the coordinates of the joints of the main subject and representative coordinates representing the main subject (such as the center of gravity position and face position) as main subject information in RAM 154. This completes the main subject determination process.

[0035] As described above, according to the first embodiment, when there are multiple main subject candidates, the main subject score is calculated from the pose detection reliability and the subject priority, and the subject with the highest main subject score is selected as the main subject. This makes it possible to select the main subject taking into consideration the pose detection information and the registered subject information.

[0036] <Second embodiment> In the second embodiment, the basic configuration of the imaging device 100 is the same as in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0037] FIG. 5 is a flowchart of the main subject determination process in this embodiment. After acquiring the subject's posture reliability in S505, posture type estimation unit 207 estimates the posture type of the subject whose posture has been detected in S506. The posture type is estimated using the joint coordinates estimated by posture acquisition unit 204 and the coordinates of the unique object acquired by object detection unit 206. For example, in the case of volleyball, if the wrist joint of one hand is above the head and the ball, which is a unique object, is located near the wrist, it can be estimated that the posture is for spiking. Here, the posture type is estimated using the joint coordinates and the coordinates of the unique object, but posture type may also be estimated using other methods such as deep learning.

[0038] In S510, the main subject determination unit 212 performs a main subject determination process based on the posture type estimated in S506 and the subject priority level acquired in S308. Among the detected subjects, subjects with postures related to the score are determined as main subjects in the following order of priority: subjects with postures related to the score, registered subjects, and subjects with postures not related to the score. For example, if a subject with a posture related to the score that should be prioritized is detected, the subject with the posture related to the score that should be prioritized is determined as the main subject, even if the main subject scores of the registered subjects and subjects with postures not related to the score are higher. In this way, when an item that should be prioritized is detected or authenticated, there is a method of determining the corresponding subject as the main subject. Alternatively, a method of calculating the main subject score by assigning a higher weight to subjects corresponding to items that should be prioritized may be employed.

[0039] According to the determination method of this embodiment, when there are multiple registered subjects, the main subject is determined in order of subject priority. If a subject in a pose related to scoring, such as shooting, is present within the field of view, that subject is given priority, and if there are no subjects related to scoring, a registered subject is given priority, allowing for selection of a main subject according to the situation. Examples of poses related to scoring include shooting or heading in soccer, spiking in volleyball, and shooting in basketball. However, these poses are not limited to these, and poses related to scoring can be determined depending on the sport. Conversely, poses not related to scoring are determined to be at least one of passing and dribbling in soccer and basketball, and at least one of tossing and receiving in volleyball.

[0040] Furthermore, if the main subject has already been determined to be a registered subject in the main subject determination before the time of interest, the main subject may not be switched depending on the posture type of the detected subject. For example, even if the main subject is a registered subject and only subjects with postures that do not contribute to the score appear, the main subject will remain a registered subject.

[0041] As described above, according to the second embodiment, when there are multiple main subject candidates, the main subject is determined based on the posture type and the subject priority, which makes it possible to select the main subject taking into consideration posture detection information and registered subject information.

[0042] <Third embodiment> In the third embodiment, the basic configuration of the imaging device 100 is the same as in the first and second embodiments. Below, differences from the first embodiment will be mainly described.

[0043] 6 is a flowchart of the main subject determination process in this embodiment. In S609, the main subject determination unit 212 performs the main subject determination process by prioritizing the posture detection reliability included in the posture information acquired in S605. Of the detected subjects, the subject with the highest posture detection reliability is determined to be the main subject. In other words, if there is a subject (first subject) with a specific posture detected, the subject with the specific posture detected is prioritized, regardless of how high the authentication reliability of the subjects (subjects other than the first subject) with which the specific subject was identified were detected.

[0044] If there is no detected subject with a posture detection reliability equal to or greater than the threshold (if no specific posture is detected from any of the subjects), the main subject is more likely to be determined in the order of registered subjects with high subject priority and detected subjects without posture information or authentication information. With the control described above, even if the main subject was determined to be a registered subject in the main subject determination before the time of interest, the main subject is switched to the subject with the highest posture detection reliability. Figure 7 shows an example of subject information when the main subject determination unit 212 determines the main subject. When there is multiple subject information as shown in Figure 7(a), the subject 703 with the highest posture reliability is determined as the main subject. Furthermore, when the posture reliability threshold is set to 100 in Figure 7(b), there is no detected subject with a posture detection reliability equal to or greater than the threshold, so the subject 709 with the highest subject priority is determined as the main subject. Through the above main subject determination process, for example, in a sports game, if a person is taking a posture such as shooting, the main subject can be prioritized as the main subject. Furthermore, when a person taking a posture such as shooting is not within the field of view, the registered person can be prioritized as the main subject.

[0045] As described above, according to the third embodiment, when there are multiple main subject candidates, the main subject is determined based on the posture type. Among the detected subjects, the subject with the highest posture detection reliability is determined to be the main subject. If there is no subject among the detected subjects with a posture detection reliability equal to or higher than a threshold, the main subject is selected in the following order: registered subjects with high subject priority, and detected subjects without posture information or authentication information. This makes it possible to select the main subject while prioritizing subjects with posture detection and taking into consideration the information of the registered subjects.

[0046] <Fourth embodiment> In the fourth embodiment, the basic configuration of the imaging device 100 is the same as in the first to third embodiments. Below, differences from the first embodiment will be mainly described. In this embodiment, instead of calculating a score based on the subject priority registered when a specific subject registered in advance is authenticated in the first to third embodiments, the reliability of the authentication result is calculated and the reliability is reflected in the score.

[0047] FIG. 8 is a flowchart of the main subject determination process in this embodiment. In S809, the main subject determination unit 212 performs the main subject determination process based on the orientation detection reliability included in the orientation information acquired in S805 and the authentication reliability acquired in S808. The orientation detection reliability and authentication reliability are added together for each subject to calculate a main subject score, and the subject with the highest score is determined to be the main subject. FIG. 9 shows an example of subject information used by the main subject determination unit 212 to determine a main subject. When there is multiple pieces of subject information as shown in FIG. 9, the subject 901 with the highest main subject score is determined to be the main subject. In this embodiment, simple addition is used to calculate the main subject score, but other calculation methods such as multiplication may be used, or the score may be weighted.

[0048] As described above, according to the fourth embodiment, when there are multiple main subject candidates, the main subject score is calculated from the pose detection reliability and the authentication reliability, and the subject with the highest main subject score is selected as the main subject. This makes it possible to select the main subject taking into consideration the pose detection information and the information of the registered subjects.

[0049] <Fifth embodiment> In the fifth embodiment, the basic configuration of the imaging device 100 is the same as in the first to fourth embodiments. Below, differences from the first embodiment will be mainly described.

[0050] 10 is a flowchart of the main subject determination process in this embodiment. In S1009, the main subject determination unit 212 performs main subject determination by giving top priority to the authentication information acquired in S1008. That is, if there is a subject (second subject) that has been recognized as a specific subject, the subject recognized as the specific subject is given priority and determined to be the main subject, no matter how highly reliable the detection reliability of a subject (subject other than the second subject) whose specific posture has been detected.

[0051] If there are no registered subjects among the detected subjects (i.e., if no specific subject is identified from any of the subjects), the subjects with the highest pose detection reliability are determined to be the main subject, followed by detected subjects with no pose information or authentication information. FIG. 11 shows an example of subject information when the main subject determination unit 212 determines the main subject. When there is multiple subject information as shown in FIG. 11(a), the subject 1102 with the highest subject priority is determined to be the main subject. Also, in FIG. 11(b), since there are no registered subjects among the detected subjects, the subject 1108 with the highest pose detection reliability is determined to be the main subject. With the above main subject determination process, for example, if you want to prioritize photographing a specific person in a sports game, you can continue to select that person as the main subject as long as that person is within the field of view. Also, when that person is not within the field of view, a person taking a pose such as a shot can be prioritized as the main subject.

[0052] As described above, according to the fifth embodiment, when there are multiple main subject candidates, the main subject is determined based on the subject priority. Among the detected subjects, the subject with the highest subject priority is determined to be the main subject, and if there are no registered subjects among the detected subjects, the main subject is selected in the order of subjects with high posture detection reliability, and detected subjects without posture information or authentication information. This makes it possible to select the main subject while prioritizing registered subjects and taking posture detection information into consideration.

[0053] Sixth Embodiment In the sixth embodiment, the basic configuration of the imaging device 100 is the same as in the first to fifth embodiments. The following mainly describes the differences from the first embodiment. This embodiment can be incorporated into the first to fifth embodiments as an interrupt process. That is, in each embodiment, a configuration may be adopted in which a user can set a priority that takes precedence over the priority setting in each embodiment.

[0054] 12 is a flowchart of the main subject determination process in this embodiment. In S1209, the priority setting acquisition unit 213 acquires the priority setting set by the user. The priority setting is used to specify whether to prioritize the posture detection subject or the registered subject as the main subject. For example, on a selection screen displayed on the display unit 150, the user can select the subject to prioritize from automatic, posture priority, and authentication priority. If automatic is set, the process proceeds to S1210; if posture priority is set, the process proceeds to S1212; and if authentication priority is set, the process proceeds to S1213.

[0055] In S1210, the main subject determination unit 212 checks whether subject priority has been registered. If registered, the process proceeds to S1211, and if not, the process proceeds to S1212 since the same processing as for posture priority may be performed.

[0056] In S1211, the main subject determination unit 212 performs main subject determination processing based on the orientation detection reliability included in the orientation information acquired in S1205 and the subject priority acquired in S1208. The processing in S1211 is the same as S311 in the first embodiment.

[0057] In S1212, the main subject determination unit 212 performs main subject determination processing by prioritizing the orientation detection reliability included in the orientation information acquired in S1205. The processing in S1212 is the same as S609 in the third embodiment.

[0058] In S1213, the main subject determination unit 212 performs main subject determination by prioritizing the subject priority included in the authentication information acquired in S1208. The process in S1212 is the same as S1009 in the fifth embodiment.

[0059] As described above, according to the sixth embodiment, the user can set whether to prioritize the orientation detection subject or the registered subject as the main subject, which makes it possible to determine the main subject in accordance with the user's wishes while taking into account the orientation detection information and the registered subject information.

[0060] (Other embodiments) The object of the present invention can also be achieved as follows: A storage medium storing software program code describing procedures for realizing the functions of each of the above-described embodiments is supplied to a system or device, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code stored in the storage medium.

[0061] In this case, the program code itself read from the storage medium will realize the novel functions of the present invention, and the storage medium storing the program code and the program will constitute the present invention.

[0062] Furthermore, examples of storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, etc. Also usable are CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0063] The functions of the above-described embodiments are realized by making the computer executable the read program code. Furthermore, the functions of the above-described embodiments may be realized by an operating system (OS) or the like running on the computer performing some or all of the actual processing based on the instructions of the program code.

[0064] The following case is also included: First, program code is read from a storage medium and written into memory on an expansion board inserted into a computer or on an expansion unit connected to the computer. Then, based on the instructions of the program code, a CPU or other device on the expansion board or unit performs some or all of the actual processing.

[0065] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0066] (Configuration 1) a detection means for detecting a subject and a specific posture of the subject from an image; an authentication means for authenticating a specific subject based on information about the specific subject that has been registered in advance; a determining means for determining a main subject from among a plurality of subjects detected from an image based on a result of detection of the specific posture by the detecting means and an authentication result by the authenticating means; 1. An image processing device comprising:

[0067] (Configuration 2) The image processing device according to configuration 1, characterized in that when the specific posture of a first subject is detected by the detection means, the determination means determines the first subject to be a main subject, even if there is a subject other than the first subject that has been recognized as the specific subject by the authentication means.

[0068] (Configuration 3) the detection means calculates a score of the likelihood of the specific posture, and the authentication means determines a priority of the authenticated specific subject based on a priority of the subject set for each of the specific subjects; 3. The image processing device according to configuration 1 or 2, wherein the determining means determines the main subject based on the score and the priority.

[0069] (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the determining means determines the main subject based on a posture type included in the posture information of the subject.

[0070] (Configuration 5) The image processing device according to any one of configurations 1 to 4, wherein the detection means detects each joint point of the subject and the position of each joint point from the image, and detects a posture related to scoring and a posture not related to scoring based on the position of each joint and the position of the ball.

[0071] (Configuration 6) The image processing device according to configuration 5, wherein the postures related to scoring include at least one of a shooting posture and a spiking posture, and the postures not related to scoring include at least one of a passing posture, a receiving posture and a toss posture.

[0072] (Configuration 7) The image processing device according to any one of configurations 1 to 6, wherein the determination means, when a second subject is recognized as the specific subject, determines the second subject as the main subject even if there is a subject other than the second subject for which the specific posture has been detected.

[0073] (Configuration 8) the detection means calculates a score of the likelihood of the specific pose, and the authentication means calculates an authentication reliability indicating the likelihood of the specific subject; 8. The image processing device according to any one of configurations 1 to 7, wherein the determining means determines the main subject based on the score and the authentication reliability.

[0074] (Configuration 9) a setting means for setting, by a user operation, which of the detection result of the specific posture by the detection means and the authentication result by the authentication means should be given higher priority in determining a main subject, 9. The image processing device according to any one of configurations 1 to 8, wherein the determining means determines the main subject based on the setting by the setting means.

[0075] (Configuration 10) the determining means determines a main subject for an image captured by the imaging means, 10. The image processing device according to any one of configurations 1 to 9, further comprising a control unit that controls the imaging unit based on information about the main subject determined by the determination unit.

[0076] (Configuration 11) 11. The image processing device according to any one of configurations 1 to 10, wherein the detection means detects each joint point of the subject and a position of each joint point from the image, and detects the specific posture by calculating a score of likelihood of the specific posture from the position of each joint point.

[0077] (Configuration 12) The image processing device described in configuration 11, characterized in that the detection means detects each joint point of the subject and the position of each joint point using a first learning model that has been machine-learned, and calculates a score of the likelihood of the specific pose using a second learning model that has been machine-learned.

[0078] (Method 1) a detection step of detecting an object and a specific posture of the object from the image; an authentication step of authenticating a specific subject based on information about the specific subject that has been registered in advance; a determining step of determining a main subject from a plurality of subjects detected from an image based on a detection result of the specific posture by the detection means and an authentication result by the authentication means; 1. A method for controlling an image processing apparatus, comprising:

[0079] (Program 1) A computer-executable program that describes the steps of the method for controlling an image processing device according to Method 1. [Explanation of symbols]

[0080] 100 Imaging device 141 Image sensor 151 Main control unit 152 Image processing section 201 Image acquisition unit 202 Subject detection unit 203 Attitude detection unit 204 Attitude acquisition part 205 Probability Calculation Unit 206 Object detection unit 207 Posture type estimation unit 208 Registration and Authentication Department 209 Feature Extraction Unit 210 Authentication Department 211 Registration Information Acquisition Department 212 Main subject determination section 213 Priority setting acquisition section 214 Registration information output section

Claims

1. a detection means for detecting a subject and a specific posture of the subject from an image; an authentication means for authenticating a specific subject based on information about the specific subject that has been registered in advance; a determining means for determining a main subject from among a plurality of subjects detected from an image based on a result of detection of the specific posture by the detecting means and an authentication result by the authenticating means; 1. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein when the specific posture of a first subject is detected by the detection means, the determination means determines the first subject to be the main subject, even if there is a subject other than the first subject that has been recognized as the specific subject by the authentication means.

3. the detection means calculates a score of the likelihood of the specific posture, and the authentication means determines a priority of the authenticated specific subject based on a priority of the subject set for each of the specific subjects; The image processing device according to claim 1 , wherein the determining unit determines the main subject based on the score and the priority.

4. 4. The image processing apparatus according to claim 3, wherein the determining means determines the main subject based on a posture type included in the posture information of the subject.

5. 2. The image processing device according to claim 1, wherein the detection means detects each joint point of the subject and the position of each joint point from the image, and detects a posture related to scoring and a posture not related to scoring based on the position of each joint and the position of the ball.

6. 6. The image processing device according to claim 5, wherein the poses related to scoring include at least one of a shooting pose and a spiking pose, and the poses not related to scoring include at least one of a passing pose, a receiving pose and a toss pose.

7. 2. The image processing device according to claim 1, wherein when a second subject is recognized as the specific subject, the determining means determines the second subject as the main subject even if there is a subject other than the second subject for which the specific posture has been detected.

8. the detection means calculates a score of the likelihood of the specific pose, and the authentication means calculates an authentication reliability indicating the likelihood of the specific subject; The image processing apparatus according to claim 1 , wherein the determining unit determines the main subject based on the score and the authentication reliability.

9. a setting means for allowing a user to set which of the detection result of the specific posture by the detection means and the authentication result by the authentication means should be given higher priority in determining a main subject, 9. The image processing device according to claim 1, wherein the determining unit determines the main subject based on the setting made by the setting unit.

10. the determining means determines a main subject for an image captured by the imaging means, 9. The image processing apparatus according to claim 1, further comprising a control unit that controls the image capturing unit based on information about the main subject determined by the determination unit.

11. 9. The image processing device according to claim 1, wherein the detection means detects each joint point of the subject and a position of each joint point from the image, and detects the specific posture by calculating a score of likelihood of the specific posture from the position of each joint point.

12. 12. The image processing device according to claim 11, wherein the detection means detects each joint point of the subject and the position of each joint point using a first learning model that has been machine-learned, and calculates a score of the likelihood of the specific pose using a second learning model that has been machine-learned.

13. a detection step of detecting an object and a specific posture of the object from the image; an authentication step of authenticating a specific subject based on information about the specific subject that has been registered in advance; a determining step of determining a main subject from a plurality of subjects detected from an image based on a detection result of the specific posture by the detection means and an authentication result by the authentication means; 1. A method for controlling an image processing apparatus, comprising:

14. A computer-executable program that describes the steps of the method for controlling an image processing apparatus according to claim 13.

Images