Determination device, imaging device, determination method, and program

Abstract

To improve the accuracy of behavioral determination based on body parts of subjects detected in images. [Solution] The present invention provides an action determination device comprising: detection means that performs detection processing to detect multiple types of body parts of a subject in an image; selection means that selects one of a plurality of determination methods for determining the subject's behavior based on the results of the detection processing, wherein each of the plurality of determination methods uses the positional relationship of two or more types of body parts from the plurality of types of body parts to determine the behavior; and control means that controls the determination of the subject's behavior according to the determination method selected by the selection means.

Description

Technical Field

[0006] , , , , , , , ,

[0005] , , , , ,

[0001] The present invention relates to an action determination device, an imaging device, an action determination method, and a program.

Background Art

[0002] In the fields of sports analysis and security, there is a demand for detecting specific postures and actions of people in sports images and surveillance camera images. The technology of human pose estimation (hereinafter referred to as "pose estimation") is a technology that attempts to represent the pose of a person in an image with a simple skeleton by estimating each joint of the person and its connection. By performing learning and inference with a machine learning model based on this human pose information, specific postures and actions can be detected.

[0003] By pose estimation using the method of Non-Patent Document 1, a plurality of body parts of a person in an image (for example, head, neck, both shoulders, both elbows, both wrists, both pelvises, both knees, both ankles, etc.) can be detected. Learning can be performed with a machine learning model based on data such as the coordinates of the detected plurality of body parts, the distances between body parts, and angles, and actions can be determined (inferred).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] 在翻译过程中,我严格遵循了您提供的规则,保留了所有的文本标签、特殊占位符,并确保了翻译后的文本与原文具有相同的行数和格式。对于一些特定的术语和文献名称,我尽量采用了常见的英文表达方式。如果您对翻译结果有任何疑问或需要进一步的解释,请随时告诉我。However, depending on the situation, some body parts may not be detected, or the detection reliability of detected body parts may be low. For example, in sports such as basketball or rugby, where people are densely packed, or in crowded situations with surveillance cameras, ankles may be hidden and not detected, or the detection reliability may be low.

[0007] In this case, in order to perform inference using a model trained on all body parts to be detected, it is necessary to use information such as initial values ​​(or default values) or to use unreliable information as is. As a result, the inference accuracy may decrease, and the accuracy of behavioral determination may decline.

[0008] Patent Document 1 discloses switching the parameters of a posture estimation model using the geometric relationship with the subject being photographed. In this case, the decrease in the accuracy of posture estimation can be suppressed, but it cannot address cases where body parts cannot be detected or where the detection reliability is low.

[0009] This invention has been made in view of the above circumstances, and aims to provide a technology that improves the accuracy of behavioral determination based on body parts of a subject detected in an image. [Means for solving the problem]

[0010] To solve the above problems, the present invention provides an action determination device comprising: detection means for performing detection processing to detect multiple types of body parts of a subject in an image; selection means for selecting one of a plurality of determination methods for determining the action of the subject based on the result of the detection processing, wherein each of the plurality of determination methods uses the positional relationship of two or more types of body parts from the plurality of types of body parts to determine the action; and control means for controlling the determination of the action of the subject according to the determination method selected by the selection means. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the accuracy of behavioral determination based on body parts of a subject detected in an image.

[0012] Further features and advantages of the present invention will become clearer from the accompanying drawings and the description of the embodiments for carrying out the invention below. [Brief explanation of the drawing]

[0013] [Figure 1] A block diagram showing an example of the functional configuration of a lens-interchangeable camera, which is an example of an imaging device according to the first embodiment. [Figure 2] A flowchart for the process of determining the subject's behavior (behavior determination process). [Figure 3] A diagram illustrating an example of the process of connecting body parts. [Figure 4] A diagram illustrating examples of multiple determination methods (options for determination methods) and examples of criteria for selecting a determination method according to the first embodiment. [Figure 5] A diagram illustrating another example of the selection criteria for the determination method according to the first embodiment. [Figure 6] A diagram illustrating yet another example of selection criteria for a determination method according to the first embodiment. [Figure 7] A diagram illustrating examples of multiple determination methods (options for determination methods) and examples of criteria for selecting a determination method according to the second embodiment. [Modes for carrying out the invention]

[0014] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0015] [First Embodiment] FIG. 1 is a block diagram showing a functional configuration example of an interchangeable-lens camera, which is an example of an imaging device according to the first embodiment. The imaging device in FIG. 1 is composed of an interchangeable lens unit 10 and a camera body 20. The camera body 20 functions as an action determination device for determining the action of a subject. A lens control unit 106 that comprehensively controls the operation of the entire lens and a camera control unit 30 that comprehensively controls the operation of the entire camera system including the lens unit 10 can communicate with each other through terminals (not shown) provided on the lens mount.

[0016] First, the configuration of the lens unit 10 will be described. The fixed lens 101, the aperture 102, and the focus lens 103 constitute an imaging optical system. The aperture 102 is driven by an aperture drive unit 104 to control the amount of incident light to an image sensor 201 described later. The focus lens 103 is driven by a focus lens drive unit 105. The focal length of the imaging optical system changes according to the position of the focus lens 103. The aperture drive unit 104 and the focus lens drive unit 105 are controlled by the lens control unit 106 to determine the aperture amount of the aperture 102 and the position of the focus lens 103.

[0017] The lens operation unit 107 is a group of input devices for a user to perform settings related to the operation of the lens unit 10. Examples of settings include switching between AF (auto focus) / MF (manual focus) modes, adjusting the position of the focus lens 103 by MF, setting the operation range of the focus lens 103, setting the shake correction mode, and the like. When the lens operation unit 107 is operated, the lens control unit 106 performs control according to the operation.

[0018] The lens control unit 106 controls the aperture drive unit 104 and the focus lens drive unit 105 according to control commands and control information received from the camera control unit 30 described later. The lens control unit 106 also transmits lens control information to the camera control unit 30.

[0019] Next, the configuration of the camera body 20 will be described. The camera body 20 is configured to generate an imaging signal from the light beam that has passed through the imaging optical system of the lens unit 10. The image sensor 201 is composed of a CCD or CMOS sensor. The light beam incident from the imaging optical system of the lens unit 10 is imaged on the light-receiving surface of the image sensor 201 and converted into a signal charge corresponding to the amount of incident light by photodiodes provided in the pixels arranged on the image sensor 201. The signal charge accumulated in each photodiode is sequentially read out from the image sensor 201 as a voltage signal (imaging signal or AF signal) corresponding to the signal charge. The readout is performed by a drive pulse output by the timing generator 214 according to a command from the camera control unit 30.

[0020] The CDS / AGC / AD converter 202 performs correlated double sampling, gain adjustment, and AD conversion on the imaging signal and AF signal read from the image sensor 201 to remove reset noise. The CDS / AGC / AD converter 202 outputs the processed imaging signal and AF signal to the image input controller 203 and the AF signal processing unit 204, respectively.

[0021] The image input controller 203 stores the imaging signal output from the CDS / AGC / AD converter 202 as an image signal in the SDRAM 209 via the bus 21. The image signal stored in the SDRAM 209 is read out by the display control unit 205 via the bus 21 and displayed on the display unit 206. In recording mode, where the image signal is recorded, the image signal stored in the SDRAM 209 is recorded in a recording medium 208 such as semiconductor memory by the recording medium control unit 207.

[0022] ROM210 stores computer programs (control programs and processing programs) executed by the camera control unit 30, as well as various data necessary for the execution of these computer programs. Flash ROM211 stores various setting information related to the operation of the camera body 20, which has been set by the user.

[0023] The detection unit 31 within the camera control unit 30 performs detection processing to detect body parts of the subject (such as joints and head) in the imaging signal (image) input from the image input controller 203, and stores the results of the detection processing (detection results) in a work memory (not shown). The detection processing can be implemented using any known technology, such as a neural network, decision tree, or other machine learning model. The following description will focus on the case where the subject is a person, but the subject in this embodiment is not limited to a person and may be an animal such as a dog.

[0024] The method selection unit 32 selects one of several determination methods for determining the subject's behavior based on the detection results. Details of the processing of the method selection unit 32 and the determination methods will be described later.

[0025] The determination control unit 33 controls the system to determine the subject's actions according to the determination method selected by the method selection unit 32. The actions to be determined include, for example, suspicious behavior that should be detected by a surveillance camera, or the action of swinging a tennis racket.

[0026] The camera control unit 30 controls each part of the camera body 20 while exchanging information with it. Furthermore, the camera control unit 30 performs various processes corresponding to user operations, such as turning the power on / off, changing various settings, image capture, autofocus, and playback of recorded images, in response to input from the camera operation unit 213 based on user operations. In addition, the camera control unit 30 transmits control commands to the lens unit 10 (lens control unit 106) and information from the camera body 20 to the lens control unit 106, and obtains information from the lens unit 10 from the lens control unit 106. The camera control unit 30 is composed of a microcomputer and controls the entire camera system, including the lens unit 10, by executing computer programs stored in the ROM 210.

[0027] Next, with reference to Figure 2, the processes performed by the camera body 20 will be described. Figure 2 is a flowchart of the process for determining the subject's behavior (behavior determination process). Unless otherwise specified, each step of this flowchart is performed by the camera control unit 30 executing a computer program stored in the ROM 210. The functions of the detection unit 31, method selection unit 32, and determination control unit 33 within the camera control unit 30 are performed by the camera control unit 30 executing a computer program.

[0028] In S201, the detection unit 31 performs detection processing to detect multiple types of body parts of the subject in the image input from the image input controller 203. In the following description, the multiple types of body parts are assumed to be 14 types of body parts, such as the head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles.

[0029] In S202, the detection unit 31 connects the body parts detected as a result of the detection process and identifies one or more subjects.

[0030] Here, we will refer to Figure 3 to explain an example of the process of connecting body parts. Figure 3(a) shows an example of the connection result of the 14 types of body parts explained in S201. In this case, the 14 types of body parts are connected by 13 connection nodes. Below, for the sake of simplicity, we will explain the case of connecting six types of body parts, namely the head, center of gravity, left and right hands, and left and right feet, as shown in Figure 3(b).

[0031] For example, suppose an image like the one shown in Figure 3(c) is input to the detection unit 31. In this case, two heads are detected as shown in Figure 3(d). Similarly, two centers of gravity are detected as shown in Figure 3(e), two right hands are detected as shown in Figure 3(f), two left hands are detected as shown in Figure 3(g), two right feet are detected as shown in Figure 3(h), and two left feet are detected as shown in Figure 3(i).

[0032] As shown in Figure 3(b), five nodes are required to connect body parts: node 301 between the head and the center of gravity, node 302 between the right hand and the center of gravity, node 303 between the left hand and the center of gravity, node 304 between the right foot and the center of gravity, and node 305 between the left foot and the center of gravity. As shown in Figure 3(d), if two heads are detected, it means there are two people in the image, so the detection unit 31 performs connection processing for each person.

[0033] For example, the detection unit 31 selects the head 311 as the first object to be processed. When connecting from the head 311 to the centroid, there are two candidates for the centroid: centroid 312 and centroid 313. The detection unit 31 selects the centroid 312 as the connection destination according to any known algorithm, for example, by selecting the candidate that is closer to the head 311. This establishes a connection between the head 311 and the centroid 312. The detection unit 31 repeats this selection of connection destinations until it finally completes the connection with five nodes as shown in Figure 3(b). After that, the detection unit 31 performs the same process for the other head. By performing this connection process for each subject, the detection unit 31 can identify each subject in the image. Now, let's refer to Figure 2 again. In S203, the method selection unit 32 selects one of several determination methods for determining the subject's behavior based on the results of the detection process by the detection unit 31, and in S204, the determination control unit 33 controls the system to determine the subject's behavior according to the selected determination method. If multiple subjects are identified in S202, the method selection unit 32 selects a determination method for each subject, and the determination control unit controls the system to determine the action for each subject.

[0034] Each of the multiple classification methods is configured to use the positional relationships of two or more body parts from a plurality of types of body parts to determine the subject's behavior. Furthermore, each of the multiple classification methods is composed of any known technique, such as a neural network, decision tree, or other machine learning model, or a simple rule-based algorithm. Different techniques (algorithms) may be used for each classification method. Also, different parameters (e.g., different weights of the neural network) may be used for each of two or more classification methods that use the same technique (algorithm).

[0035] Here, referring to Figure 4, we will explain examples of multiple judgment methods (options for judgment methods) and examples of criteria for selecting a judgment method.

[0036] In Figure 4, determination methods A and B are examples of multiple determination methods. Determination method A is a machine learning model configured to make inferences about the subject's behavior based on the positional relationships of 14 types of body parts: head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. That is, determination method A determines the subject's behavior by inferring the probability of behavior based on the positional relationships of these 14 types of body parts. Determination method B is a machine learning model configured to make inferences about the subject's behavior based on the positional relationships of 10 types of body parts: head, neck, left and right shoulders, left and right elbows, left and right hips, and left and right knees. That is, determination method B determines the subject's behavior by inferring the probability of behavior based on the positional relationships of these 10 types of body parts.

[0037] The behavioral probability referred to here is the probability that the subject is performing the behavior to be judged. For example, when the technology of this embodiment is applied to a surveillance camera, judgment methods A and B infer the probability that the subject is performing suspicious behavior.

[0038] As shown in Figure 4(a), all 14 types of body parts are detected in the subject 411 as a result of the detection process by the detection unit 31. In this case, the method selection unit 32 selects determination method A, which uses the 14 types of body parts. Since the body parts used in determination method A match the body parts that were actually detected, it is expected that the probability of action will be inferred with high accuracy.

[0039] On the other hand, as shown in Figures 4(b) and 4(c), for the subject 412, as a result of the detection process by the detection unit 31, only 10 types of body parts were detected, excluding the left and right wrists and left and right ankles. In this case, if the determination method A, which uses 14 types of body parts as shown in Figure 4(b), is selected, the accuracy of the behavior probability inference may decrease. This is because, since determination method A requires 14 types of body parts, it is necessary to use some kind of initial value (or default value) parameter for the 4 types of body parts that were not detected (left and right wrists and left and right ankles).

[0040] Therefore, as shown in Figure 4(c), for the subject 412, the method selection unit 32 selects determination method B, which uses 10 types of body parts. Since the body parts used in determination method B correspond to the body parts actually detected, it is expected that the inference of behavioral probability will be performed with higher accuracy compared to when determination method A is selected.

[0041] Thus, if a specific type (first type) of body part is not detected as a result of the detection process by the detection unit 31, it is possible to suppress a decrease in the accuracy of the behavior determination by selecting a determination method that does not use the undetected body part to determine the subject's behavior. In other words, it is possible to improve the accuracy of the behavior determination compared to selecting a determination method configured to use the undetected body part.

[0042] Note that the multiple determination methods (selection method options) are not limited to those shown in Figure 4. For example, in addition to determination methods A and B, the multiple determination methods may include determination method X (not shown) which uses 12 types of body parts excluding the left and right ankles. In this case, for subjects (not shown) in which 12 types of body parts excluding the left and right ankles are detected, the method selection unit 32 can select determination method X. If only determination methods A and B exist as selection method options, determination method B will be selected, but in this case, the detected left and right wrists will be wasted. By increasing the variations in the selection method options, the possibility of selecting a determination method that makes efficient use of the actually detected body parts can be improved, thereby improving the determination accuracy.

[0043] Referring again to Figure 2, in S205, the judgment control unit 33 performs processing based on the result of the behavior judgment. For example, the camera control unit 30 can select the subject with the highest probability of performing the behavior to be judged (e.g., suspicious behavior) from among multiple subjects. As another example, the judgment control unit 33 may create a list of subjects from among multiple subjects in which the probability of performing the behavior to be judged (e.g., suspicious behavior) is above a threshold. If the corresponding judgment method differs for each subject and the scale and bias of the behavior probability differ, the camera control unit 30 can perform normalization of the behavior probability so that the behavior probabilities can be directly compared between subjects.

[0044] Next, with reference to Figure 5, other examples of criteria for selecting the determination method in S203 will be described. In the example in Figure 4, the method selection unit 32 selected the determination method based on the body part actually detected as a result of the detection process, but in the example in Figure 5, the method selection unit 32 selected the determination method based on the detection reliability of the body part. Note that the method selection unit 32 may use a combination of the selection criteria in Figure 4 and the selection criteria in Figure 5.

[0045] In Figure 5, column 501 shows the 14 types of body parts that are the target of detection processing by the detection unit 31. Column 502 shows the detection reliability of the body parts detected by the detection unit 31. For example, even if the left ankle is detected as a result of the detection process and its position (coordinates) is identified, that position is not necessarily reliable. Column 502 shows the degree of reliability of the position and other data for each body part using a value from 0 to 1 (1 being the most reliable). Any known technique can be used to determine the detection reliability.

[0046] Column 503 shows the types of body parts used in determination method A, and column 504 shows the types of body parts used in determination method B. In columns 503 and 504, the row for the body part used is marked with "〇". Details of determination methods A and B are explained above with reference to Figure 4.

[0047] If a body part with low detection reliability is used to determine the subject's behavior, the accuracy of the determination may decrease. Therefore, the method selection unit 32 selects a determination method so that body parts with detection reliability below a threshold are not used. This threshold is predetermined according to the precision and recall rates required for the product. In the example in Figure 5, the detection reliability of the right wrist, left ankle, and right ankle is below the threshold (0.5 in the example in Figure 5). Therefore, the method selection unit 32 selects determination method B, which does not use the right wrist, left ankle, and right ankle.

[0048] Thus, if the detection reliability of a specific type (second type) of body part detected as a result of the detection process is lower than a threshold, the method selection unit 32 selects a determination method from among the multiple determination methods that determines the subject's behavior without using the body part with lower detection reliability than the threshold. This makes it possible to suppress a decrease in the accuracy of behavior determination. In other words, it is possible to improve the accuracy of behavior determination compared to selecting a determination method configured to use the body part with lower detection reliability than the threshold.

[0049] Next, with reference to Figure 6, another example of the criteria for selecting the determination method in S203 will be described. In the example in Figure 6, the method selection unit 32 selects a determination method based on the degree of proximity between the subject corresponding to the determination method to be selected (the subject of interest) and another subject (or whether or not another subject exists). The method selection unit 32 may also use one or both of the selection criteria in Figure 4 and the selection criteria in Figure 5 in combination with the selection criteria in Figure 6.

[0050] As shown in Figure 6, consider the case where body parts of subjects 601 and 602 are detected in the image. When selecting a determination method for subject 601, subject 602 is a different subject from subject 601 (the subject of interest).

[0051] When subject 601 (subject of interest) and subject 602 (another subject) are in close proximity, the detection accuracy of certain types of body parts (third type), such as wrists and ankles, may decrease. Furthermore, the connection accuracy of certain types of body parts (third type) (the accuracy of the connection process in S202 in Figure 2) may also decrease. Therefore, if determination method A, which uses wrists and ankles, is selected, the accuracy of action determination may decrease. Thus, when subject of interest and another subject are in close proximity, the method selection unit 32 selects determination method B, which determines the action of subject of interest without using wrists and ankles. Details of determination methods A and B are explained above with reference to Figure 4.

[0052] Figure 6(a) shows an example of the process for determining the degree of proximity between subject 601 and subject 602. First, the method selection unit 32 calculates the average of the distance from the neck to the left hip joint and the distance from the neck to the right hip joint as the length of the torso of subject 601. Next, the method selection unit 32 calculates the distance between the necks of subject 601 and subject 602 (neck-to-neck distance). If the neck-to-neck distance is shorter than the length of the torso of subject 601, the detection accuracy of the wrists and ankles of subject 601 may be reduced. Therefore, if the neck-to-neck distance is shorter than the length of the torso of subject 601, the method selection unit 32 determines that subject 601 and subject 602 are in close proximity and selects determination method B, which does not use the wrists and ankles. Conversely, if the neck-to-neck distance is greater than or equal to the length of the torso of subject 601, the method selection unit 32 determines that subject 601 and subject 602 are not in close proximity and selects determination method A, which uses the wrists and ankles.

[0053] Figure 6(b) shows another example of the process for determining the degree of proximity between subject 601 and subject 602. If the two subjects are adjacent vertically, the accuracy of the body part connection process performed in S202 may decrease (for example, body parts of different subjects may be connected). Therefore, in the example of Figure 6(b), the method selection unit 32 also considers the vertical positional relationship between subject 601 and subject 602. First, as in the case of Figure 6(a), the method selection unit 32 calculates the average value of the distance from the neck to the left hip joint and the distance from the neck to the right hip joint as the torso length of subject 601. Next, the method selection unit 32 calculates the horizontal distance (horizontal neck distance) and the vertical distance (vertical neck distance) between the necks of subject 601 and subject 602. If the horizontal neck distance is shorter than the torso length of subject 601, and the vertical neck distance is shorter than twice the torso length of subject 601, the detection accuracy or connection accuracy of the wrists and ankles of subject 601 may decrease. Therefore, if the horizontal distance between the necks is shorter than the length of the torso of subject 601, and the vertical distance between the necks is shorter than twice the length of the torso of subject 601, the method selection unit 32 determines that subject 601 and subject 602 are in close proximity and selects determination method B, which does not use wrists and ankles. Conversely, if the horizontal distance between the necks is greater than or equal to the length of the torso of subject 601, or if the vertical distance between the necks is greater than or equal to twice the length of the torso of subject 601, the method selection unit 32 determines that subject 601 and subject 602 are not in close proximity and selects determination method A, which uses wrists and ankles.

[0054] Thus, if, as a result of the detection process, a body part of a subject other than the subject of interest is detected, the method selection unit 32 determines whether the subject of interest and the other subject are in close proximity. If the subject of interest and the other subject are in close proximity, the method selection unit 32 selects a determination method from among the multiple determination methods that determines the behavior of the subject of interest without using a specific type (third type) of body part. This makes it possible to suppress a decrease in the accuracy of behavior determination. In other words, it is possible to improve the accuracy of behavior determination compared to selecting a determination method configured to use a specific type (third type) of body part.

[0055] The method for determining whether a subject of interest and another subject are in close proximity is not particularly limited, but for example, as explained above with reference to Figure 6, a method can be used that determines this based on the distance between body parts of the same type, such as the neck of the subject of interest and the neck of the other subject.

[0056] Furthermore, the method selection unit 32 may select a determination method that determines the behavior of the subject of interest without using a specific type (third type) of body part, regardless of the degree of proximity between subjects, if another subject is present (i.e., if a body part of another subject is detected). This makes it possible to reduce the processing load while suppressing a decrease in the accuracy of behavior determination.

[0057] As described above, according to the first embodiment, the camera body 20 performs a detection process to detect multiple types of body parts of a subject in an image, and selects one of multiple determination methods for determining the subject's behavior based on the results of the detection process. Each of the multiple determination methods uses the positional relationship of two or more body parts from the multiple types of body parts to determine the behavior. The camera body 20 is then controlled to determine the subject's behavior according to the selected determination method. Examples of criteria for selecting a determination method based on the results of the detection process are shown in the examples described with reference to Figures 4, 5, and 6. This makes it possible to improve the accuracy of behavior determination.

[0058] Furthermore, depending on the combination of body parts, the angle of a specific body part can be determined from the positional relationship of that combination. For example, the angle of the right knee can be determined from the positional relationship of the right hip joint, right knee, and right ankle. Therefore, in the above explanation, determining the subject's behavior based on the positional relationship of multiple types of body parts essentially includes determining the subject's behavior based on the angle of a specific body part.

[0059] [Second Embodiment] In the second embodiment, a case will be described in which the detection process by the detection unit 31 (Figure 2) is configured to detect a predetermined type or multiple types of objects in addition to multiple types of body parts. In this embodiment, the basic configuration and operation of the lens unit 10 and the camera body 20 are the same as in the first embodiment. The following will mainly describe the differences from the first embodiment.

[0060] Referring to Figure 7, examples of multiple determination methods (options for determination methods) and examples of selection criteria for determination methods according to the second embodiment will be explained.

[0061] In Figure 7, determination methods A, C, and D are examples of multiple determination methods (options for determination methods). Details of determination method A are as described in the first embodiment with reference to Figure 4. Determination method C is a machine learning model configured to make inferences about the subject's behavior based on the positional relationships of 14 types of body parts and a tennis racket. That is, determination method C is configured to determine the subject's behavior in a situation where a tennis racket is present (i.e., a situation in which the sport of tennis is likely to be played). Determination method D is a machine learning model configured to make inferences about the subject's behavior based on the positional relationships of 14 types of body parts and a soccer ball. That is, determination method D is configured to determine the subject's behavior in a situation where a soccer ball is present (i.e., a situation in which the sport of soccer is likely to be played).

[0062] As shown in Figure 7(a), as a result of the detection process by the detection unit 31, 14 types of body parts are detected for the subject 711, but objects such as a tennis racket (a predetermined type of object) are not detected. In this case, the method selection unit 32 selects determination method A, which uses the positional relationships of the 14 types of body parts.

[0063] On the other hand, as shown in Figure 7(b), for the subject 712, as a result of the detection process by the detection unit 31, a tennis racket 713 (a predetermined type of object) is detected in addition to 14 types of body parts. In this case, the method selection unit 32 selects determination method C, which uses the positional relationship between the 14 types of body parts and the tennis racket 713.

[0064] Furthermore, as shown in Figure 7(c), for the subject 714, as a result of the detection process by the detection unit 31, a soccer ball 715 (a predetermined type of object) is detected in addition to 14 types of body parts. In this case, the method selection unit 32 selects determination method D, which uses the positional relationship between the 14 types of body parts and the soccer ball 715.

[0065] When detecting the actions of a subject in sports, the contribution of a predetermined type of object (such as a soccer ball) to the determination may be high depending on the sport. Therefore, if a determination method that uses a predetermined type of object, such as determination method C or D, is selected even though no predetermined type of object is detected, the determination accuracy will decrease. For this reason, if no predetermined type of object is detected, the method selection unit 32 selects a determination method that does not use a predetermined type of object, such as determination method A. This suppresses the decrease in determination accuracy. However, it is not essential that the method selection unit 32 can select a determination method that does not use a predetermined type of object, such as determination method A.

[0066] As described above, according to the second embodiment, the camera body 20 performs a detection process to detect a predetermined type of object in addition to multiple types of body parts, and selects one of multiple determination methods for determining the subject's behavior based on the results of the detection process. Each of the multiple determination methods includes a determination method (such as determination method C or D) that uses the positional relationship between two or more types of body parts from the multiple types of body parts and a predetermined type of object in order to determine the behavior. If a predetermined type of object is detected as a result of the detection process, the camera body 20 selects a determination method that uses the positional relationship between two or more types of body parts from the multiple types of body parts and a predetermined type of object in order to determine the behavior. The camera body 20 then controls itself to determine the subject's behavior according to the selected determination method.

[0067] Thus, in the second embodiment, when a predetermined type of object corresponding to a specific situation (for example, a situation where tennis is being played) is detected, the camera body 20 selects a determination method that uses the predetermined type of object in addition to body parts. This makes it possible to perform behavior determination in a manner suitable for the specific situation, thereby improving the determination accuracy.

[0068] In the above explanation, determination methods C and D are assumed to use two or more types of body parts (14 types in the example of Figure 7) and predetermined types of objects. However, the multiple determination methods (options of determination methods) may include determination methods that use one type of body part or determination methods that use two or more types of objects. In general, each of the multiple methods only needs to be configured to use the positional relationship of at least one type of body part and at least one type of object to determine the action. Furthermore, the detection unit 31 only needs to be configured to perform detection processing to detect one or more types of body parts of the subject and predetermined types of objects (or multiple types of objects) in the image. By adopting such a configuration, for example, if the head of the subject and a soccer ball are detected as a result of the detection processing, it becomes possible to determine the action of heading based on the positional relationship between the head and the soccer ball.

[0069] Alternatively, instead of detecting a predetermined type of object, the camera body 20 may be configured to allow the user to select a sport in advance via a user interface.

[0070] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0071] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0072] 10...Lens unit, 20...Camera body, 30...Camera control unit, 31...Detection unit, 32...Method selection unit, 33...Decision control unit, 201...Image sensor, 209...SDRAM, 210...ROM, 211...Flash ROM

Claims

1. A detection means that performs detection processing to detect multiple types of body parts of a subject in an image, A selection means for selecting one of a plurality of determination methods for determining the behavior of the subject based on the results of the detection process, wherein each of the plurality of determination methods uses the positional relationship of two or more body parts from the plurality of body parts in order to determine the behavior. Control means for determining the behavior of the subject according to the determination method selected by the selection means, An action determination device characterized by comprising the following:

2. If, as a result of the detection process, the first type of body part is not detected, the selection means selects one of the plurality of determination methods that determines the subject's behavior without using the first type of body part. The behavior determination device according to feature 1.

3. If the detection reliability of the second type of body part detected as a result of the detection process is lower than a threshold, the selection means selects a determination method from the plurality of determination methods that determines the subject's behavior without using the second type of body part. The behavior determination device according to claim 1 or 2.

4. If, as a result of the detection process, a body part of a subject other than the subject is detected, the selection means selects a determination method from the plurality of determination methods that determines the subject's behavior without using a third type of body part. The behavior determination device according to any one of claims 1 to 3.

5. The system further includes a determination means for determining whether the subject and the other subject are in close proximity. If it is determined that the subject and the other subject are in close proximity, the selection means selects one of the plurality of determination methods that determines the subject's behavior without using the third type of body part. The behavior determination device according to feature 4.

6. The selection means determines whether the subject and the other subject are in close proximity based on the distance between the same type of body parts of the subject and the other subject. The behavior determination device according to feature 5.

7. At least one of the aforementioned multiple determination methods is a machine learning model configured to perform inferences about the subject's behavior based on the positional relationships of two or more body parts from among the multiple types of body parts. The behavior determination device according to any one of claims 1 to 6.

8. The detection process is configured to detect a predetermined type of object in addition to the multiple types of body parts. The aforementioned plurality of determination methods include a determination method that uses the positional relationship between two or more body parts from the plurality of types of body parts and the predetermined type of object in order to determine the action, If the detection process results in the detection of an object of a predetermined type, the selection means selects the determination method which uses the positional relationship between two or more body parts from the plurality of body parts and the object of the predetermined type in order to determine the action. The behavior determination device according to any one of claims 1 to 7.

9. The detection process is configured to detect a predetermined type of object in addition to the multiple types of body parts. If, as a result of the detection process, an object of a predetermined type is detected, the selection means, instead of selecting one of the multiple determination methods, selects a determination method that uses the positional relationship between one or more body parts from the multiple types of body parts and the object of the predetermined type in order to determine the behavior of the subject. The behavior determination device according to feature 1.

10. A detection means that performs detection processing to detect one or more body parts and multiple types of objects in an image, A selection means for selecting one of a plurality of determination methods for determining the behavior of the subject based on the results of the detection process, wherein each of the plurality of determination methods uses the positional relationship of at least one body part from the one or more types of body parts and at least one object from the plurality of types of objects in order to determine the behavior. Control means for determining the behavior of the subject according to the determination method selected by the selection means, Equipped with, The selection means selects a determination method from among the plurality of determination methods that determines the behavior of the subject using the type of object detected as a result of the detection process. An action determination device characterized by the following features.

11. An action determination device according to any one of claims 1 to 10, The imaging means for generating the aforementioned image, An imaging device characterized by comprising:

12. An action determination method performed by an action determination device, A detection process that performs detection processing to detect multiple types of body parts of a subject in an image, A selection step of selecting one of a plurality of determination methods for determining the behavior of the subject based on the results of the detection process, wherein each of the plurality of determination methods uses the positional relationship of two or more body parts from the plurality of body parts in order to determine the behavior. A control step that controls the system to determine the behavior of the subject according to the determination method selected in the selection step, An action determination method characterized by comprising the following:

13. An action determination method performed by an action determination device, A detection process that performs detection of one or more body parts and multiple types of objects in an image, A selection step of selecting one of a plurality of determination methods for determining the behavior of the subject based on the results of the detection process, wherein each of the plurality of determination methods uses the positional relationship of at least one body part from the one or more types of body parts and at least one object from the plurality of types of objects in order to determine the behavior. A control step that controls the system to determine the behavior of the subject according to the determination method selected in the selection step, Equipped with, The selection step involves selecting a determination method from among the plurality of determination methods that determines the behavior of the subject using the type of object detected as a result of the detection process. A method for determining behavior characterized by the following.

14. A program for causing a computer to function as one of the means of the action determination device described in any one of claims 1 to 10.

Images