Imaging device and its control method, program, and storage medium

Abstract

The present invention provides an imaging device that can improve the quality of images taken of each individual when photographing multiple people. [Solution] The system comprises an imaging unit that captures images of a subject and acquires an image, a person detection unit that detects a person who has been registered in advance from the image, an action detection unit that detects a specific action of a person from the image, and a management unit that manages information of the person captured by the imaging unit. The management unit records the identifier of the person who performed the specific action and the number of times the person performed the specific action when the imaging unit has been instructed to take a picture and the action detection unit has detected that the person detected by the person detection unit has performed a specific action.

Description

Technical Field

[0001] The present invention relates to an imaging device having a subject detection function.

Background Art

[0002] As part of the work of professional photographers, in shooting group sports, there is a need to photograph a plurality of players as the subjects of coverage or all the team members of the subjects of coverage. It is not just about being in the frame; rather, it is desired to capture the players' activities, specific scenes, or poses in sports. Moreover, for each player as the subject of coverage, multiple scenes of their activities are required. The number of shots of the active scenes of the subjects required for such coverage is called "shot volume" in the television camera industry. In the work of sports cameramen, ensuring the shot volume is of utmost importance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Cameramen use a lot of equipment and have to consider a wide variety of things such as securing a good position and following the flow of the game. It is very difficult to remember the shot volume for each of the multiple players. Therefore, some automatic support by a camera is desired.

[0005] In Patent Document 1, the area of an individual's face and the number of appearances are totaled by points in a series of shootings so that each person can be photographed evenly. However, the technique disclosed in Patent Document 1 is effective for events such as outings, but in professional sports photography, it is not just about having the face in the frame. As described above, the shot volume of the active scenes of each player as the subject of coverage is required.

[0006] This invention has been made in view of the above-mentioned problems, and its objective is to provide an imaging device that can improve the quality of images taken of each person when photographing multiple people. [Means for solving the problem]

[0007] The imaging device according to the present invention comprises: an imaging means for imaging a subject and acquiring an image; an individual detection means for detecting a person registered in advance from the image; an action detection means for detecting a specific action of a person from the image; and a management means for managing information of the person imaged by the imaging means, wherein the management means records the identifier of the person who performed the specific action and the number of times the person performed the specific action when the imaging means has been instructed to take a picture and the action detection means has detected that the person detected by the individual detection means has performed the specific action. [Effects of the Invention]

[0008] According to the present invention, it is possible to improve the quality of the images taken of each person when photographing multiple people. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the configuration of a lens-interchangeable digital camera, which is a first embodiment of the imaging device of the present invention. [Figure 2] A flowchart illustrating the operation of the live view process. [Figure 3] A diagram illustrating examples of joint detection and behavior detection. [Figure 4] A flowchart illustrating the process of recording a list of people. [Figure 5] A diagram showing an example of a record of a list of people. [Figure 6] A diagram showing an example of how a list of people is displayed. [Figure 7] A diagram illustrating an example of connection and operation using a network of multiple cameras. [Figure 8] A diagram showing an example of recording a list of people using multiple cameras. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] (First Embodiment) Figure 1 is a block diagram showing the configuration of a lens-interchangeable digital camera 100, which is a first embodiment of the imaging device of the present invention.

[0012] The digital camera 100 of this embodiment consists of an interchangeable lens unit 10 and a camera body 20. The lens control unit 106, which comprehensively controls the operation of the entire lens, and the camera control unit 30, which comprehensively controls the operation of the entire camera system including the lens unit 10, can communicate with each other through terminals provided on the lens mount.

[0013] First, let's explain the configuration of the lens unit 10. The fixed lens 101, aperture 102, and focus lens 103 constitute the imaging optical system. The aperture 102 is driven by the aperture drive unit 104 and controls the amount of incident light on the image sensor 201, which will be described later. The focus lens 103 is driven by the focus lens drive unit 105, and the focus distance of the imaging optical system changes according to the position of the focus lens 103. The aperture drive unit 104 and the focus lens drive unit 105 are controlled by the lens control unit 106, which determines the aperture size of the aperture 102 and the position of the focus lens 103.

[0014] The lens operation unit 107 is a group of input devices for the user to set the operations of the lens unit 10, such as switching between AF (Auto Focus) / MF (Manual Focus) modes, adjusting the position of the focus lens by MF, setting the operating range of the focus lens, and setting the shake correction mode. When the lens operation unit 107 is operated, the lens control unit 106 performs control according to the operation.

[0015] The lens control unit 106 controls the aperture drive unit 104 and the focus lens drive unit 105 according to the control commands and control information received from the camera control unit 30 described later, and also transmits lens control information to the camera control unit 30.

[0016] Next, the configuration of the camera body 20 will be described. The camera body 20 is configured to be able to acquire an image signal from the light beam that has passed through the imaging optical system of the lens unit 10.

[0017] The imaging device 201 is composed of a CCD or a CMOS sensor. The light beam incident from the imaging optical system of the lens unit 10 forms an image on the light receiving surface of the imaging device 201, and is converted into signal charges corresponding to the incident light amount by the photodiodes provided in the pixels arranged in the imaging device 201. The signal charges accumulated in each photodiode are sequentially read out from the imaging device 201 as voltage signals corresponding to the signal charges by the drive pulses output by the timing generator 214 according to the commands of the camera control unit 30.

[0018] The CDS / AGC / AD converter 202 performs correlated double sampling, gain adjustment, and AD conversion to remove reset noise from the image signal read out from the imaging device 201. The CDS / AGC / AD converter 202 outputs the image signal and the AF signal after performing these processes to the image input controller 203 and the AF signal processor 204, respectively.

[0019] The image input controller 203 stores the image signal output from the CDS / AGC / AD converter 202 as an image signal in the SDRAM 209 via the bus 21. The image signal stored in the SDRAM 209 is read by the display control unit 205 via the bus 21 and displayed on the display unit 206. The display unit 206 may be configured by installing a panel-shaped display device such as a liquid crystal or an organic EL in the camera 20, or may be configured by installing a small display device so as to look into the viewfinder.

[0020] Also, in the recording mode for recording the image signal, the image signal stored in the SDRAM 209 is recorded in a recording medium 208 such as a semiconductor memory by the recording medium control unit 207.

[0021] The ROM 210 stores a control program and a processing program executed by the camera control unit 30, and various data necessary for the execution thereof. The flash ROM 211 stores various setting information and the like regarding the operation of the camera body 20 set by the user.

[0022] The network controller 212 is a network controller for wired or wireless communication. The digital camera 100 can be connected to a server through a network. Also, a plurality of digital cameras 100 may be mutually connected.

[0023] The camera control unit 30 includes, as functional modules, a joint detection unit 301, an action detection unit 302, a person detection unit 303, a tracking unit 304, and a person list 305. In this embodiment, these functional modules are realized by the camera control unit 30 executing the control program stored in the ROM 210, but they may be configured by independent circuits.

[0024] The joint detection unit 301 detects and stores each joint, organ, and object worn or held by the person in the image signal input from the image input controller 203, as shown in Figures 3(d) to 3(i) described later. Known methods using neural networks, decision trees, or other machine learning models can be used for detection.

[0025] The action detection unit 302 takes joint information output from the joint detection unit 301 as input and infers whether the subject person is performing a specific action. A specific action is, for example, the action of swinging a racket in tennis filming or the action of shooting in soccer filming. More specifically, it detects a specific action based on the subject person's posture and objects related to the specific action, such as a racket or ball. Furthermore, when filming different sports, the action detection unit 302 may switch to a different algorithm, or it may change the parameters or neural network weights within the same algorithm.

[0026] The personal detection unit 303 first registers one or more people in advance. At that time, it assigns a different person ID (identifier) ​​to each person. Then it determines (authenticates) whether the person in the image signal input from the image input controller 203 is the same as a person that was registered in advance, and if it is the same person, it outputs the coordinates and ID.

[0027] The tracking unit 304 detects the same subject selected in the previous frame from the image signal input from the image input controller 203, and outputs its coordinates and person ID.

[0028] The personal detection unit 303 and the tracking unit 304 are similar, but they differ in that the personal detection unit 303 detects individuals that have been registered in advance, while the tracking unit detects the subject selected in the previous frame.

[0029] The person list 305 records the number of times an action was detected and photographed, coordinates, etc., for each person ID detected by each module 301-304, although the detailed control will be described later.

[0030] The detection methods in each of these modules 301-304 can utilize neural networks, decision trees, other machine learning models, or simple rule-based algorithms.

[0031] The camera control unit 30 controls various parts of the camera body 20 while exchanging information with them. Furthermore, the camera control unit 30 performs various processes corresponding to user operations, such as turning the power on / off, changing various settings, image capture, autofocus, and playback of recorded images, in response to input from the camera operation unit 213 based on user operations. In addition, the camera control unit 30 transmits control commands to the lens unit 10 (lens control unit 106) and information from the camera body 20 to the lens control unit 106, and obtains information from the lens unit 10 from the lens control unit 106. The camera control unit 30 is composed of a microcomputer and controls the entire camera system, including the interchangeable lens 10, by executing computer programs stored in the ROM 210.

[0032] The following describes the processes performed by the camera body 20. The camera control unit 30 performs the following processes according to the imaging processing program, which is a computer program.

[0033] Figure 2 is a flowchart showing the operation procedure of the live view (abbreviated as LV in the diagram, etc.) of the camera body 20. S stands for step.

[0034] The method for recording individuals who performed specific actions in S234-S237 (described later) into the person list 305, and the process for determining the tracked subject, will be described later using Figure 4.

[0035] This section describes the process that occurs on the camera body 20 after live view is started by the camera control unit 213.

[0036] In S231, the camera control unit 30 acquires a live view image.

[0037] In S232, the camera control unit 30 checks whether a shooting operation has been instructed by the camera operation unit 213, such as by pressing the release button. The actual shooting operation is performed in S209.

[0038] In S233, the camera control unit 30 searches the image for the subject that was tracked in the previous frame and obtains its coordinates.

[0039] In S234, the camera control unit 30 detects people in the image and repeats the following processing for each person detected.

[0040] In S235, the camera control unit 30 determines, based on the behavior detection unit 302, whether the person is performing a specific action. If necessary, it records the person in the person list.

[0041] In S236, the camera control unit 30 determines whether the person is the same person as one previously registered by the personal detection unit 303. If necessary, it records the person in the personal list.

[0042] In S237, the camera control unit 30 repeats the process from S234 to S237 for the number of people detected in S234.

[0043] In S238, the camera control unit 30 determines the subject to be tracked in the next live view image based on the processing from S234 to S237.

[0044] In S239, the camera control unit 30 checks whether the user has instructed the camera to take a picture, such as by pressing the shutter release button, in S232. If the camera has been instructed to take a picture, the process proceeds to S240; otherwise, it proceeds to S241.

[0045] In S240, the camera control unit 30 performs the shooting process. In the person list 305, it records the number of actions performed for the ID of the person currently performing a specific action. Details will be explained using Figures 4 and 5.

[0046] In S241, the camera control unit 30 superimposes a frame over the currently tracked subject in the LV image and displays it on the display unit 206.

[0047] In S242, the camera control unit 30 uses the camera operation unit 213 to check whether the user has instructed the system to end the live view. If the system has instructed the system to end the live view, this series of processes is terminated. If the system has not instructed the system to end the live view, it returns to S231 and repeats steps S231 to S242.

[0048] Using Figure 3, we will explain how joints are detected and connected by the joint detection unit 301, and how specific actions are detected by the action detection unit 302.

[0049] In posture estimation, it is common to detect 14 points, such as the head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles, as shown in Figure 3(a). In this case, there are 13 joint connection nodes.

[0050] In this embodiment, to avoid making the illustration too complex and difficult to see, we use six joints as shown in Figure 3(b): the head, center of gravity, left and right hands, and left and right feet. However, in actual implementation, 14 joints or more may be used.

[0051] For example, suppose an image like the one in Figure 3(c) is input to the joint detection unit 301. Then, the head's response will appear as shown in Figure 3(d) 311. Similarly, the center of gravity will appear as shown in Figure 3(e) 312, the right hand as shown in Figure 3(f) 314, the left hand as shown in Figure 3(g) 315, the right foot as shown in Figure 3(h) 316, and the left foot as shown in Figure 3(i) 317.

[0052] As shown in Figure 3(b), five connections are required to connect the joints: 301 between the head and the center of gravity, 302 between the right hand and the center of gravity, 303 between the left hand and the center of gravity, 304 between the right foot and the center of gravity, and 305 between the left foot and the center of gravity.

[0053] For example, if there are two head responses, 311 and 318, as shown in Figure 3(d), then there are two people on the screen, so the loop S204-S207 in Figure 2 will be processed twice.

[0054] In the first loop, when connecting from head 311 to the center of gravity, there are two candidate points, 312 and 313, as shown in Figure 3(e). In this connection, point 312 is selected using some known algorithm, such as selecting the one with the shorter distance. This completes one connection between the head and the center of gravity. Similarly, in the case of 5 joints, this is repeated for 5 connections, and the connections are completed as shown in Figure 3(b).

[0055] Figure 3(j) shows the process leading up to behavior detection.

[0056] In S331, the camera control unit 30 uses the joint detection unit 301 to infer the joint coordinates of each person in the screen in the above manner.

[0057] In S332, the camera control unit 30 inputs the joint coordinates to the action detection unit 302. As mentioned above, the action detection unit 302 is a machine learning model, so it infers whether or not a specific action has occurred and outputs it as a probability.

[0058] In S333, the camera control unit 30 determines that a specific action has occurred if the probability output by the action detection unit 302 exceeds a predetermined threshold, and determines that the specific action has not occurred if it does not exceed the threshold.

[0059] In this embodiment, the threshold value is fixed. The case where it is variable will be described later.

[0060] In this embodiment, a two-class classification was used: either a specific behavior or not. However, the system may be designed to detect multiple behaviors.

[0061] Using Figures 4 and 5, we will explain how to record information in the person list 305 in S234-S237.

[0062] In S401, the camera control unit 30 detects all people in the image, selects one person at a time from among them, and repeats the process from this point for each person.

[0063] In S402, the camera control unit 30 uses the action detection unit 302 to detect the actions of the person selected in S401, and the personal detection unit 303 authenticates whether the person selected in S401 is a person who has been registered in advance.

[0064] In S403, the camera control unit 30 determines whether the person selected in S401 is performing a specific action and whether a shooting instruction has been given in S232. If the person is performing a specific action and a shooting instruction has been given, the tracking unit 304 starts tracking that person and proceeds to S404. Otherwise, proceeds to S409.

[0065] In S404, the camera control unit 30 determines whether the person whose specific behavior was detected in S403 matches a person who was previously registered in S402. If the person is a previously registered individual, the tracking unit 304 continues tracking and proceeds to S405. Otherwise, the unit proceeds to S406.

[0066] In S405, the camera control unit 30 determines that the person whose specific behavior was detected by the behavior detection unit 302 matches the person detected by the personal detection unit 303, and therefore records the person's ID in the person list 305 as shown in Figure 5(a) as 501. Also, since a shooting instruction was given in S232, the camera control unit 30 increments the person's action count (shooting count) by 1 and records it.

[0067] In S406, the camera control unit 30 determines whether tracking is performed by action detection (S403) or by individual detection (S404 or S409) in the previous frame. If neither is the case, proceed to S407; if tracking is performed by individual detection rather than action detection, proceed to S408; otherwise, proceed to S413.

[0068] In S407, the camera control unit 30 records a temporary ID (with the temporary ID set to 0) in the person list 305, as shown in Figure 5(a) at 504, because a specific action was detected in S403 by the action detection unit 302 for an unknown person. Also, since a photograph is taken in response to the shooting instruction in S232, the number of actions (number of photographs) for that person is increased by 1 and recorded.

[0069] In S408, the camera control unit 30 detects a specific action of the person being tracked by the tracking unit 304 using the action detection unit 302, and also records a shooting instruction in S232. Therefore, it increments the action count (shooting count) by 1 in the ID of the person being tracked in the person list 305 and records it.

[0070] For example, if 503 in Figure 5(a) initially had index 3, ID 3, and action count 1, increase the action count to 2 to make it as shown in Figure 5(a).

[0071] In S409, the camera control unit 30 determines whether the person selected in S401 is a person who has been registered in advance. If the person is a person who has been registered in advance, the tracking unit 304 starts tracking and proceeds to S410. Otherwise, it proceeds to S413.

[0072] In S410, the camera control unit 30 determines in the previous frame whether tracking is performed by action detection (S403) or by individual detection (S404 or S409). If neither is the case, proceed to S411; if tracking is performed by action detection but not by individual detection, proceed to S412; otherwise, proceed to S413.

[0073] In S411, the camera control unit 30 records the ID of the person selected in S401 and detected by the personal detection unit 303 in S409 in the person list 305 as shown in 502 of Figure 5(a). Since no action has been detected yet, the action count is recorded without increasing.

[0074] In S412, if the person being tracked by the tracking unit 304 is detected by the personal detection unit 303 as having a person ID, and that ID is not in the person list, the camera control unit 30 does the following: For example, if the person with the temporary ID currently being tracked in Figure 5(a) 504 is found to have ID 4, the ID is updated to 4 and recorded as shown in Figure 5(b) 505.

[0075] Furthermore, if that ID is found in the person list, proceed as follows: For example, if the person with the currently tracked temporary ID, 504 in Figure 5(a), is found to be ID3, delete index3 (503) and index4 (504), and update and record the combined index3, ID3, and action count 3=(2+1) as shown in 506 in Figure 5(c).

[0076] In S413, the camera control unit 30 loops through steps S401 to S413 for each person in the image.

[0077] As described above, if individual detection and behavior detection occur simultaneously, this will be noted. Furthermore, if behavior detection occurs in the next frame or later after individual detection, the behavior count of the person with the detected ID will be increased. Also, if the person being tracked takes a specific action after behavior detection, the temporary ID will be changed to the official ID, or merged with an existing ID.

[0078] Next, using Figure 6, we will explain an example of how the person list 305 will be displayed on the display unit 206.

[0079] The number of actions taken for each person ID recorded in person list 305 represents the number of images taken of each person performing a specific action, as explained above. By showing (notifying) the photographer (user) of this number of images taken in real time, it is possible to manage the amount of footage taken and obtain guidance on what subjects should be photographed.

[0080] Figure 6(a) shows an example of the image quality display. The current image quality is displayed for all individuals who have been previously registered in the personal detection unit 303, as indicated by code 601. In this case, it can be seen that player B has a low image quality.

[0081] In the example shown in Figure 6(d), by displaying (highlighting) a frame (604,605) around the registered person in the image and directly writing the amount of footage taken within that frame, it is possible to support professional photographers' shooting in a more intuitive and easy-to-understand manner.

[0082] As described above, according to this embodiment, it is possible to shoot while viewing the amount of footage taken of each person on the display member, manage the amount of footage taken, and obtain guidelines for shooting.

[0083] (Second embodiment) In this second embodiment, the same configuration and control as in the first embodiment will not be described, and the differences will be explained.

[0084] Here are two examples of how to define a person who doesn't yield many good photos.

[0085] For example, in Figure 5(b), there are four pre-registered individuals, IDs 1 through 4. If we take the average of their action counts, we get (1+0+2+1)÷4=1. This means that the individual with fewer action counts than average is ID 2, and we can see that we need to capture more footage of this individual.

[0086] Alternatively, it is possible to set the number of photos to be taken per person on camera 20 in advance using the camera control unit 213. If it is set to one photo in advance, and the current situation is as shown in Figure 5(b), then it is clear that ID2 should be prioritized for shooting.

[0087] Figure 6 shows an example of the display on the display unit 206 at this time.

[0088] Figure 6(b) shows an example of the photo yield display. As indicated by symbol 602, the current photo yield is displayed for all individuals who have been previously registered in the personal detection unit 303. In this case, since athlete B, whose score is below average, should be given higher priority for photography, he is displayed higher up to make it easier to see.

[0089] Figure 6(c) shows an example of a photo yield display. As indicated by symbol 603, only player B, whose performance is below average, is displayed, making it easier to see which players should be prioritized for photography.

[0090] Figure 6(e) shows an example of a display showing the amount of good shots taken. As shown by symbol 605, a person frame is superimposed on player B, who is below average, and the amount of good shots taken is also displayed there if necessary. This makes it easier to see which players should be prioritized for shooting.

[0091] As mentioned earlier in Figure 3(j), a threshold is set for the action probability output in S332, and the action is determined. At this time, it is possible to lower the threshold for action determination for people with few recorded images, or to raise the inference probability.

[0092] By doing this, the S403 becomes more likely to detect the subject's actions and track it. If a photo is taken while tracking, the amount of footage will increase even further.

[0093] Conversely, if the amount of footage obtained from a person whose behavior is being judged is sufficient, and there are other people whose footage is below average, methods such as raising the threshold or discounting the inference probability may be used.

[0094] As described above, according to the second embodiment, it is possible to photograph people who are not producing good footage while checking them on the display component, and thus obtain guidelines for shooting. In addition, it becomes easier to detect the actions of people who are not producing good footage, resulting in improved footage compared to normal shooting.

[0095] (Third embodiment) In this third embodiment, the same configuration and control as in the first embodiment will not be described, and the differences will be explained.

[0096] Figure 7(a) shows multiple digital cameras 100 connected to a server via a network. Instead of each digital camera 100 having its own person list 305, one list is stored on the server 701. In this case, the person list 305 will contain the time code (time information) for each individual action, as shown in Figure 8(a), rather than the number of actions. Therefore, to find out the number of actions, one would count the number of time codes listed.

[0097] Using Figures 4 and 8, we will explain the method of recording to the person list 305 in steps S234 to S237 of Figure 2, specifically for cases where multiple cameras are used.

[0098] Since the flowchart itself remains unchanged, we will use the same Figure 4. We will also assume that timecodes are shared among multiple cameras.

[0099] In Figure 4, at S401, the camera control unit 30 detects all the people in the image, selects one person at a time from among them, and repeats the process from this point for each person.

[0100] In S402, the camera control unit 30 uses the action detection unit 302 to detect the actions of the person selected in S401, and the personal detection unit 303 authenticates whether the person selected in S401 is a person who has been registered in advance.

[0101] In S403, the camera control unit 30 determines whether the person selected in S401 is performing a specific action and whether a shooting instruction has been given in S232. If the person is performing a specific action and a shooting instruction has been given, the tracking unit 304 starts tracking that person and proceeds to S404. Otherwise, proceeds to S409.

[0102] In S404, the camera control unit 30 determines whether the person whose specific behavior was detected in S403 matches a person who was previously registered in S402. If the person is a previously registered individual, the tracking unit 304 continues tracking and proceeds to S405. Otherwise, the unit proceeds to S406.

[0103] In S405, the camera control unit 30 determines that the person whose specific behavior was detected by the behavior detection unit 302 matches the person detected by the personal detection unit 303, and therefore records the person's ID in the person list 305 as shown at 801 in Figure 8(a). Specifically, it records the person's ID and the time code in which the behavior was detected.

[0104] In S406, the camera control unit 30 determines whether tracking is performed by action detection (S403) or by individual detection (S404 or S409) in the previous frame. If neither is the case, proceed to S407; if tracking is performed by individual detection rather than action detection, proceed to S408; otherwise, proceed to S413.

[0105] In S407, the camera control unit 30 records a temporary ID (with the temporary ID set to 0) and the current time code in the person list 305, as shown in Figure 8(a) at 805, because the action detection unit 302 detected a specific action in an unknown person in S403.

[0106] In S408, the camera control unit 30 detects a specific action of the person being tracked by the tracking unit 304 using the action detection unit 302, and also records a shooting instruction in S232. Therefore, it increments the action count (shooting count) by 1 in the ID of the person being tracked in the person list 305 and records it.

[0107] For example, if 803 in Figure 8(a) initially had index 3, ID 3, and had performed one action in the past, and therefore had one time code recorded, then the current time code would be added and recorded as 804.

[0108] In S409, the camera control unit 30 determines whether the person selected in S401 is a person who has been registered in advance. If the person is a person who has been registered in advance, the tracking unit 304 starts tracking and proceeds to S410. Otherwise, it proceeds to S413.

[0109] In S410, the camera control unit 30 determines in the previous frame whether tracking is performed by action detection (S403) or by individual detection (S404 or S409). If neither is the case, proceed to S411; if tracking is performed by action detection but not by individual detection, proceed to S412; otherwise, proceed to S413.

[0110] In S411, the camera control unit 30 records the ID of the person selected in S401 and detected by the personal detection unit 303 in S409 in the person list 305 as shown in Figure 8(a) as 802. Since no action has been detected yet, the time code is left blank.

[0111] In S412, if the person being tracked by the tracking unit 304 is detected by the personal detection unit 303 as having a person ID, and that ID is not in the person list, the camera control unit 30 does the following: For example, if the person with the temporary ID currently being tracked, 805 in Figure 8(a), is found to have ID 4, then the camera control unit 30 records 4 as the ID, as shown in 806 in Figure 8(b).

[0112] Furthermore, if that ID is found in the person list, proceed as follows: For example, if the person with the currently tracked temporary ID 803 in Figure 8(a) is found to be ID3, then in addition to index3, ID3, and the existing time codes 807 and 808, time code 809 is recorded as shown in Figure 8(c).

[0113] As described above, if individual detection and behavior detection occur simultaneously, this will be noted. Furthermore, if behavior detection occurs in the next frame or later after individual detection, the behavior count of the person with the detected ID will be increased. Also, if the person being tracked takes a specific action after behavior detection, the temporary ID will be changed to the official ID, or merged with an existing ID.

[0114] Figures 7(b) to 7(f) illustrate an example of shooting with multiple cameras.

[0115] Subject 702 is playing tennis, and one of the actions to be detected is "swinging the racket."

[0116] Figure 7(b) shows the positional relationship between the cameras and the subject, assuming that the subject 702 is being photographed by two cameras, A and B. Camera A is oriented 703, which is in front of the subject 702. Camera B is oriented 704, which is behind the subject 702.

[0117] Figure 7(c) shows an example of subject 702's play, where subject 702 faces the designated camera and takes action at each timecode.

[0118] Figures 7(d) to 7(f) show the player list 305 shared on server 701, and are an example of the gameplay shown in Figure 7(c).

[0119] <When the timecode in Figure 7(c) is 00:10:00> From camera A's perspective, subject 702 is standing still but its face is visible, so subject 702 is detected as player ID1 through individual detection. Since it is standing still, there is no action detection, and the person list 305 is recorded as shown in Figure 7(d), and tracking begins. This corresponds to processing S411. From camera B's perspective, subject 702 is standing still with its back to the camera, so neither individual detection nor action detection reacts.

[0120] <When the timecode in Figure 7(c) is 00:20:00> From camera A's perspective, the tracked subject 702 is swinging a racket, so the action of player ID 1 is detected, and the timecode is recorded in index 1 of the person list 305 as shown in Figure 7(e). This corresponds to processing S408. From camera B's perspective, subject 702 is seen from behind but is swinging a racket, so the action is detected, and the system attempts to perform processing S407. However, as shown in Figure 7(e), the action with the same timecode has already been recorded, so the entry into the person list 305 in processing S407 is canceled.

[0121] <When the timecode in Figure 7(c) is 00:30:00> From camera A's perspective, the tracked subject 702 is facing away and standing still, so the tracking of player ID1 continues. From camera B's perspective, subject 702 is facing forward and standing still, so the personal detection unit 303 detects it as player ID1. Since it is standing still, no action is detected, and an attempt is made to add it to the person list 305, but since ID1 already exists, the addition is canceled. This proceeds to process S411.

[0122] <When the timecode in Figure 7(c) is 00:40:00> From camera A's perspective, subject 702 is seen from behind, swinging a racket. If tracking were successful, it would be identifiable as subject ID1, but to increase the variations in the example, let's assume that tracking failed, for example, by moving away from the camera. In this case, no processing of person list 305 occurs in camera A.

[0123] From camera B's perspective, subject 702 is swinging a racket in front of it. It is already being tracked as ID1, and its action is detected, with the timecode recorded in index1, ID1 as shown in Figure 7(f). Since the timecode is different from the previous action at 00:20:00, it is recorded as the second action.

[0124] This allows the person list 305 to be shared across multiple cameras. Even if multiple cameras detect actions simultaneously, the timecode can be used to avoid duplication. Furthermore, even if one camera loses sight of a subject, other cameras can detect them and count the actions of the person with the same ID. This also allows for managing the amount of footage captured when a single match is filmed by a team of multiple cameras and cameramen.

[0125] As described above, according to the third embodiment, the effects of the first and second embodiments can be shared not only with one camera but with multiple cameras, enabling team-based management of footage and improvement of the amount of footage captured.

[0126] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0127] The disclosures herein include the following imaging devices and their control methods, programs, and storage media.

[0128] (Item 1) An imaging means for capturing an image of a subject and acquiring an image, A means for detecting a person who has been registered in advance from the aforementioned image, Action detection means for detecting specific actions of a person from the aforementioned image, The system comprises a management means for managing information about a person captured by the imaging means, The management means is characterized in that, when the imaging means is instructed to take a photograph, and the action detection means detects that a person detected by the personal detection means has taken the specific action, it records the identifier of the person who took the specific action and the number of times the person has taken the specific action.

[0129] (Item 2) The imaging device according to item 1, wherein the aforementioned image is a live view image, and further comprises a tracking means for detecting a person determined in the previous frame in subsequent frames.

[0130] (Item 3) The imaging device according to item 1 or 2, characterized in that the management means records a person as a temporary person if the action detection means detects a specific action, even if the personal detection means has not authenticated the person in the image as the registered person, and updates the record as the registered person if the person is authenticated as the registered person in a subsequent frame.

[0131] (Item 4) The imaging device according to any one of items 1 to 3, characterized in that the action detection means detects the specific action based on the posture of a person and an object associated with the specific action.

[0132] (Item 5) The imaging device according to any one of items 1 to 4, characterized in that the management means differs in the method of recording the information to be managed depending on whether the authentication of a person and the detection of the specific action are performed in the same frame, whether the specific action is detected first and the authentication of a person is performed in a subsequent frame, or whether the authentication of a person is performed first and the specific action is detected in a subsequent frame.

[0133] (Item 6) The imaging device according to any one of items 1 to 5, characterized in that the management means records the number of times a person has performed the specific action when the person is recorded as an image.

[0134] (Item 7) The imaging device according to any one of items 1 to 6, characterized in that the management means makes it easier to determine that a person who has been recorded as having performed the specific action a few times is performing the specific action.

[0135] (Item 8) The imaging device according to item 7, characterized in that the number of recorded instances of the aforementioned specific action being performed is less than the number of recorded instances set in advance.

[0136] (Item 9) The imaging device according to item 7, characterized in that the number of times the aforementioned specific action has been recorded is small, which means that the number of times the aforementioned specific action has been recorded is less than the average number of times the aforementioned specific action has been recorded for the aforementioned person who was registered in advance.

[0137] (Item 10) The imaging device according to any one of items 1 to 9, further comprising a notification means for notifying the user of the number of times each person has performed the aforementioned specific action.

[0138] (Item 11) The imaging device according to item 10, characterized in that the notification means is a display means.

[0139] (Item 12) The imaging device according to item 11, characterized in that the display means displays the number of times that the specific action was performed for all of the persons who were registered in advance.

[0140] (Item 13) The imaging device according to item 11, characterized in that the display means displays individuals who have performed the specific action fewer times at the top.

[0141] (Item 14) The imaging device according to item 11, characterized in that the display means highlights and displays individuals who have performed the specific action a small number of times.

[0142] (Item 15) The imaging device according to item 14, characterized in that the display means displays a frame around a person who has performed the specific action a few times.

[0143] (Item 16) The imaging device according to item 11, characterized in that the display means displays a frame around each person and then displays the number of times each person performed the specific action.

[0144] (Item 17) An imaging device according to any one of items 1 to 16, characterized in that the information of the management means is shared among multiple imaging devices.

[0145] (Item 18) The imaging device according to item 7, characterized in that the management means records information about the time when a person performed the specific action when the plurality of imaging devices share information.

[0146] (Item 19) A method for controlling an imaging device equipped with imaging means for capturing an image of a subject and acquiring an image, A personal detection step for detecting a person who has been registered in advance from the aforementioned image, A behavior detection step for detecting a specific action of a person from the aforementioned image, The system includes a management process for managing information about a person captured by the imaging means, A control method for an imaging device, characterized in that, in the management step, a shooting instruction is given to the imaging means, and when it is detected in the action detection step that a person detected in the personal detection step has performed the specific action, the identifier of the person who performed the specific action and the number of times the person performed the specific action are recorded.

[0147] (Item 20) A program to cause a computer to execute each step of the control method for the imaging device described in item 19.

[0148] (Item 21) A computer-readable storage medium containing a program that causes a computer to execute each step of the control method for the imaging device described in item 19.

[0149] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0150] 10: Lens unit, 20: Camera body, 30: Camera control unit, 100: Digital camera, 102: Aperture, 103: Focus lens, 106: Lens control unit, 201: Image sensor, 213: Camera control unit

Claims

1. An imaging means for capturing an image of a subject and acquiring an image, A means for detecting a person who has been registered in advance from the aforementioned image, Action detection means for detecting specific actions of a person from the aforementioned image, The system comprises a management means for managing information about a person captured by the imaging means, The management means is characterized in that, when the imaging means is instructed to take a photograph, and the action detection means detects that a person detected by the personal detection means has taken the specific action, it records the identifier of the person who took the specific action and the number of times the person has taken the specific action.

2. The image is a live view image, and the imaging device according to claim 1 is further characterized by comprising a tracking means for detecting a person determined in the previous frame in subsequent frames.

3. The imaging device according to claim 1, characterized in that the management means records a person as a temporary person if the action detection means detects a specific action, even if the personal detection means has not authenticated the person in the image as the registered person, and updates the record as the registered person if the person is authenticated as the registered person in a subsequent frame.

4. The imaging device according to claim 1, characterized in that the action detection means detects the specific action based on the posture of the person and an object associated with the specific action.

5. The imaging device according to claim 1, characterized in that the management means differs in the method of recording the information to be managed depending on whether the authentication of a person and the detection of the specific action are performed in the same frame, whether the specific action is detected first and the authentication of a person is performed in a subsequent frame, or whether the authentication of a person is performed first and the specific action is detected in a subsequent frame.

6. The imaging device according to claim 1, characterized in that the management means records the number of times a person has performed the specific action when the person is recorded as an image.

7. The imaging device according to claim 1, characterized in that the management means makes it easier to determine that a person who has been recorded as having performed the specific action a small number of times is performing the specific action.

8. The imaging device according to claim 7, characterized in that the number of recorded instances of the aforementioned specific action being performed is less than the number of recorded instances set in advance.

9. The imaging device according to claim 7, characterized in that the number of times the aforementioned specific action has been recorded is small, which means that it is less than the average number of times the aforementioned specific action has been recorded for the previously registered person.

10. The imaging device according to claim 1, further comprising a notification means for notifying the user of the number of times each person has performed the aforementioned specific action.

11. The imaging apparatus according to claim 10, characterized in that the notification means is a display means.

12. The imaging device according to claim 11, characterized in that the display means displays the number of times the specific action was performed for all of the persons who were registered in advance.

13. The imaging device according to claim 11, characterized in that the display means displays individuals who have performed the specific action fewer times in a higher position.

14. The imaging device according to claim 11, characterized in that the display means highlights and displays individuals who have performed the specific action a small number of times.

15. The imaging device according to claim 14, characterized in that the display means displays a frame around a person who has performed the specific action a small number of times.

16. The imaging device according to claim 11, characterized in that the display means displays a frame around each person and then displays the number of times each person performed the specific action.

17. The imaging device according to claim 1, characterized in that the information of the management means is shared among multiple imaging devices.

18. The imaging device according to claim 17, characterized in that the management means records information about the time when a person performed the specific action when the plurality of imaging devices share information.

19. A method for controlling an imaging device equipped with imaging means for capturing an image of a subject and acquiring an image, A personal detection step for detecting a person who has been registered in advance from the aforementioned image, A behavior detection step for detecting a specific action of a person from the aforementioned image, The system includes a management process for managing information about a person captured by the imaging means, A control method for an imaging device, characterized in that, in the management step, a shooting instruction is given to the imaging means, and when it is detected in the action detection step that a person detected in the personal detection step has performed the specific action, the identifier of the person who performed the specific action and the number of times the person performed the specific action are recorded.

20. A program for causing a computer to execute each step of the control method for the imaging device described in claim 19.

21. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method for the imaging device described in claim 19.

Images