Control device and control method

Abstract

To provide an imaging control device that can track a subject with high moving speed with high accuracy while suppressing unnecessary PTZ control.SOLUTION: The imaging control device is designed to control the imaging direction of an imaging apparatus that captures an image of a subject. The imaging control device estimates the posture of the subject detected from the image captured by the imaging apparatus. The imaging control device sets a first region to the imaging range of the imaging apparatus on the basis of whether the estimated posture of the subject is a predetermined posture. The imaging control device controls the imaging direction on the basis of whether the subject is in the first region.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging control device, an imaging device, an imaging control method, and a program. [Background technology]

[0002] Conventionally, there is a technology that controls the pan-tilt-zoom (hereinafter referred to as PTZ) of an imaging device according to the movement of a subject to be tracked, and controls the imaging direction of the imaging device so that the subject is captured within the imaging range of the imaging device. Also, when automatically tracking a subject using an imaging device, there is a technology that sets a dead zone within the imaging range of the imaging device so that the imaging control device does not perform unnecessary PTZ control on the imaging device. If the subject is captured within the dead zone, the imaging control device will not perform unnecessary PTZ control on the imaging device. However, if a subject captured within the dead zone moves out of the dead zone at high speed, the imaging device may be slow to track the subject.

[0003] In consideration of the above problems, Patent Document 1 proposes a technique for tracking a fast-moving subject by not setting a dead zone in the imaging range of an imaging device when the subject is moving fast. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-171438 Summary of the Invention [Problem to be solved by the invention]

[0005] However, Patent Document 1 requires time to calculate the moving speed of the subject to be tracked, and therefore has a problem in that the subject cannot be tracked because the subject moves out of the imaging range while the moving speed of the fast-moving subject is being calculated.

[0006] Therefore, an object of the present invention is to provide an imaging control device that can track a fast-moving subject with high accuracy while suppressing unnecessary PTZ control. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, an imaging control device according to one embodiment of the present invention has the following configuration: An imaging control device that controls an imaging direction of an imaging device that images a subject, and includes: estimation means that estimates an orientation of the subject detected from an image captured by the imaging device; area control means that sets a first area in an imaging range of the imaging device based on whether the estimated orientation of the subject is a predetermined orientation; and imaging control means that controls the imaging direction based on whether the subject is present in the first area. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an imaging control device that can track a fast-moving subject with high accuracy while suppressing unnecessary PTZ control. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview of an imaging system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing the hardware configuration of an imaging control device according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the functional configuration of the imaging control device according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a result of estimating the posture of a subject. [Figure 5] FIG. 10 is a diagram showing an example of a preparatory movement of a subject. [Figure 6] FIG. 10 is a diagram showing an example of a preparatory movement of a subject. [Figure 7] FIG. 4 is a diagram showing an example of a dead zone; [Figure 8] FIG. 10 is a diagram illustrating an example of changing a dead zone area. [Figure 9] FIG. 10 is a diagram illustrating an example of changing a dead zone area. [Figure 10] A diagram showing a subject outside the dead zone. [Figure 11] 4 is a flowchart showing processing of the imaging control device according to the first embodiment. [Figure 12] FIG. 10 is a block diagram showing the functional configuration of an imaging control device according to a second embodiment. [Figure 13] 10 is a flowchart showing processing of an imaging control device according to a second embodiment. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of an imaging control device according to a third embodiment. [Figure 15] 10A and 10B are diagrams illustrating a method for determining the movement of a subject during a preparatory movement. [Figure 16] 10A and 10B are diagrams illustrating a method for determining the movement of a subject during a preparatory movement. [Figure 17] 10 is a flowchart showing processing of an imaging control device according to a third embodiment. [Figure 18] 10A and 10B are diagrams illustrating imaging control performed by the imaging device after a subject makes a preparatory movement. [Figure 19] FIG. 10 is a diagram illustrating how a preparatory movement is determined from information other than the subject's joints. [Figure 20] FIG. 10 is a diagram illustrating an example of a method for calculating the center of gravity of a subject. [Figure 21] 10A and 10B are diagrams illustrating changes in the center position of a subject according to the movement of the subject in the conventional method and the present invention. [Figure 22] 13 is a flowchart illustrating processing of an automatic photography system according to a fifth embodiment. [Figure 23] 10A and 10B are diagrams illustrating a method for determining whether the subject's left and right feet have been swapped. [Figure 24] FIG. 3 is a schematic diagram illustrating an inference model of the operation determination unit 309. [Figure 25] FIG. 10 is a diagram showing an example of training data for training an inference model. [Figure 26] 13 is a flowchart illustrating processing of an automatic photography system according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (First embodiment) FIG. 1 is a diagram showing an overview of an imaging system according to the first embodiment.

[0012] The imaging system 10 is a system for capturing an image of a subject. The imaging system 10 includes an imaging device 101, an input device 102, a driving device 103, an imaging control device 104, a display device 115, and a network 150. The imaging device 101, the input device 102, the driving device 103, and the imaging control device 104 are connected via the network 150. The imaging control device 104 and the display device 115 are also connected via a video interface.

[0013] The imaging system 10 tracks a subject based on an image acquired from the imaging device 101 and a tracking setting for the subject acquired from the input device 102. The imaging control device 104 controls the imaging direction of the imaging device 101 via the drive device 103 so that the subject to be tracked is captured within the imaging range of the imaging device 101. At this time, the imaging control device 104 changes the tracking setting according to the movement of the subject to be tracked, thereby enabling tracking and capturing an image of the subject even when the subject is moving at high speed. The imaging system 10 then displays the image captured by the imaging device 101 on the display device 115.

[0014] The imaging device 101 is a device that captures an image of a subject while changing the imaging direction, and is, for example, a PTZ camera or a network camera. The imaging device 101 outputs the captured image to the imaging control device 104. Note that although there is one imaging device 101 in this embodiment, there may be two or more imaging devices.

[0015] The input device 102 is a device that accepts object tracking settings from a user, and includes, for example, a GUI (Graphical User Interface) that operates on a web browser, a mouse, a keyboard, and buttons. The input device 102 outputs the object tracking settings input by the user to a region setting unit 310, which will be described later. The tracking settings refer to setting a dead zone region in the imaging range of the imaging device 101.

[0016] The driving device 103 is a device that controls the imaging direction of the imaging device 101, and includes, for example, gears and motors for performing PTZ control. The driving device 103 drives the PTZ of the imaging device 101 based on a PTZ control value received from a control unit 313 (described later).

[0017] The imaging control device 104 is a device that controls the imaging direction of the imaging device 101, and includes, for example, a PC, a smartphone, and a tablet. In this embodiment, the imaging device 101 and the imaging control device 104 exist independently, but the imaging control device 104 may be installed in the imaging device 101. The imaging control device 104 performs object tracking processing based on the position (e.g., two-dimensional coordinates) of the object detected from each still image that constitutes the video and the object tracking setting. Here, the imaging control device 104 determines the movement (posture) of the object based on the result of estimating the posture of the object. If the imaging control device 104 determines that the object is moving rapidly, it changes the setting of the dead zone area already set in the imaging range. This allows the imaging control device 104 to capture and track the object without losing it, even if the object moves rapidly.

[0018] FIG. 2 is a diagram showing the hardware configuration of the imaging control device according to the first embodiment.

[0019] The imaging control device 104 includes a CPU 200 , a RAM 201 , a ROM 202 , a storage unit 203 , and an I / F 204 .

[0020] The CPU 200 is a central processing unit that controls each device of the imaging system 10 .

[0021] The RAM 201 is a memory for temporarily storing computer programs executed by the CPU 200, and is, for example, a volatile memory such as an SRAM or a DRAM.

[0022] The ROM 202 is a memory that stores programs for the CPU 200 to control each device of the image capture system 10, and is, for example, a non-volatile memory such as an EPROM.

[0023] The storage unit 203 is a device that stores programs and video data, and includes, for example, a hard disk drive (HDD) and a solid state drive (SSD).

[0024] The I / F 204 is a device for transmitting and receiving data to and from each device of the imaging system 10. The I / F 204 also transmits and receives data to and from an external device (not shown) via the network 150.

[0025] 3 is a diagram showing an example of the functional configuration of the imaging control device according to the first embodiment. The functions of the imaging control device 104 described below are realized by the CPU 200 executing a program stored in the ROM 202 or the like.

[0026] The imaging control device 104 includes an acquisition unit 305, a detection unit 306, a tracking unit 307, a posture estimation unit 308, a motion determination unit 309, a region setting unit 310, a recording unit 311, an operation determination unit 312, a control unit 313, and an output unit 314.

[0027] The acquisition unit 305 acquires an image from the image capturing device 101 and outputs the acquired image to the detection unit 306 and the output unit 314 .

[0028] The detection unit 306 detects a subject from each image constituting the video based on a predetermined subject detection method. The subject detection method is, but is not limited to, a template matching method and a semantic region segmentation method. The template matching method and the semantic region segmentation method are well-known techniques, so a description of these methods will be omitted. The detection unit 306 outputs the position (e.g., two-dimensional coordinates) of the subject detected from the image to the tracking unit 307.

[0029] The tracking unit 307 tracks a subject based on the position of the subject detected from the image. If the tracking unit 307 is not tracking a subject, it selects a subject to track from among the subjects detected from the image and starts tracking the selected subject. On the other hand, if the tracking unit 307 is already tracking a subject, it continues tracking the subject based on the position of the subject. The method of selecting the subject to track may be any method that can select one subject from multiple subjects detected from the image. For example, the tracking unit 307 selects the subject closest to the center of the image as the subject to track.

[0030] The tracking unit 307 determines the position of the subject in the image at the current time based on the position of the subject in the image at an earlier time and the position of the subject detected by the detection unit 306. For example, the tracking unit 307 may determine the position of the subject to be tracked by associating the position predicted from the movement history of the subject to be tracked with the detected position of the subject. The tracking unit 307 outputs the result of the subject tracking process to the posture estimation unit 308 and the operation determination unit 312.

[0031] The posture estimation unit 308 estimates the posture of the subject based on the position (coordinates) of the subject received from the tracking unit 307. The posture estimation unit 308 can estimate the posture of the subject with high accuracy using a posture estimation technique based on Deep Learning. There are also posture estimation techniques provided by OSS (Open Source Software), such as OpenPose and DeepPose. In this embodiment, the posture of the subject is estimated using one of the posture estimation techniques based on Deep Learning. The posture estimation unit 308 extracts the position (coordinates) of the subject detected from the image, and estimates the posture of the subject (joint points, etc.) using the posture estimation technique for the extracted position (coordinates). The posture of the subject refers to connection information between joint points (so-called human skeletal information) obtained based on the detected positions of the joint points of the subject (see FIG. 4). The posture estimation unit 308 outputs the posture estimation result of the subject to the movement determination unit 309.

[0032] The movement determination unit 309 analyzes the tilt and angle of specific parts in the posture of the subject based on the posture estimation result of the subject by the posture estimation unit 308. Based on the analysis result of the posture of the subject, the movement determination unit 309 determines whether the subject has made a preparatory movement (hereinafter referred to as a preparatory movement) that is taken before making a rapid movement. In this embodiment, the movement determination unit 309 determines the preparatory movement of the subject based on a rule base, but is not limited to this. For example, the movement determination unit 309 may determine the preparatory movement of the subject using a learning model (deep learning) that has learned video images recording the preparatory movements of the subject as learning data. Preparatory movements include, for example, a posture in which the height from the ground to the center of gravity of the subject is lower than a threshold and a posture in which an axis passing through the center of the subject is inclined with respect to the ground. However, the posture is not limited to the above postures as long as it is a posture that the subject can take before starting to move rapidly.

[0033] FIG. 4 is a diagram showing an example of the result of estimating the posture of the subject.

[0034] Image 400 shows a subject 401. Subject 401 has a neck joint 402, a left elbow joint 403, a right elbow joint 404, a left wrist joint 405, a right wrist joint 406, a waist joint 407, a left knee joint 408, a right knee joint 409, a left ankle joint 410, and a right ankle joint 411. Note that "left" or "right" placed before each joint indicates the direction when subject 401 is viewed from the front.

[0035] Angle 412 represents the angle of the left arm. Angle 412 is the angle between an axis connecting the neck joint 402 and the left elbow joint 403 and an axis connecting the left elbow joint 403 and the left wrist joint 405, and is 180° in FIG. 4. Angle 413 represents the angle of the right arm. Angle 413 is the angle between an axis connecting the neck joint 402 and the right elbow joint 404 and an axis connecting the right elbow joint 404 and the right wrist joint 406, and is 180° in FIG. 4.

[0036] Angle 414 represents the angle of the left leg. Angle 414 is the angle between an axis connecting the waist joint 407 and the left knee joint 408 and an axis connecting the left knee joint 408 and the left ankle joint 410, and is 180° in FIG. 4. Angle 415 represents the angle of the right leg. Angle 415 is the angle between an axis connecting the waist joint 407 and the right knee joint 409 and an axis connecting the right knee joint 409 and the right ankle joint 411, and is 180° in FIG. 4.

[0037] Fig. 5 is a diagram showing an example of a preparatory movement of a subject, and Fig. 6 is a diagram showing an example of a preparatory movement of a subject.

[0038] The movement determination unit 309 determines that the subject is making a preparatory movement when the subject is making the movement (posture) shown in the right diagram of FIG. 5 and the right diagram of FIG. 6. Note that the preparatory movements of the subject shown in FIGS. 5 and 6 are merely examples, and the movements are not limited to these. The movement determination unit 309 may determine whether the subject is making a preparatory movement, for example, based on the angle of the subject's left arm or right arm and the height from the ground to the center of gravity of the subject. In this embodiment, the movement determination unit 309 determines the preparatory movement of the subject based on a rule base, but is not limited to this. For example, the movement determination unit 309 may determine the preparatory movement of the subject using a learning model (deep learning) that learns video images recording the preparatory movements of the subject as learning data.

[0039] FIG. 5 will now be described. A subject 501 is shown in an image 500. The subject 501 has a neck joint 502 and a waist joint 503. The subject 501 shown on the left side of FIG. 5 is not making a preparatory movement. On the other hand, the subject 501 shown on the right side of FIG. 5 is making a preparatory movement.

[0040] Angle 504 represents the inclination of the movement (posture) of subject 501. Angle 504 is the angle formed between axis 505 connecting neck joint 502 and waist joint 503 and axis 506 extending perpendicularly from the ground. If angle 504 is equal to or greater than a threshold, movement determination unit 309 determines that subject 501 is making a preparatory movement. Here, the threshold is 30°, but is not limited to this. Note that, on the left side of FIG. 5, angle 504 does not exist (i.e., angle 504 is 0°), and therefore movement determination unit 309 determines that subject 501 is not making a preparatory movement.

[0041] 6 will be described below. A subject 601 is shown in an image 600. The subject 601 has a waist joint 602, a right knee joint 603, and a right ankle joint 604. Angle 605 represents the angle of the right leg. Angle 605 is the angle between an axis connecting the waist joint 602 and the right knee joint 603 and an axis connecting the right knee joint 603 and the right ankle joint 604, and is, for example, an angle of 180° or less.

[0042] If angle 605 is within a threshold, movement determination unit 309 determines that subject 601 is making a preparatory movement. Here, the threshold is 120°, but is not limited to this. Movement determination unit 309 then outputs the result of determining whether or not the subject is making a preparatory movement to region setting unit 310.

[0043] Returning to the explanation of Fig. 3, the area setting unit 310 sets a dead zone area in the imaging range of the imaging device 101 based on the tracking setting input by the user, and outputs the setting result to the recording unit 311. Note that the area setting unit 310 may also set a tracking sensitivity in addition to setting the dead zone area. The dead zone area refers to an area in which the imaging control device 104 does not perform PTZ control of the imaging device 101 when the center position of the subject is within the dead zone area.

[0044] FIG. 7 is a diagram showing an example of the dead zone.

[0045] An image 700 shows a subject 701 and a dead zone 702. The dead zone 702 is a rectangular area indicated by a dashed line. If the subject 701 is present in the dead zone 702, the imaging control device 104 does not perform PTZ control of the imaging device 101.

[0046] Area setting unit 310 is an area control means, and sets a dead zone area when the determination result of movement determination unit 309 indicates that a preparatory movement of subject 701 is occurring. Note that area setting unit 310 may change a dead zone area that has been set in advance. Here, changing a dead zone area includes reducing the size of a dead zone area set in the imaging range and invalidating the dead zone area (i.e., deleting the dead zone area).

[0047] Fig. 8 is a diagram illustrating an example of changing the dead zone region, and Fig. 9 is a diagram illustrating an example of changing the dead zone region.

[0048] A subject 801 appears in an image 800. Because the subject 801 is making a preparatory movement, the area setting unit 310 invalidates (i.e., deletes) the dead zone that the user set in advance in the image 800, which will be the imaging range. As a result, the imaging control device 104 performs PTZ control of the imaging device 101 in response to the subject 801 starting to move, thereby preventing the subject 801 from being lost.

[0049] Image 900 shows a subject 901 and a dead zone area 902. Dead zone area 902 is smaller than dead zone area 903 input in advance by the user. When changing the size of dead zone area 903, area setting unit 310 may change the size of dead zone area 903 depending on the position of subject 901 in image 900. When the determination result of movement determination unit 309 indicates that no preparatory movement has been made by subject 901, area setting unit 310 sets dead zone area 903 input in advance by the user in image 900.

[0050] 7, the subject 701 is not performing a preparatory movement, so the region setting unit 310 sets the dead zone region 702 input in advance by the user in the image 700. Then, the region setting unit 310 outputs the setting of the dead zone region 702 to the operation determination unit 312.

[0051] Returning to the explanation of Fig. 3, the recording unit 311 records the setting result of the dead zone area received from the area setting unit 310. The recording unit 311 also outputs the recorded setting of the dead zone area to the area setting unit 310.

[0052] The operation determination unit 312 determines whether to control the PTZ of the imaging device 101 based on whether the position (e.g., two-dimensional coordinates) of the subject received from the tracking unit 307 is within the dead zone area received from the area setting unit 310.

[0053] 7, the operation determination unit 312 determines that the PTZ control of the image capturing device 101 will not be performed because the center position 703 of the object 701 is in the dead zone area 702. Here, the object 701 is not performing a preparatory movement, so the dead zone area 702 is set in the image 700. This prevents the image capturing control device 104 from performing unnecessary PTZ control on the image capturing device 101 even if the object 701 moves slightly due to unsteadiness or the like.

[0054] 8, the operation determination unit 312 determines to perform PTZ control of the image capturing device 101 because there is no dead zone in the image 800. Here, the subject 801 is performing a preparatory movement, and therefore there is a possibility that the subject 801 will move at high speed outside the image 800. Therefore, the image capturing control device 104 performs PTZ control of the image capturing device 101 in response to the subject 801 starting to move, thereby preventing the subject 801 from being lost.

[0055] 9, the operation determination unit 312 determines that the PTZ control of the image capturing device 101 will not be performed because the center position 904 of the object 901 is in the dead zone area 902. Here, the object 901 is performing a preparatory movement, so the dead zone area 902, which is smaller than the dead zone area 903, is set in the image 900. As a result, the image capturing control device 104 does not perform PTZ control in response to the object 901 starting to move, but can perform PTZ control more quickly than when the dead zone area 903 is set.

[0056] FIG. 10 shows a diagram in which a subject is captured outside the dead zone.

[0057] The operation determination unit 312 determines that the PTZ control of the image capturing device 101 should be performed because the center position 1003 of the object 1001 is outside the dead zone area 1002. Then, the control unit 313 performs the PTZ control of the image capturing device 101 even if the object 1001 is not performing a preparatory movement. The operation determination unit 312 outputs the determination result indicating whether or not to perform a PTZ operation of the image capturing device 101 and the position of the object (for example, two-dimensional coordinates) to the control unit 313.

[0058] Returning to the explanation of Fig. 3, the control unit 313 calculates a PTZ control value for the image capturing device 101 based on the determination result from the operation determination unit 312 and the position of the subject. For example, the control unit 313 calculates the PTZ control value so that the position of the subject captured in the image at the current time is near the center of the imaging range of the image capturing device 101, but this is not limited to this. For example, the control unit 313 may determine the PTZ control value by a method of directly specifying the PTZ control value, or a method of specifying the direction and speed of each of panning and tilting. The control unit 313 outputs the PTZ control value to the drive device 103.

[0059] The output unit 314 outputs the video received from the acquisition unit 305 to the display device 115. The display device 115 displays the video received from the output unit 314 on the screen.

[0060] 11 is a flowchart showing the processing of the imaging control device according to the first embodiment. The imaging system 10 starts capturing an image of a subject when started by a user operation.

[0061] In S1101 , the acquisition unit 305 acquires an image from the image capturing device 101 and outputs the acquired image to the detection unit 306 and the output unit 314 .

[0062] In S1102, the input device 102 accepts a setting of the dead zone area from the user and outputs the setting of the dead zone area to the area setting unit 310. The area setting unit 310 outputs the setting of the dead zone area to the recording unit 311. The recording unit 311 records the setting of the dead zone area.

[0063] In S1103, the detection unit 306 detects a subject from the image based on a known subject detection method. The detection unit 306 outputs the position and image of the detected subject to the tracking unit 307.

[0064] In S1104, the tracking unit 307 selects the object to be tracked based on the position of the object. The tracking unit 307 outputs the position (for example, two-dimensional coordinates) and image of the object to be tracked as a tracking processing result to the posture estimation unit 308 and the operation determination unit 312.

[0065] In S1105, posture estimation unit 308 estimates the posture of the subject based on the positions (coordinates) of the subject's joint points received from tracking unit 307. Specifically, posture estimation unit 308 extracts the positions (coordinates) of the subject's joint points detected from the image, and estimates the posture of the subject using posture estimation technology for the extracted positions (coordinates) of the joint points. The posture of the subject refers to connection information between joint points (so-called human skeletal information) obtained based on the positions of the subject's joint points detected by posture estimation unit 308. The posture estimation unit 308 outputs the posture estimation result of the subject to movement determination unit 309.

[0066] In S1106, the movement determination unit 309 determines whether the subject is making a preparatory movement of rapidly moving based on the posture of the subject to be tracked. If the movement determination unit 309 determines that the subject is making a preparatory movement (Yes in S1106), the process proceeds to S1108. If the movement determination unit 309 determines that the subject is not making a preparatory movement (No in S1106), the process proceeds to S1107.

[0067] In S1107, the area setting unit 310 sets the dead zone area input by the user, which is acquired from the recording unit 311, to the imaging range of the imaging device 101. That is, the area setting unit 310 performs control to maintain the setting of the dead zone area as the initial setting.

[0068] In S1108, if the dead zone area input by the user has been set in the imaging range of the imaging device 101, the area setting unit 310 changes the setting of the dead zone area that has already been set.

[0069] In S1109, the operation determination unit 312 determines whether the center position of the subject (for example, two-dimensional coordinates) is in the dead zone. If the operation determination unit 312 determines that the center position of the subject is in the dead zone (Yes in S1109), the process proceeds to S1112. If the operation determination unit 312 determines that the center position of the subject is not in the dead zone (No in S1109), the process proceeds to S1110.

[0070] In step S1010, the control unit 313 generates a PTZ control command based on the center position of the object to be tracked. The control unit 313 outputs the PTZ control command to the driving device 103.

[0071] In S1111, the driving device 103 changes the imaging direction and imaging range by driving the PTZ of the imaging device 101 based on the PTZ control command.

[0072] In S1112, the output unit 314 outputs the video captured by the imaging device 101 whose PTZ parameters have been changed to the display device 115. If there are two or more imaging devices 101, the output unit 314 outputs to the display device 115 the video corresponding to the number of imaging devices 101.

[0073] In S1113, the CPU 200 determines whether or not an OFF switch (not shown) has been pressed to stop the operation of the image capture system 10. If the CPU 200 determines that the OFF switch (not shown) to stop the operation of the image capture system 10 has not been pressed (No in S1113), the process returns to S1101. If the CPU 200 determines that the OFF switch (not shown) to stop the operation of the image capture system 10 has been pressed (YES in S1113), the process ends.

[0074] According to the first embodiment, when a dead zone is set in the imaging range, the setting of the dead zone is changed depending on whether or not the subject makes a preparatory movement. This allows the PTZ control of the imaging device to be performed immediately when the subject starts to move, and allows the subject to be tracked without being lost outside the imaging range.

[0075] (Second embodiment) In the second embodiment, when the posture estimation result of the subject indicates that the subject is making a preparatory movement, the dead zone setting and the "PTZ acceleration" are changed. As a result, the second embodiment can track the subject even if the subject to be tracked starts to move rapidly. In the second embodiment, differences from the first embodiment will be described.

[0076] 12 is a block diagram showing the functional configuration of an imaging control device according to the second embodiment. Blocks having the same functions as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and therefore descriptions of those blocks will be omitted.

[0077] The imaging control device 104 further includes an acceleration change unit 1216 that corresponds to acceleration control means. The drive device 103 drives the PTZ of the imaging device 101 based on the PTZ control value received from the control unit 313 and the PTZ acceleration from the acceleration change unit 1216.

[0078] When the determination result of the movement determination unit 309 indicates that a preparatory movement of the subject is occurring, the acceleration change unit 1216 changes the PTZ acceleration to a value greater than the initial setting PTZ acceleration. The acceleration change unit 1216 outputs the changed PTZ acceleration to the driving device 103. Note that the initial setting PTZ acceleration is the PTZ acceleration set in advance by the user, but is not limited to this. Furthermore, the acceleration change unit 1216 can change not only the PTZ acceleration but also the "PTZ speed."

[0079] On the other hand, if the determination result of the movement determination unit 309 does not indicate that the subject is making a preparatory movement, the acceleration change unit 1216 changes the PTZ acceleration to the PTZ acceleration at the initial setting. The acceleration change unit 1216 outputs the PTZ acceleration at the initial setting to the driving device 103.

[0080] 13 is a flowchart showing the processing of the imaging control device according to the second embodiment. In FIG. 13, differences from FIG. 11 will be explained.

[0081] In S1301 , the acceleration change unit 1216 changes the PTZ acceleration to the initial setting PTZ acceleration, and outputs the initial setting PTZ acceleration to the driving device 103 .

[0082] In S1302 , the acceleration change unit 1216 changes the PTZ acceleration to a value greater than the initial PTZ acceleration, and outputs the changed PTZ acceleration to the driving device 103 .

[0083] According to the second embodiment, when a dead zone is set in the imaging range, the setting of the dead zone and the PTZ acceleration are changed depending on whether or not the subject makes a preparatory movement. This allows the PTZ control of the imaging device to be performed immediately when the subject starts to move, and enables tracking of the subject without losing the subject outside the imaging range.

[0084] (Third embodiment) In the third embodiment, similar to the second embodiment, when the posture estimation result of the subject to be tracked indicates that the subject is making a preparatory movement, the dead zone setting and the PTZ acceleration are changed. Also, in the third embodiment, when the subject stops making a preparatory movement for a predetermined period of time, the dead zone setting and the PTZ acceleration setting are restored to their original state. In the third embodiment, the differences from the first and second embodiments will be described.

[0085] 14 is a block diagram showing the functional configuration of an imaging control device according to the third embodiment. Blocks having the same functions as those in the first and second embodiments are denoted by the same reference numerals as those in the first and second embodiments, and therefore descriptions of those blocks will be omitted.

[0086] The imaging control device 104 further includes a history recording unit 1417 and a movement determination unit 1418 .

[0087] History recording unit 1417 records the motion determination result of motion determination unit 309 and the position of the subject. At this time, history recording unit 1417 records the motion determination result and the position of the subject for at least several seconds. History recording unit 1417 outputs the recorded results to movement determination unit 1418.

[0088] The movement determination unit 1418 determines whether or not the subject taking the preparatory movement (predetermined posture) has actually moved within a predetermined time based on the recording results of the history recording unit 1417. The predetermined time may be a time sufficient to determine whether or not the subject has moved, and may be, for example, one second, but is not limited to this. Note that the movement determination unit 1418 determines that the subject has moved if the time during which the subject continues to make the preparatory movement is within the predetermined time.

[0089] FIG. 15 is a diagram illustrating a method for determining the movement of a subject during a preparatory movement.

[0090] Image 1500 shows subject 1501, subject 1502, and movement distance 1503. Subject 1501 represents the subject one second before the current time, and subject 1502 represents the subject at the current time. Movement distance 1503 represents the distance the subject has moved within a predetermined time (the difference between the positions of subjects 1501 and 1502). Here, movement determination unit 1418 determines that the subject has moved if movement distance 1503 is greater than a threshold. The threshold may be any distance that allows determination that the subject has moved, and may be, for example, 1 m, but is not limited to this. The predetermined time is 1 second, but is not limited to this.

[0091] FIG. 16 is a diagram illustrating a method for determining the movement of a subject during a preparatory movement.

[0092] Image 1600 shows object 1601, object 1602, movement distance 1603, and dead zone area 1604. Object 1601 represents the object one second before the current time, and object 1602 represents the object at the current time. Movement distance 1603 represents the distance the object has moved within a predetermined time. Here, movement determination unit 1418 determines that the object has not moved if movement distance 1603 is smaller than a threshold. The predetermined distance is 1 m, and the predetermined time is 1 second, but are not limited to these. Movement determination unit 1418 outputs the determination result regarding whether the object has moved to area setting unit 310 and acceleration change unit 1216.

[0093] The region setting unit 310 sets the setting of the dead zone area received from the input device 102 in the imaging range of the imaging device 101 and outputs the setting result to the recording unit 311. Furthermore, when the determination result of the movement determination unit 1418 indicates that the subject has moved, the region setting unit 310 invalidates (deletes) the dead zone area as a setting change of the dead zone area due to user input, as shown in Fig. 15. On the other hand, when the determination result of the movement determination unit 1418 does not indicate that the subject has moved, the region setting unit 310 sets the dead zone area due to user input, as shown in Fig. 16.

[0094] If the determination result of the movement determination unit 1418 indicates that the subject has moved, the acceleration change unit 1216 changes the PTZ acceleration to a value greater than the PTZ acceleration at the time of initial setting, and outputs the changed PTZ acceleration to the driving device 103. On the other hand, if the determination result of the movement determination unit 1418 does not indicate that the subject has moved, the acceleration change unit 1216 changes the PTZ acceleration to the PTZ acceleration at the time of initial setting, and outputs the PTZ acceleration at the time of initial setting to the driving device 103.

[0095] 17 is a flowchart showing the processing of the imaging control device according to the third embodiment. In FIG. 17, differences from FIG. 11 will be explained.

[0096] In S1701, the acceleration change unit 1216 changes the PTZ acceleration to the initial setting PTZ acceleration, and outputs the changed initial setting PTZ acceleration to the driving device 103.

[0097] In S1702, the movement determination unit 1418 determines whether the subject has been performing preparatory movements for a predetermined time based on the result of the subject's movement determination by the history recording unit 1417 and the position (coordinate) history information. If the movement determination unit 1418 determines that the subject has been performing preparatory movements for a predetermined time (Yes in S1702), the process proceeds to S1705. If the movement determination unit 1418 determines that the subject has not been performing preparatory movements for a predetermined time (No in S1702), the process proceeds to S1703.

[0098] In step S1703, if a dead zone has been set by user input, the region setting unit 310 sets the dead zone obtained by changing the dead zone by user input as the imaging range of the imaging device 101.

[0099] In S1704 , the acceleration change unit 1216 changes the PTZ acceleration to a value greater than the initial PTZ acceleration, and outputs the changed PTZ acceleration to the driving device 103 .

[0100] In S1705, the movement determination unit 1418 determines whether the distance moved by the subject within a predetermined time is equal to or greater than a threshold, based on the result of the subject's movement determination by the history recording unit 1417 and the history information of the subject's position (coordinates). If the movement determination unit 1418 determines that the distance moved by the subject within the predetermined time is equal to or greater than the threshold (Yes in S1705), the process proceeds to S1703. If the movement determination unit 1418 determines that the distance moved by the subject within the predetermined time is not equal to or greater than the threshold (No in S1705), the process proceeds to S1706.

[0101] In S1706, the area setting unit 310 sets the dead zone area setting input by the user, acquired from the recording unit 311, in the imaging range of the imaging device 101.

[0102] In S1707 , the acceleration change unit 1216 changes the PTZ acceleration to the initial setting PTZ acceleration, and outputs the initial setting PTZ acceleration to the driving device 103 .

[0103] According to the third embodiment, the setting of the dead zone and the PTZ acceleration are changed based on the duration of the subject's preparatory movement and the distance traveled by the subject within a predetermined time. This allows for instant tracking of a subject that is performing preparatory movement and actually moving. Furthermore, when tracking a subject that is stationary or making small movements while continuing its preparatory movement (i.e., performing a feint movement), setting the dead zone can suppress unnecessary PTZ control.

[0104] (Variation) In a modification of the third embodiment, the moving direction of the subject is estimated from the movement history of the subject, and when it is determined that the subject has made a preparatory movement, the imaging direction is controlled to the same direction as the moving direction of the subject.

[0105] Fig. 18 is a diagram illustrating imaging control performed by an imaging device after a subject makes a preparatory movement. Fig. 18(a) shows a diagram illustrating estimation of the subject's movement direction when the subject makes a preparatory movement. Fig. 18(b) shows imaging control performed in accordance with the subject's movement direction.

[0106] In FIG. 18(a), an image 1800 shows an object 1801, a dead zone area 1803, and a movement direction 1804. The movement direction 1804 represents the movement direction of the object 1801 estimated by the movement determination unit 1418 based on the movement history of the object. In FIG. 18(b), the image 1800 shows the object 1801 and a dead zone area 1803. A pan direction 1805 represents the direction in which the pan of the image capture device 101 is changed. The operation determination unit 312 determines, based on the estimation result (movement direction 1804) of the movement determination unit 1418, to control the image capture device 101 in the pan direction 1805 with a predetermined pan value. The predetermined pan value is a value that allows a space to be formed between the object 1801 in the image in FIG. 18(b) and the right edge of the image. This space is provided so that the image capture control device 104 has some leeway when capturing an image of the object 1801. The larger the space, the greater the time available for tracking the subject 1801. This allows the imaging control device 104 to capture (track) the subject 1801 without losing it.

[0107] Furthermore, the movement determination unit 309 may weight each of a plurality of candidates for preparatory movements (postures) made by the subject, and determine whether or not the subject has made a preparatory movement based on whether or not the weighted score of the determined movement exceeds a threshold. The candidates for preparatory movements include, for example, a posture in which the height from the ground to the center of gravity of the subject in an upright state is low, a posture in which angle 504 in FIG. 5 is formed, and a posture in which angle 605 in FIG. 6 is formed.

[0108] Furthermore, posture estimation unit 308 may estimate, as posture information of the subject, for example, the facial orientation and gaze direction of the subject in addition to elbow joints, knee joints, hip joints, ankle joints, wrist joints, etc. Then, movement determination unit 309 may determine whether the subject is making a preparatory movement based on whether the facial orientation or gaze direction of the subject is directed outside the dead zone.

[0109] 19A and 19B are diagrams illustrating how preparatory movements are determined from information other than the subject's joints. Fig. 19A is a diagram illustrating how preparatory movements are determined from the direction of the subject's face. Fig. 19B is a diagram illustrating how preparatory movements are determined from the subject's line of sight.

[0110] In the left diagram of FIG. 19(a), the posture estimation unit 308 estimates the facial direction of the subject 1901 based on the positions (coordinates) of the facial features of the subject 1901. The facial features are the positions of the left eye, right eye, and mouth. The movement determination unit 309 determines that the subject 1901 is making a preparatory movement when the facial direction of the subject 1901 is directed to the left side of the dead zone area 1902. Note that the right diagram of FIG. 19(a) shows a case where the facial direction of the subject 1901 is directed to the right side of the dead zone area 1902, but the movement determination unit 309 can determine the preparatory movement of the subject 1901 using the same method as above.

[0111] In the left diagram of FIG. 19(b), the posture estimation unit 308 estimates the gaze direction of the subject 1903 based on the positions (coordinates) of the facial features of the subject 1903. The facial features are the positions of the left and right eyes. The movement determination unit 309 determines that the subject 1903 has made a preparatory movement when the gaze direction of the subject 1903 is directed to the left side of the dead zone area 1904. Note that the right diagram of FIG. 19(b) shows a case where the gaze direction of the subject 1903 is directed to the right side of the dead zone area 1904, but the movement determination unit 309 can determine the preparatory movement of the subject 1903 using the same method as above.

[0112] (Fourth embodiment) In the first to third embodiments, a method has been described in which the center position of a subject is used to determine whether the subject is in a dead zone. In the fourth embodiment, a method will be described in which the "center of gravity" calculated from position information of the subject's joints is used as the center position of the subject.

[0113] The determination process in S1109 determines whether the center position of the subject is in the dead zone. If the center position of the subject is in the center of a rectangular frame (hereinafter referred to as a bounding box) surrounding the subject, the center position of the subject will move just by moving the subject's hands and / or feet. Therefore, it is necessary to take into account the amount of change in the center position of the subject and to enlarge the dead zone. By setting the center position of the subject to the center of gravity of the subject's torso (excluding the hands and feet), it is possible to reduce the change in the center position due to movement of the subject's hands and / or feet. Therefore, it is possible to make the dead zone smaller than before.

[0114] FIG. 20 is a diagram illustrating an example of a method for calculating the center of gravity of a subject.

[0115] Image 2000 shows an example of joint points of a human body (subject) estimated by posture estimation unit 308, and shows an example of joint points of the subject that are different from those in FIG. 4. Image 2000 shows right shoulder 2001, left shoulder 2002, right hip 2003, left hip 2004, right knee 2005, left knee 2006, right heel 2007, left heel 2008, and nose 2009 as joint points of the subject. Center of gravity 2010 is the center of gravity of the subject, and is illustrated by, for example, a "+". Note that "left" and "right" in image 2000 represent "left" and "right" as seen from the subject's perspective.

[0116] In this embodiment, posture estimation unit 308 calculates center of gravity 2010 of the subject based on position information of joint points of the torso excluding the hands, feet (legs), and head of the subject. Specifically, posture estimation unit 308 calculates center of gravity 2010 of the subject based on the coordinates of four points: right shoulder 2001, left shoulder 2002, right hip 2003, and left hip 2004. Note that in this embodiment, posture estimation unit 308 uses position information of the shoulders and hips excluding position information of the hands, feet, and head to calculate center of gravity 2010 of the subject.

[0117] However, as long as it is in line with the purpose of appropriately determining when the subject starts to move, the body parts (joint points) of the subject used to calculate the center of gravity 2010 of the subject may be a combination with other body parts. For example, when the subject performs a high kick in martial arts (e.g., karate) in which the foot moves widely on the spot, the posture estimation unit 308 does not use position information of the subject's feet to calculate the center of gravity 2010 of the subject. On the other hand, the posture estimation unit 308 may use position information of the subject's feet to calculate the center of gravity 2010 of a subject playing a sport in which the center of gravity 2010 of the subject does not change widely.

[0118] Furthermore, posture estimation unit 308 may output the reliability of the estimation for each part (joint point) of the subject, and may change the part (joint point) used to calculate the subject's center of gravity 2010 depending on the reliability of each output part of the subject. For example, if the likelihood (i.e., reliability) of the estimated position information of the subject's right hip 2003 and left hip 2004 is lower than a threshold, posture estimation unit 308 may calculate the subject's center of gravity 2010 using only the position information of the subject's right shoulder 2001 and left shoulder 2002.

[0119] FIG. 21 is a diagram illustrating the change in the center position of a conventional subject and the center position of a subject of the present invention in response to the movement of the subject.

[0120] FIG. 21(a) shows an example in which the center of a bounding box is the center position of the subject. Image 2100 shows a state in which the subject is stationary. In image 2100, the subject is standing upright and surrounded by bounding box 2101. Center 2102 is the center of bounding box 2101 and is indicated by a star. Images 2110 and 2120 show a state in which the subject is not moving but is making large movements of its arms and legs in a predetermined position. In image 2110, the subject is performing, for example, an upper roundhouse kick (high kick). In image 2120, the subject is performing a golf backswing. The subjects in images 2110 and 2120 are surrounded by bounding boxes 2111 and 2121, respectively. Center 2112 is the center of bounding box 2111 and is indicated by a star. Center 2122 is the center of bounding box 2121 and is indicated by a star.

[0121] Amount of position change 2113 is the amount of position change between center of gravity 2102 and center of gravity 2112. Amount of position change 2123 is the amount of position change between center of gravity 2102 and center of gravity 2122. As indicated by amount of position change 2113 and amount of position change 2123, when the center of the bounding box is set as the center position of the subject, the position of the center of gravity changes significantly depending on the movements of the subject's hands and feet. There is a possibility that the movement determination unit 309 will erroneously detect the start of movement of the subject based on amount of position change 2113 or amount of position change 2123.

[0122] FIG. 21(b) shows an example in which the center of gravity calculated from the position information of the parts (joint points) of the subject is set as the center position of the subject.

[0123] Images 2130, 2140, and 2150 correspond to images 2100, 2110, and 2120, respectively, and therefore detailed description thereof will be omitted. However, centers of gravity 2132 to 2152 of the subjects in images 2130 to 2150 are different from centers of gravity 2102 to 2122 of the subjects in images 2100 to 2120. Centers of gravity 2132, 2142, and 2152 are calculated from position information of the subject's left and right shoulders and waist, and correspond to center of gravity 2010 in FIG. 20. As shown in FIG. 21(b), centers of gravity 2132, 2142, and 2152 are located at approximately the same position, and, for example, when center of gravity 2132 is used as a reference, the amount of positional change between centers of gravity 2142 and 2152 relative to center of gravity 2132 is very small. Since the posture estimation unit 308 calculates the center of gravity of the subject based on position information of the shoulders and waist of the human body (subject), the calculated center of gravity is not significantly affected by the movements of the subject's hands and feet. The movement determination unit 309 can determine whether the subject has started to move based on whether the calculated position of the subject's center of gravity is within a dead zone set in the imaging range. For example, if the calculated position of the subject's center of gravity is within a dead zone set in the imaging range, the movement determination unit 309 determines that the subject has not started to move. On the other hand, if the calculated position of the subject's center of gravity is not within a dead zone set in the imaging range, the movement determination unit 309 determines that the subject has started to move. The movement determination unit 309 may also determine whether the subject has started to move based on whether the amount of lateral movement of the calculated position of the subject's center of gravity within the imaging range is equal to or greater than a threshold. Note that the amount of lateral movement does not include the amount of movement due to a change in the imaging direction of the imaging device. This allows the movement determination unit 309 to properly determine that the subject has not started to move, thereby reducing false detection of the subject's start of movement.

[0124] In this embodiment, the dead zone is controlled by comparing the dead zone in the captured image with the position of the subject in the captured image while the camera is stationary and capturing the subject without PTZ control. Alternatively, the dead zone may be controlled so that the camera detects the start of subject movement and automatically tracks the subject while the user manually controls the camera's PTZ using a controller (not shown). In this case, the dead zone control requires the motion determination unit 309 to determine whether the subject has started moving based on the position of the subject in real space, since the position of the subject in the captured image changes due to the user's PTZ control. Here, the position of the subject in real space is a position obtained by transforming the calculated position of the subject's center of gravity using a known coordinate transformation method. For example, if the position of the subject in real space moves by more than a predetermined value, the motion determination unit 309 determines that the subject is outside the dead zone. The method for determining whether the subject is outside the dead zone in real space is not limited to the above. As an example, assuming a virtual sphere centered on the camera, the direction of the lens optical axis, the direction of the subject position, and the dead zone can be expressed by polar coordinates on the sphere. The operation determination unit 309 can determine whether the subject position on the polar coordinates is inside or outside the dead zone. Note that, although the description in this embodiment is based on the premise that a dead zone is set in the imaging range, a configuration may also be adopted in which PTZ control is performed instantly based on the movement of the subject center position (i.e., the position of the center of gravity of the subject) without providing a dead zone.

[0125] As described above, the posture estimation unit 308 can calculate the center of gravity of the subject based on the position information of the subject's torso, excluding the hands, feet, and head. Then, when the movement determination unit 309 regards the calculated center of gravity as the center position of the subject, it can accurately determine when the subject starts to move without being affected by changes in the center of gravity position due to the movement of the subject's hands and feet. As a result, it is possible to make the dead zone in the video (imaging range) smaller than before, and to perform PTZ control immediately in response to the start of the subject's movement. This makes it possible to continue capturing the subject so that it remains within the imaging range.

[0126] (Fifth embodiment) In the first to third embodiments, a method has been described in which a preparatory movement of a subject is used to determine whether to change the dead zone (to make the dead zone smaller or to disable the dead zone). In the fifth embodiment, an example will be described in which, when it is detected that the subject has started to move, the dead zone is changed (specifically, disabled) and the control unit 313 immediately performs PTZ control. A method will be described in which detection of the subject's left and right feet switching is used to detect the subject's start of movement.

[0127] 22 is a flowchart illustrating the processing of the automatic photography system according to the fifth embodiment. Note that the processing of S1101 to S1105 and S1107 to S1113 is the same as in the first embodiment, and therefore the description thereof will be omitted.

[0128] In S2206, the movement determination unit 309 determines whether or not the subject's left and right feet have been swapped, based on the input posture information of the subject. If the movement determination unit 309 determines that the subject's left and right feet have not been swapped (No in S2206), the process proceeds to S1107. If the movement determination unit 309 determines that the subject's left and right feet have been swapped (Yes in S2206), the process proceeds to S1108. The method for determining whether the subject's left and right feet have been swapped will be described later.

[0129] In S1108, the area setting unit 310 invalidates the setting of the dead zone area even if the dead zone area input by the user has been set in the video (image). As a result, in S1109, the operation determination unit 309 determines that the subject is not present in the dead zone area, and the control unit 313 immediately performs PTZ control in accordance with the movement of the subject.

[0130] In S1107, the region setting unit 310 sets the dead zone region input by the user as the imaging range.

[0131] The posture information input to the movement determination unit 309 in S2206 is position information of the subject's right heel 2007 and left heel 2008. The movement determination unit 309 compares the input posture information with the previous state (position information of the subject's joint points recorded in the recording unit 311) and determines whether the subject's left and right feet have been swapped. The movement determination unit 309 acquires information about the previous state by referring to the information recorded in the recording unit 311.

[0132] FIG. 23 is a diagram illustrating a method for determining whether the subject's left and right feet have been swapped.

[0133] Image 2300 shows the subject standing upright (stationary) at time t0. Right heel 2007 and left heel 2008 are the joint points of the subject's feet. Here, right heel 2007 and left heel 2008 indicate the positions of the "right" and "left" heels, respectively, when seen from the subject's perspective. In this case, the relationship "x-axis coordinate of right heel 2007 < x-axis coordinate of left heel 2008" holds. Note that after time t0, the subject begins walking in the x-axis direction.

[0134] Image 2301 shows a state in which the subject's feet have swapped positions at time t1. The subject's feet swapping positions refers to the position of the right foot being swapped with the position of the left foot, and more specifically, the x-axis coordinate of the right heel 2007 becoming larger than the x-axis coordinate of the left heel 2008. In this case, the relationship "x-axis coordinate of right heel 2007 > x-axis coordinate of left heel 2008" holds. The movement determination unit 309 can determine the start of movement of the subject based on the magnitude relationship between the x-axis coordinates of the right heel 2007 and the left heel 2008 (i.e., the subject's left and right feet swapping). Note that the movement determination unit 309 uses position information of the subject's left and right heels to determine the subject's left and right feet swapping positions. However, if position information of the left and right heels cannot be obtained based on the estimation result by the posture estimation unit 308, the movement determination unit 309 may use position information of the left and right knees of the subject, etc., instead of the position information of the left and right heels of the subject. For example, the posture estimation unit 308 may output the reliability of the estimation for each part (joint point) of the subject, and may select parts (joint points) of the subject other than the left and right heels depending on the level of the reliability for each part of the subject that has been output.

[0135] When a subject starts moving (i.e., walking), the subject's left and right feet switch places. Therefore, if changes in the position information of the left and right feet are used to determine when the subject has started moving, the accuracy of determining when the subject has started moving can be improved. When the subject's left and right feet switch places, the area setting unit 310 disables the dead zone area of ​​the imaging range. Then, the control unit 313 immediately performs PTZ control in accordance with the movement of the subject. This makes it possible to continue capturing images of the subject so that the subject remains within the imaging range.

[0136] (Sixth embodiment) In the fifth embodiment, an example was described in which, when a subject's movement start motion is detected, the dead zone is changed (specifically, disabled), and the control unit 313 immediately performs PTZ control. In particular, a method for detecting a change in the subject's left and right feet to detect the subject's movement start motion was described. In the sixth embodiment, a method using an inference model for detecting the subject's movement start motion will be described. The movement determination unit 309 has an inference model that has learned positive examples and negative examples of the subject's movement start motion. Therefore, the movement determination unit 309 has a function of outputting whether or not the subject has started moving based on joint movement information as posture information of the subject. When the inference result based on the inference model of the movement determination unit 309 indicates that the subject has started moving, the region setting unit 310 changes (specifically, disabled) the dead zone of the imaging range, and the control unit 313 immediately performs PTZ control of the camera.

[0137] FIG. 24 is a schematic diagram illustrating an inference model of the operation determining unit 309.

[0138] The inference model is unique to this embodiment. When posture information is input to the inference model as an input value X, the inference model outputs, as an output value Y, whether the subject has started to move (whether it has started to move) and a reliability. In this embodiment, posture information is input to the inference model in a time series, and the inference model estimates whether the subject has started to move. This embodiment uses machine learning techniques that can handle time series information, such as well-known techniques such as RNN (Recurrent Neural Network) and LSTM (Long Short-Term Memory).

[0139] Fig. 25 is a diagram showing an example of training data for training an inference model. Fig. 25(a) is training data of a positive case indicating that the subject has started to move, and shows posture information of the subject moving (walking) in the X-axis direction at times t0 to t3.

[0140] Figures 25(b) and 25(c) are training data of negative examples showing that the subject did not start moving. Figure 25(b) shows posture information capturing the subject stepping in place from time t0 to t3. Figure 25(c) shows posture information capturing the subject making a golf swing from time t0 to t3.

[0141] The inference model learns whether or not the subject has started to move based on the training data in Figures 25(a) to (c) as the input value X. In this way, by training the inference model using training data that includes positive and negative cases, the inference model can extract features related to the movement of the subject, such as changes in the center of gravity coordinates contained in the posture information, and can determine with high accuracy whether or not the subject has started to move.

[0142] 26 is a flowchart illustrating the processing of the automatic photography system according to the sixth embodiment. Note that the processing of S1101 to S1105 and S1107 to S1113 is the same as in the first embodiment, and therefore the description thereof will be omitted.

[0143] In S2606, the movement determination unit 309 normalizes the input posture information of the subject. When the size of the subject in the captured video is small, the distance between the coordinates of one joint point and the other joint point calculated from the posture information is also short, resulting in the subject's joint points being densely packed together. If posture information with densely packed joint points is input to the inference model, the accuracy of the inference model's inference regarding whether the subject has started moving may decrease. Therefore, the movement determination unit 309 normalizes the distance between a predetermined combination of joint points to a predetermined length. It is preferable to use a combination of parts (joint points) such that the distance between the parts (joint points) does not change due to individual differences. For example, in FIG. 20, the movement determination unit 309 changes the coordinates (positions) of each joint point so that the length of the line between the right shoulder 2001 and left shoulder 2002 and the line perpendicular to the line where the nose 2009 intersects with the line become a predetermined length. This makes it possible to improve the inference model's inference accuracy regarding whether the subject has started moving.

[0144] In S2607, the movement determination unit 309 inputs normalized posture information (i.e., posture information with corrected joint point positions) to the inference model and obtains an inference result of whether or not the subject has started moving and its reliability as the output of the inference model. If the movement determination unit 309 determines that the output result of the inference model indicates that the subject has started moving, the process proceeds to S2608. On the other hand, if the movement determination unit 309 determines that the output of the inference model indicates that the subject has not started moving, the process proceeds to S1107.

[0145] In S2608, if the operation determination unit 309 determines that the reliability of the subject starting to move as an output result of the inference model is higher than a predetermined value (Yes in S2608), the process proceeds to S1108.

[0146] In S1108, the operation determination unit 309 invalidates the setting of the dead zone even if a setting of the dead zone input by the user exists. As a result, in S1109, the operation determination unit 309 determines that the subject does not exist in the dead zone, and the control unit 313 immediately performs PTZ control in accordance with the movement of the subject.

[0147] In S2608, if the operation determination unit 309 determines that the reliability of the subject's movement has started as the output result of the inference model is lower than a predetermined value (No in S2608), the process proceeds to S1107. In this case, the control unit 313 suppresses PTZ control based on the setting of the dead zone input by the user. Note that in this embodiment, if the reliability of the subject's movement has started is lower than a predetermined value, it is determined that the subject has not started moving. Here, if suppressing unnecessary PTZ control is prioritized, the predetermined value may be set high. On the other hand, if tracking the subject's rapid movement is prioritized, the predetermined value may be set low. An input means may be provided that allows the user to arbitrarily set the predetermined value.

[0148] The inference model can infer whether the subject is moving based on the normalized posture information. If the inference result of the inference model is used to determine whether the subject has started to move, the accuracy of determining whether the subject has started to move will be further improved. The area setting unit 310 disables the dead zone area of ​​the imaging range when the inference model outputs an inference result indicating that the subject has started to move. Then, the control unit 313 immediately performs PTZ control in accordance with the movement of the subject. This makes it possible to continue capturing the subject so that it remains within the imaging range.

[0149] The methods for detecting the start of movement of a subject in the fifth and sixth embodiments have been described on the assumption that dead zone control is performed when automatically tracking a subject. However, the methods for detecting the start of movement of a subject may also be used to switch a PTZ camera from a manual operation mode to an automatic tracking mode. For example, if a user detects the start of movement of a subject while manually operating the PTZ camera using a controller (not shown), the PTZ camera may switch its operation mode to the automatic tracking mode of the subject.

[0150] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0151] The disclosure of this specification includes the following imaging control device, imaging device, imaging control method, and program. (Item 1) An imaging control device that controls an imaging direction of an imaging device that images a subject, an estimation means for estimating a posture of the subject detected from an image captured by the imaging device; an area control means for setting a first area in an imaging range of the imaging device based on whether the estimated posture of the subject is a predetermined posture; and an imaging control means for controlling the imaging direction based on whether the subject is present in the first area. Imaging control device. (Item 2) The size of the first region is larger when the posture of the subject is not a predetermined posture than when the posture of the subject is a predetermined posture. Item 1. An imaging control device according to item 1. (Item 3) an acceleration control unit that controls the acceleration when changing the imaging direction based on whether the posture of the subject is a predetermined posture or not; Item 1 or 2. An imaging control device. (Item 4) the acceleration control means performs control to increase the acceleration when the imaging direction is changed when the posture of the subject is a predetermined posture. Item 3. An imaging control device according to item 3. (Item 5) the acceleration control means performs control to maintain the acceleration when the imaging direction is changed if the posture of the subject is not a predetermined posture. Item 3. An imaging control device according to item 3 or 4. (Item 6) the estimation means estimates a movement direction of the subject based on a movement history of the subject; the imaging control means controls the imaging direction to be the same as the estimated movement direction when the posture of the subject is a predetermined posture. 6. An imaging control device according to any one of items 1 to 5. (Item 7) the estimation means estimates a joint point of the subject based on the detected position of the subject; the area control means sets the first area based on a comparison between a direction of a line connecting the joint points or a tilt of an axis passing through the center of the subject with respect to the ground and a threshold value; 7. An imaging control device according to any one of items 1 to 6. (Item 8) the area control means sets the first area based on whether or not the height from the ground to the center of gravity of the subject is lower than a threshold value. 8. An imaging control device according to any one of items 1 to 7. (Item 9) the area control means sets the first area based on whether the face direction or the line of sight direction of the subject is a predetermined direction or not. 9. An imaging control device according to any one of items 1 to 8. (Item 10) The predetermined direction is a direction in which the face direction or the line of sight direction of the subject is directed outside the imaging range. Item 9. An imaging control device according to item 9. (Item 11) the region control means performs control to reduce the size of the first region when the time during which the posture of the subject continues to be the predetermined posture is within a threshold value; the acceleration control means performs control to increase the acceleration when changing the imaging direction when the time is within a threshold value. 6. The imaging control device according to any one of items 3 to 5. (Item 12) when the time during which the posture of the subject continues to be in a predetermined posture exceeds a threshold, the area control means performs control to reduce the size of the first area based on whether or not a distance moved by the subject within the time is equal to or greater than a threshold; When the time exceeds a threshold, the acceleration control means performs control to increase the acceleration when changing the imaging direction based on whether the distance is equal to or greater than a threshold. 6. The imaging control device according to any one of items 3 to 5. (Item 13) a detection means for detecting the subject from the image; 13. An imaging control device according to any one of items 1 to 12. (Item 14) an imaging means for imaging the subject in an imaging direction controlled by the imaging control means; The imaging control device according to any one of items 1 to 13 is provided. Imaging device. (Item 15) An imaging control method executed by an imaging control device that controls an imaging direction of an imaging device that images a subject, comprising: an estimation step in which an estimation means of the imaging control device estimates a posture of the subject detected from an image captured by the imaging device; an area control step in which an area control means of the imaging control device sets a first area as an imaging range of the imaging device based on whether the estimated posture of the subject is a predetermined posture; an imaging control step in which an imaging control means of the imaging control device controls the imaging direction based on whether the subject is present in the first area; Imaging control method. (Item 16) A program for causing a computer to execute an imaging control method executed by an imaging control device that controls an imaging direction of an imaging device that images a subject, The imaging control method includes: an estimation step in which an estimation means of the imaging control device estimates a posture of the subject detected from an image captured by the imaging device; an area control step in which an area control means of the imaging control device sets a first area as an imaging range of the imaging device based on whether the estimated posture of the subject is a predetermined posture; an imaging control step in which an imaging control means of the imaging control device controls the imaging direction based on whether the subject is present in the first area; program. (Item 17) An imaging control device that controls an imaging direction of an imaging device that images a subject, an estimation means for estimating a posture of the subject detected from an image captured by the imaging device; a determining means for determining whether the subject has started to move based on the posture information of the subject estimated by the estimating means; and an imaging control means for controlling the imaging direction based on the determination result of the determination means. Imaging control device. (Item 18) the determining means determines whether the subject has started to move based on whether the position of the center of gravity of the subject calculated from the torso of the subject, among the posture information of the subject estimated by the estimating means, is present in a predetermined area within the imaging range of the imaging device. Item 18. An imaging control device according to item 17. (Item 19) the determining means determines whether the subject has started to move based on a lateral movement amount of the position of the center of gravity of the subject within the imaging range of the imaging device. Item 19. An imaging control device according to item 18. (Item 20) the determining means determines whether the subject has started to move based on the position in real space obtained by converting the position of the center of gravity of the subject; 20. An imaging control device according to any one of items 17 to 19. (Item 21) the estimation means outputs, as the posture information, a reliability indicating the likelihood of an estimation result for each part of the subject; the determining means selects a part of the subject to be used for calculating the position of the center of gravity of the subject from the posture information based on the reliability. 21. An imaging control device according to any one of items 17 to 20. (Item 22) further comprising a detection means for detecting a walking motion of the subject based on a comparison between a position of one part of the subject and a position of another part located on the opposite side of the one part, among the posture information of the subject estimated by the estimation means; the determining means determines whether the subject has started to move based on the detection result of the detecting means. 22. An imaging control device according to any one of items 17 to 21. (Item 23) the detection means selects a part of the subject to be used for detecting a walking motion of the subject based on each reliability indicating a likelihood of an estimation result for each part of the subject as the posture information; The walking motion of the subject is a motion in which the left and right feet of the subject are switched. Item 23. An imaging control device according to item 22. (Item 24) the determining means determines whether the subject has started to move based on the posture information of the subject estimated by the estimating means and an inference model that has been trained in advance using teacher data. 24. An imaging control device according to any one of items 17 to 23. (Item 25) the training data includes positive case data and negative case data, The positive case data is data representing posture information of the subject in a time series from a start of movement to a movement during the movement, The negative example data is data representing posture information of actions performed by the subject without moving in a time series. Item 25. An imaging control device according to item 24. (Item 26) The inference model outputs an inference result inferring whether the subject is moving or not and a reliability indicating the likelihood of the inference result; the determining means determines whether the subject has started to move based on the inference result by the inference model and the reliability. Item 26. An imaging control device according to item 24 or 25. (Item 27) The determination means normalizes the posture information of the subject so that the posture information to be input to the inference model becomes data of a predetermined size. 27. An imaging control device according to any one of items 24 to 26. (Item 28) the normalization process is a process of normalizing the posture information so that a distance between a position of one part of the subject and a position of another part of the subject in the posture information has a predetermined length. Item 28. An imaging control device according to item 27.

[0152] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0153] 10: Imaging system 101: Imaging device 102: Input device 103: Drive unit 104: Imaging control device 115:Display device 150: Network 200:CPU 201:RAM 202:ROM 203: Storage section 204:I / F

Claims

1. a detection means for detecting a subject from an image captured by the imaging device; a control means for controlling either an imaging direction of the imaging device or a zoom of the imaging device so as to track and image the subject; determining means for determining the posture of the subject; The control means setting tracking parameters based on the posture of the subject; a control device that controls either an imaging direction of the imaging device or a zoom of the imaging device based on the parameter corresponding to the posture of the subject;

2. the tracking-related parameter is a dead zone in which neither the imaging direction of the imaging device nor the zoom of the imaging device is performed, The control device according to claim 1 , wherein the control means controls either the imaging direction of the imaging device or the zoom of the imaging device by changing the size of the dead zone in accordance with the posture of the subject.

3. The control device according to claim 2 , wherein the control means controls the imaging direction of the imaging device when the subject is not present in the dead zone.

4. The control device according to claim 2 , wherein the control means does not control the imaging direction of the imaging device when the subject is present in the dead zone.

5. The control device according to claim 2 , wherein the size of the dead zone when the subject is in a predetermined posture is smaller than the size of the dead zone when the subject is not in the predetermined posture.

6. The tracking-related parameter is a dead zone area in which neither the imaging direction of the imaging device nor the zoom of the imaging device is performed, The control device according to claim 1 , wherein the control device controls either the imaging direction of the imaging device or the zoom of the imaging device by invalidating a dead zone when the posture of the subject is a predetermined posture.

7. The tracking parameter is an acceleration when changing the imaging direction, The control device according to claim 1 , wherein the control means controls the acceleration when the imaging direction is changed in accordance with the posture of the subject.

8. The control device according to claim 7 , wherein the magnitude of the acceleration when the subject is in a predetermined posture is greater than the magnitude of the acceleration when the subject is not in the predetermined posture.

9. The control device according to claim 1 , wherein the determining means determines whether the posture of the subject is a predetermined posture based on the direction of a line connecting joint points of the subject or the inclination of an axis passing through the center of the subject with respect to the ground.

10. 2. The control device according to claim 1, wherein the determining means determines whether the posture of the subject is a predetermined posture based on the height from the ground to the center of gravity of the subject.

11. 2. The control device according to claim 1, wherein the determining means determines whether the posture of the subject is a predetermined posture based on whether the orientation of the face or the direction of the line of sight of the subject is a predetermined orientation.

12. The control device according to claim 11 , wherein the predetermined direction is a direction in which the subject's face or line of sight is directed outside an imaging range of the imaging device.

13. A control method performed by a control device, Detecting a subject from an image captured by an imaging device; determining a pose of the subject; setting tracking parameters based on the posture of the subject; a control method for controlling either an imaging direction of the imaging device or a zoom of the imaging device based on the parameter corresponding to the posture of the subject, and causing the imaging device to track and image the subject.

Images