Control device and control method thereof
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- CANON KK
- Filing Date
- 2022-04-26
- Publication Date
- 2026-08-05
Smart Images
Figure 0007900945000001 
Figure 0007900945000002 
Figure 0007900945000003
Abstract
Description
Technical Field
[0001] The present invention relates to tracking control of a shooting target by an imaging device.
Background Art
[0002] In lectures, sports, etc., video distribution and recording are often performed. Also, automatic tracking shooting for the purpose of cost reduction such as labor costs has begun to be put into practical use. In automatic tracking shooting, in the process of controlling the imaging range of the imaging device following the movement of the tracking target, the tracking target may be lost (hereinafter referred to as "lost").
[0003] When a loss occurs during tracking shooting, control is performed such as returning to the position and orientation of the home position or staying at the angle of view at the time when the loss occurred. Also, in Patent Document 1, a configuration for expanding the imaging range around the tracking target disappearance position and continuing the tracking shooting, and a configuration for turning the imaging range to the home position if the disappearance position is at the edge of the screen have been proposed.
Prior Art Documents
Patent Documents
[0006] This invention has been made in view of these problems and aims to provide a technology that enables more advanced tracking control of the target being photographed. [Means for solving the problem]
[0007] To solve the above-mentioned problems, the control device according to the present invention has the following configuration. That is, the control device that controls tracking shooting by the imaging device is A detection means for detecting a target to be tracked from the image obtained by the aforementioned imaging device, A tracking means that tracks the target by controlling the shooting direction of the imaging device based on the detection position of the target to be tracked by the detection means, If the tracking means loses track of the target while the tracking means is tracking the target, the tracking means performs a loss processing. It has, The processing means is A determination means for determining whether the direction of movement of the tracked target immediately before the loss occurs can be determined with a first degree of accuracy or higher, If the determination means determines that the direction of movement can be derived with a first degree of accuracy or higher, a determination means determines the standby shooting direction, which is the shooting direction of the imaging device after the loss occurs, based on the direction of movement. to have death, The determination means derives the movement direction of the tracked target and the accuracy of that movement direction immediately before the loss occurs, based on the control history of the shooting direction in the tracking means. The accuracy is derived such that, in the control history corresponding to a series of consecutive frames immediately before the loss occurs, the direction of movement of the tracked object is the same in the series of frames, and the positional difference between adjacent frames of the tracked object in the series of frames is greater than or equal to a threshold. . [Effects of the Invention]
[0008] According to the present invention, it is possible to provide a technology that enables more advanced tracking control of the target being photographed. [Brief explanation of the drawing]
[0009] [Figure 1] This is a diagram showing the overall configuration of the automatic shooting system according to the first embodiment. [Figure 2] This is a block diagram showing the functional configuration of the tracking device according to the first embodiment. [Figure 3] It is a flowchart of the tracking process in the first embodiment. [Figure 4] It is a detailed flowchart of the lost process in the first embodiment. [Figure 5] It is a diagram for explaining the method of determining the lost direction. [Figure 6] It is a diagram for explaining the method of deriving the standby view angle. [Figure 7] It is a block diagram showing the functional configuration of the tracking device according to the second embodiment. [Figure 8] It is a detailed flowchart of the lost process in the second embodiment. [Figure 9] It is a diagram for explaining the method of determining the lost direction. [Figure 10] It is a diagram for explaining the detection size at the target view angle. [Figure 11] It is a block diagram showing the functional configuration of the tracking device according to the third embodiment. [Figure 12] It is a detailed flowchart of the lost process in the third embodiment.
Mode for Carrying Out the Invention
[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.
[0011] (First Embodiment) As a first embodiment of the tracking device according to the present invention, a tracking device in an automatic photographing system will be described below as an example. In particular, the pan-tilt (PT) control of the photographing range when the tracking target is lost (lost) will be described.
[0012] <System Configuration and Device Configuration> FIG. 1 is a diagram showing the overall configuration of an automatic shooting system A1000 including a tracking device A1100 according to the first embodiment. The tracking device A1100 controls the PT drive device A1102 that drives the video input device A1101 in the PT direction based on the video input from the video input device A1101, which is a shooting device, to control the shooting range of the video input device A1101. Further, the automatic shooting system A1000 includes a client device A1200, which is a user terminal for remotely controlling the tracking device A1100. The tracking device A1100 and the client device A1200 are connected via, for example, a network A1500.
[0013] The tracking device A1100 controls the video input device A1101 and the PT drive device A1102 based on the setting information input from the client device A1200 to perform shooting, and outputs the captured video to the client device A1200. The client device A1200 provides the video input via the network to the user via a video display device such as a monitor. Further, the client device A1200 receives setting information from the user via an input device such as a keyboard or a mouse, and outputs the received setting information to the tracking device A1100 via the network A1500.
[0014] FIG. 2 is a block diagram showing the functional configuration of the tracking device A1100 according to the first embodiment. The tracking device A1100 controls the imaging range and shoots the tracking target based on the video acquired from the video input device A1101 and the setting information acquired from the client device A12Q0.
[0015] The tracking device A1100 includes a setting information input unit A1103, a tracking target detection unit A1104, a tracking processing unit A1105, a lost direction determination unit A1106, a standby picture angle determination unit A1107, an imaging range control unit A1108, and a communication unit A1109.
[0016] The setting information input unit A1103 outputs the setting information received from the client device A1109 to the tracking target detection unit A1104 and the tracking processing unit A1105. Here, the setting information includes information on the target position and target size. The target position is the position coordinate within the frame image, and the tracking device performs PT control so that the tracking target is located at the location of the target position (see Figure 5, which will be described later). In addition, the setting information may also include other information such as the criteria for determining when the tracking target is lost.
[0017] The tracking target detection unit A1104 analyzes the video (frame image) input from the video input device A1101 based on the input setting information. It then detects the tracking target within the frame image and outputs the detection information to the tracking processing unit A1105. Here, it is assumed that the tracking target position (the position coordinates of the tracking target within the frame image) will be output as detection information, but other positional information such as spatial coordinates may also be used as long as it indicates the position of the tracking target. In addition, additional information such as detection size and detection accuracy may also be included as detection results.
[0018] The tracking processing unit A1105 performs tracking of the target based on the input detection information and the target position input from the setting information input unit A1103. Specifically, as part of the tracking process, it stores the position of the target within the frame image as the tracking history. It also determines whether the target has been lost based on the tracking history. Here, it is determined that the target has been lost if detection of the target fails for x consecutive frames with respect to the input video. For simplicity, we will assume x=1 below.
[0019] Then, if the tracking processing unit A1105 determines that no loss has occurred, it outputs the tracking target position and the target position to the imaging range control unit A1108. On the other hand, if it determines that a loss has occurred, it outputs the tracking history to the loss direction determination unit A1106.
[0020] The Lost Direction Determination Unit A1106 derives the direction of loss of the tracked target based on the input tracking history. The lost direction is the direction of the tracked target's movement (vector direction) immediately before the loss occurs. In the first embodiment, it also determines whether the lost direction is clear (= derived with a predetermined degree of accuracy or higher). Specifically, the lost direction is determined to be clear if the tracked target's position relative to the target position is separated by y frames or more consecutively in the same direction immediately before the loss, and the position difference is greater than or equal to a threshold. If the Lost Direction Determination Unit A1106 determines that the derived lost direction is clear, it outputs the lost direction as the determination result; if it determines that the derived lost direction is unclear, it outputs information indicating that it is unclear as the determination result. The Lost Direction Determination Unit A1106 outputs the calculated lost direction and the pre-loss detection position (the tracked target's position immediately before the loss occurs) to the standby field of view determination unit A1107.
[0021] The standby field of view determination unit A1107 calculates the standby field of view (standby position orientation) which is the imaging range with a high probability of re-detection, based on the input loss direction determination and the pre-loss detection position. Specifically, if the input result is a loss direction, it determines that there is a position orientation with a high probability of the tracking target being present, and sets the field of view with PT control toward the loss direction within the range where the pre-loss detection position is not cut off as the standby field of view. On the other hand, if the input result indicates that the determination is unclear, it determines that there is no field of view with a high probability of the tracking target being present, and sets the current field of view as the standby field of view. The standby field of view determination unit A1107 outputs the standby field of view to the imaging range control unit A1108.
[0022] If no loss occurs, the imaging range control unit A1108 continues tracking control based on the tracking target position and target position input from the tracking processing unit A1105. That is, it controls the imaging range by performing PT control so that the tracking target position and target position coincide. On the other hand, if a loss occurs, as a loss processing, it performs PT control based on the standby field of view input from the standby field of view determination unit A1107. Specifically, the imaging range control unit A1108 outputs a PT control command to the PT drive unit A1102 and the video to the communication unit A1109.
[0023] The PT drive unit A1102 performs PT drive based on the input PT control command to move the imaging range. The communication unit A1109 outputs the video input from the imaging range control unit A1108 to the client device A1200. It also outputs the setting information input unit A1103 from the client device A1200.
[0024] <Device Operation> Figure 3 is a flowchart of the tracking process in the first embodiment. The automatic shooting system A1000 starts tracking shooting and video distribution based on, for example, user operation on the client device A1200.
[0025] In S001, the tracking device A1100 begins receiving (inputting) video output from the video input device A1101. In S002, the target detection unit A1104 detects the target to be tracked from the input video and outputs the detection result.
[0026] In S003, the tracking unit A1105 determines whether the tracked target has been lost. If true (YES in S003), it outputs the tracking history and proceeds to S004. If false (NO in S003), it outputs the tracked target's position and proceeds to S005.
[0027] In S004, the tracking device A1100 performs a lost target process based on the tracking history (the position of the tracked target before it was lost). Details of the lost target process will be described later with reference to Figure 4. Meanwhile, in S005, the tracking processing unit A1105 updates the tracking history based on the position of the tracked target. It also outputs the position of the tracked target and the target position.
[0028] In S006, if no loss has occurred (proceeding from S005), the imaging range control unit A1108 controls the imaging range by performing PT control so that the input tracking target position and the target position coincide. Here, it controls the imaging range so that the tracking target is at the center of the frame image. On the other hand, if a loss has occurred (proceeding from S004), it controls the imaging range to the input standby field of view. In this case, based on the PT speed at the time of loss or immediately before loss and the standby field of view, it may calculate and control the PT speed control so that it smoothly transitions to the standby field of view.
[0029] In S007, the tracking device A1100 determines whether a user operation to stop tracking shooting has been performed on the client device A1200. If false (NO in S007), proceed to S001 and continue tracking shooting. On the other hand, if true (YES in S007), terminate tracking shooting.
[0030] Figure 4 is a detailed flowchart of the lost tracking process (S004) in the first embodiment. As described above, the tracking device A1100 performs the lost tracking process based on the tracking history.
[0031] In S101, the lost direction determination unit A1106 calculates the lost direction of the tracking target based on the input tracking history. In the first embodiment, the lost direction corresponds to the direction within the image plane of the captured image (up, down, left, right, etc.) and is associated with the pan-tilt (PT) coordinate values.
[0032] In S102, the lost direction determination unit A1106 determines whether the lost direction calculated in S101 is clear (= derived with a predetermined degree of accuracy or higher). If true (YES in S102), it outputs the calculated lost direction and the pre-loss detection position and proceeds to S103; if false (NO in S102), it proceeds to S104.
[0033] Figure 5 illustrates the method for determining the direction of loss (S101, S102). The lower part of Figure 5 shows the frame image immediately before loss (i.e., the frame in which the tracking target was last detected). The upper part shows the frame image from y frames earlier (past).
[0034] In the frame image immediately before the loss, point P001 indicates the target position within the frame image (in this case, the center of the frame image). Point P002 indicates the tracking target position within the frame image. In the frame image from the y-frame before, point P003 indicates the position of point P002. Also, point P004 indicates the tracking target position detected immediately before the loss.
[0035] As described above, in the first embodiment, the direction of loss is determined to be clear if the tracking target position is separated from the target position by y frames or more consecutively in the same direction immediately before the loss, and the position difference is greater than or equal to a threshold k. Here, y=2. Note that the direction of loss may be determined by other methods.
[0036] In the frame image shown in the lower part of Figure 5, the movement away from the target position occurs for two consecutive frames immediately before the loss, and the difference (k+α) is greater than the threshold k. Therefore, in this example of frame image, the direction of loss is clear and determined to be "left".
[0037] In S103, the standby field of view determination unit A1107 calculates the standby field of view based on the input loss direction and the pre-loss detection position. Specifically, it calculates the field of view in which the tracking target is likely to reappear as the standby field of view. As described above, in the first embodiment, the standby field of view is the field of view in which the PT control is performed in the loss direction (in this case, "left") within a range where the pre-loss detection position is not cut off.
[0038] Figure 6 illustrates the method for deriving the standby field of view. The upper part of Figure 6 shows an example of the field of view (shooting range) P014 at the time of the last detection before loss determination, and the lower part of Figure 6 shows an example of the calculated standby field of view P016.
[0039] Position P011 indicates the geographical location of the video input device A1101, which is the imaging device; position P012 indicates the geographical location of the tracked object immediately before it was lost; and arrow P013 indicates the geographical direction of the loss.
[0040] In situation P010, if a target is lost immediately after detection at position P012, the standby field of view P016 is defined as the field of view where position P012 is at the opposite end (in this case, the right end) of the imaging range from the direction of target loss. Here, the standby field of view is defined as the field of view obtained by PT control in the direction of loss within the range where position P012 is not cut off, but the standby field of view may also be defined as the field of view where position P012 is cut off.
[0041] Furthermore, a predetermined initial direction (initial position / attitude), which is the home position, may be set as the standby field of view at any of the following times: immediately after a loss is detected, after receiving instructions from the user, or after the loss state has continued for a predetermined period of time. If the calculated standby field of view is outside the driveable range of the PT drive unit A1102 or outside the trackable imaging area, the home position may be set as the standby field of view.
[0042] Furthermore, if the direction of loss is the vertical direction of the frame image, the calculated standby field of view may be corrected according to the position of the tracked target acquired up to that point (or set by the user). For example, it may be corrected so that it does not exceed the position of a person's head or fall below their feet.
[0043] In S104, the standby field of view determination unit A1107 sets the current field of view information as the standby field of view. In other words, it determines that it cannot determine the field of view in which the target being tracked is most likely to be located, and therefore decides not to change the field of view by PT control.
[0044] If the standby field of view is determined by the standby field of view determination unit A1107 in S103 or S104 as described above, the loss process ends and the process proceeds to S006.
[0045] As explained above, according to the first embodiment, if the tracked target is lost during tracking control, the actions performed are changed depending on whether or not the direction of the lost target is clear. Specifically, if the direction of the lost target is clear, PT control is performed to the field of view where the tracked target is most likely to be located, based on the tracking history before the loss occurred. By performing such control, it becomes easier to re-detect the tracked target and resume tracking control after the loss has occurred.
[0046] (Second Embodiment) In the second embodiment, control when the tracked object moves in the depth direction of the captured image will be described. That is, in the first embodiment, PT control of the shooting range when the tracked object is lost was described, but in the second embodiment, zoom (Z) control of the shooting range when the tracked object is lost will be described.
[0047] <System Configuration and Equipment Configuration> Regarding the system configuration, in Figure 1, the only difference is that the tracking device A1100 and PT drive device A1102 are replaced by the tracking device A2100 and PTZ drive device A2102, respectively. The PTZ drive device A2102 controls the shooting range of the video input device A1101 by controlling the zoom (Z) in addition to driving the PT direction of the video input device A1101 based on the control by the tracking device A2100. The other devices are the same as in the first embodiment (Figure 1), so a detailed explanation is omitted. For the sake of simplicity, in the following explanation, the PTZ drive device A2102 will be described as only performing zoom (Z) driving.
[0048] Figure 7 is a block diagram showing the functional configuration of the tracking device A2100 according to the second embodiment. Note that the functional parts with the same reference numerals as in the first embodiment (Figure 2) have the same functions and therefore their explanation is omitted.
[0049] The tracking device A2100 includes a setting information input unit A1103, a tracking target detection unit A2104, a tracking processing unit A2105, a lost direction determination unit A2106, a standby field of view determination unit A2107, an imaging range control unit A1108, and a communication unit A1109.
[0050] The PTZ drive unit A2102 performs PTZ driving based on the input PTZ control command to move the imaging range.
[0051] The tracking target detection unit A2104 analyzes the video (frame image) input from the video input device A1101 based on the input setting information. It then detects the tracking target within the frame image and outputs the detection information to the tracking processing unit A2105. Here, it is assumed that the detection information will include the tracking target position (the position coordinates of the tracking target within the frame image) and the tracking target size (the size of the tracking target within the frame image). However, other position information and size information that reveals the position and size of the tracking target may also be used. In addition, additional information such as the detected size and detection accuracy may be included as detection results.
[0052] The tracking processing unit A2105 performs tracking of the target based on the input detection information and the target position input from the setting information input unit A1103. Specifically, as part of the tracking process, it stores the target position and target size as tracking history. It also determines whether the target has been lost based on the tracking history. Here, it is determined that the target has been lost if detection of the target fails for x consecutive frames with respect to the input video. For simplicity, we will assume x=1 below.
[0053] Then, if the tracking processing unit A2105 determines that no loss has occurred, it outputs the tracking target position, tracking target size, and target position to the imaging range control unit A1108. On the other hand, if it determines that a loss has occurred, it outputs the tracking history to the loss direction determination unit A2106.
[0054] The Lost Direction Determination Unit A2106 derives the direction of loss of the tracked target based on the input tracking history. As described above, in the second embodiment, the lost direction is the direction of movement of the tracked target along the depth direction of the captured image immediately before the loss occurs. In addition, in the second embodiment, it is also determined whether the lost direction is clear (= derived with a predetermined degree of accuracy or higher). Specifically, if the size of the tracked target at the target position changes significantly (approaching) or significantly (moving away) for y frames or more consecutively immediately before the loss, it is determined that the lost direction is clear. If the Lost Direction Determination Unit A2106 determines that the derived lost direction is clear, it outputs the lost direction as the determination result, and if it determines that the derived lost direction is unclear, it outputs information indicating that it is unclear as the determination result. The Lost Direction Determination Unit A2106 outputs the calculated lost direction and the pre-loss detection size (the size of the tracked target immediately before the loss occurs) to the standby field of view determination unit A2107.
[0055] The standby field of view determination unit A2107 calculates the standby field of view (zoom (Z) value) which is the imaging range with a high probability of re-detection, based on the input loss direction determination and the detection size before loss. Specifically, if the loss direction is input as the determination result, it determines that there is a zoom value with a high probability of the tracked target being present and calculates the standby field of view. For example, the standby field of view is set to the zoom value at which the size of the tracked target immediately before loss is a detectable size (greater than or equal to the minimum detection size and less than or equal to the maximum detection size). In this case, if the loss direction is "away from the camera," it is best to use the maximum detection size, and if it is "approaching the camera," it is best to use the minimum size. For a detailed explanation of how to calculate the standby field of view, please refer to Figures 8 to 10 below.
[0056] On the other hand, if information indicating that the determination result is unclear is input, it is determined that there is no zoom value where the tracked target is likely to be present, and the current zoom value is set as the standby field of view. The standby field of view determination unit A2107 outputs the standby field of view to the imaging range control unit A1108.
[0057] <Device Operation> The overall tracking process is the same as in the first embodiment (Figure 3), so a detailed explanation will be omitted. Below, the details of the lost process (S004) in the second embodiment will be explained with reference to Figure 8.
[0058] Figure 8 is a detailed flowchart of the lost tracking process (S004) in the second embodiment. As described above, the tracking device A2100 performs the lost tracking process based on the tracking history.
[0059] In S201, the lost direction determination unit A2106 calculates the lost direction of the tracked object based on the input tracking history. However, unlike the first embodiment, here the lost direction is calculated as the zoom value corresponding to the depth direction as seen from the tracking device.
[0060] In S202, the lost direction determination unit A2106 determines whether the lost direction calculated in S201 is clear (i.e., derived with a predetermined degree of accuracy or higher). If true (YES in S202), it outputs the calculated lost direction and the pre-loss detection size and proceeds to S203; if false (NO in S202), it proceeds to S206. The determination method will be explained in detail with reference to Figures 9 and 10.
[0061] Figure 9 is a diagram illustrating the method for determining the direction of loss. Frame images P201 to P203 represent three frames in chronological order that exemplify a situation in which loss occurs when the tracked target is moving away from it.
[0062] The top panel of Figure 9 shows frame image P201 from the y-frame before (past) in which the loss occurred. The middle panel shows frame image P202 from immediately before the loss occurred (i.e., the last frame in which the tracked target was detected). The bottom panel shows frame image P203 from immediately after the loss occurred (i.e., the first frame in which the tracked target was lost). P205 shows the tracked target, and P206 / P207 shows the detection frame of the tracked target (a rectangle circumscribing the tracked target).
[0063] Figure 10 illustrates the detection size within the target field of view (shooting range). In Figure 10, P212 indicates the target size of the detection frame in the frame image P211 of the target field of view (shooting range). P213 represents the maximum size of the detection frame (maximum detection size), and P214 represents the minimum size of the detection frame (minimum detection size). Tracking targets with a detection frame larger than the maximum detection size and tracking targets with a detection frame smaller than the minimum detection size will result in loss of focus.
[0064] As described above, in the second embodiment, the direction of loss is determined to be clear if the size of the tracked object at the target position changes significantly (approaching) or significantly (moving away) for y frames or more consecutively immediately before loss. Here, y=1, and it is determined to be clear if the change is greater than or equal to a threshold of m%. In other words, in the situation shown in Figure 9, it is determined to be clear if P207 is less than or equal to (100-m)% of the size of P206. Note that the direction of loss may also be determined by other methods, such as using the distance between the tracked pair or the history of zoom values.
[0065] In the situation shown in Figure 9, although a zoom-in command is issued in frame image P201 because the detection frame is small relative to the target field of view, the detection frame is even smaller relative to the target field of view in frame image P202. Therefore, a further zoom-in command is issued, but the tracked target cannot be detected in frame image P203 and is lost. Therefore, it is determined that the direction of loss is "away from the target". Here, since the difference in the size of the detection frame is greater than the threshold m%, it is determined that the direction of loss is clear.
[0066] In S203, the lost direction determination unit A2106 determines whether the lost direction calculated in S202 is the "away from" direction. If it is the "away from" direction, proceed to S204; otherwise, proceed to S205. In S204, the standby field of view determination unit A2107 sets the target size to the maximum detection size. On the other hand, in S205, the standby field of view determination unit A2107 sets the target size to the minimum detection size.
[0067] In S206, the standby field of view determination unit A2107 sets the current zoom value information as the standby field of view, terminates the loss processing, and proceeds to S006. In other words, it determines that it cannot determine a specific zoom value in which the tracked target is likely to be detected, and decides not to change the field of view by zoom control.
[0068] In S207, the standby field of view determination unit A2107 calculates a zoom value based on the zoom value immediately before loss and the target size, sets it as the standby field of view, terminates the loss process, and proceeds to S006. For example, it calculates a zoom value such that the detected size immediately before loss becomes the maximum detected size.
[0069] As described above, according to the second embodiment, if the tracked target is lost during tracking control, the operation performed is changed depending on whether the direction of loss of the tracked target is clear or not, similar to the first embodiment. Specifically, if the direction of loss is clear, zoom control is performed to a zoom value that is highly likely to detect the tracked target based on the tracking history before the loss occurred. By performing such control, it becomes easier to re-detect the tracked target and perform tracking control again after a loss occurs.
[0070] (Third embodiment) In the third embodiment, a control method capable of handling situations where the tracked object moves both in the direction along the plane of the captured image and in the depth direction will be described. Specifically, in the third embodiment, a configuration will be described in which the shooting range is controlled by combining pan, tilt, and zoom (PTZ) when the tracked object is lost.
[0071] <System Configuration and Equipment Configuration> Regarding the system configuration, since the only differences in Figure 1 are that the tracking device A1100 and PT drive device A1102 have been replaced by the tracking device A3100 and PTZ drive device A2102, respectively, a detailed explanation will be omitted.
[0072] Figure 11 is a block diagram showing the functional configuration of the tracking device A3100 according to the third embodiment. Note that the functional parts with the same reference numerals as in the second embodiment (Figure 7) have the same functions and therefore their explanation is omitted.
[0073] The tracking device A3100 includes a setting information input unit A1103, a tracking target detection unit A2104, a tracking processing unit A3105, a lost direction determination unit A3106, a standby field of view determination unit A3107, an imaging range control unit A3108, and a communication unit A1109.
[0074] The tracking processing unit A3105 performs tracking of the target based on the input detection information and the target position input from the setting information input unit A1103. Specifically, as part of the tracking process, it stores the target position and target size as tracking history. It also determines whether the target has been lost based on the tracking history. Here, it is determined that the target has been lost if detection of the target fails for x consecutive frames with respect to the input video. For simplicity, we will assume x=1 below.
[0075] Then, if the tracking processing unit A3105 determines that no loss has occurred, it outputs the tracking target position, tracking target size, and target position to the imaging range control unit A3108. On the other hand, if it determines that a loss has occurred, it outputs the tracking history to the loss direction determination unit A3106.
[0076] The lost direction determination unit A3106 derives the lost direction of the tracked object based on the input tracking history. In the third embodiment, the determination is made for both the PT direction and the zoom direction. In the third embodiment as well, it is determined whether or not the lost direction is clear for both the PT direction and the zoom direction.
[0077] Specifically, the loss direction determination unit A3106 determines that the loss direction (PT direction) is clear if, with respect to the PT direction, the tracking target position is separated from the target position by y frames or more consecutively in the same direction immediately before the loss, and the position difference is greater than or equal to a threshold. Furthermore, with respect to the zoom direction, it determines that the loss direction (zoom direction) is clear if the size of the tracking target at the target position changes significantly (closer) or significantly (further away) for y frames or more consecutively immediately before the loss.
[0078] The Lost Direction Determination Unit A3106 outputs the Lost Direction as a determination result if it determines that the derived Lost Direction is clear for both the PT direction and the zoom direction. If it determines that the derived Lost Direction is unclear, it outputs information indicating that it is unclear as a determination result. The Lost Direction Determination Unit A3106 outputs the Lost Direction calculated for both the PT direction and the zoom direction, along with pre-loss detection information, to the Standby Field of View Determination Unit A3107. The pre-loss detection information includes the tracking target position and tracking target size immediately before the loss occurs.
[0079] The standby field of view determination unit A3107 acquires the pre-loss PTZ control command information (one or more PTZ control commands performed before loss) held by the imaging range control unit A3108. Based on the acquired pre-loss PTZ control command information, the input loss direction determination, and the pre-loss detection information, the standby field of view determination unit A3107 calculates the standby field of view (PT value and zoom value) which is the imaging range in which re-detection is highly likely. Specifically, if a loss direction is input as the determination result, it is determined that there are PT values and zoom values in which the tracking target is likely to be present, and the standby field of view is calculated. A detailed method for calculating the standby field of view will be described later with reference to Figure 12.
[0080] On the other hand, if information indicating that the determination result is unclear is input, it is determined that there are no PT values and zoom values that are likely to contain a target being tracked, and the current PT value and zoom value are set as the standby field of view. The standby field of view determination unit A3107 outputs the standby field of view to the imaging range control unit A3108.
[0081] If no loss occurs, the imaging range control unit A3108 continues tracking control based on the tracking target position and target position input from the tracking processing unit A3105. Furthermore, it retains the most recent n frames of PTZ control command information as history information. For simplicity, we assume n=1 here and retain only the most recent PTZ control command. On the other hand, if a loss occurs, as a loss processing step, it outputs the history of PTZ control commands to the standby field of view determination unit A3107. Furthermore, it performs PTZ control based on the standby field of view input from the standby field of view determination unit A3107. Specifically, the imaging range control unit A3108 outputs the PTZ control command to the PTZ drive device A2102 and the video to the communication unit A1109.
[0082] <Device Operation> The overall tracking process is almost the same as in the first embodiment (Figure 3), but in S007, the imaging range control unit A3108 holds the PTZ control command as history information. Below, the details of the lost data processing (S004) in the third embodiment will be explained with reference to Figure 12.
[0083] Figure 12 is a detailed flowchart of the lost tracking process (S004) in the third embodiment. As described above, the tracking device A3100 performs the lost tracking process based on the tracking history.
[0084] In S301, the lost direction determination unit A3106 calculates the lost direction of the tracked object based on the input tracking history. As described above, the lost direction is calculated as the PT direction and the zoom direction.
[0085] In S303, the standby field of view determination unit A3107 acquires the PT direction control amount from the pre-loss PTZ control command information held by the imaging range control unit A3108. In S304, the standby field of view determination unit A3107 uses the acquired PT direction control amount and a predetermined coefficient α to calculate the field of view (PT direction) that has a high probability of being re-detected. Specifically, it calculates a PT control command with a control amount obtained by multiplying the PT direction control amount by a predetermined coefficient α.
[0086] Here, a fixed value α is assumed as the predetermined coefficient, but the coefficient may be changed depending on the amount of PT direction movement of the tracked target immediately before loss, for example, by defining a table. Alternatively, the average or maximum value of the PT direction control amount for n frames before loss may be used as the PT direction control amount before loss.
[0087] In S307, the standby field of view determination unit A3107 acquires the zoom direction control amount from the pre-loss PTZ control command information held by the imaging range control unit A3108. In S308, the standby field of view determination unit A3107 uses the acquired zoom direction control amount and a predetermined coefficient β to calculate the field of view (zoom direction) that has a high probability of being re-detected. Specifically, it calculates a zoom control command for a control amount obtained by multiplying the zoom direction control amount by a predetermined coefficient β.
[0088] Here, a fixed value β is assumed as the predetermined coefficient, but the coefficient may be changed depending on the amount of zoom direction movement of the tracked target (or the amount of change in the size of the tracked target) immediately before loss, for example, by defining a table. Alternatively, the average or maximum value of the zoom direction control amount for the n frames before loss may be used as the zoom direction control amount before loss.
[0089] As explained above, according to the third embodiment, if the tracked target is lost during tracking control, the actions performed are changed depending on whether the direction of loss of the tracked target is clear or not, similar to the first and second embodiments. In particular, control commands are calculated for both the PT direction and the zoom direction to set the field of view to one that has a high probability of being re-detected. By performing such control, it becomes easier to re-detect the tracked target and resume tracking control after a loss has occurred.
[0090] The disclosures herein include the following control devices, control methods, and programs. (Item 1) A control device for controlling tracking photography by an imaging device, A detection means for detecting a target to be tracked from the image obtained by the aforementioned imaging device, A tracking means that tracks the target by controlling the shooting direction of the imaging device based on the detection position of the target to be tracked by the detection means, If the tracking means loses track of the target while the tracking means is tracking the target, the tracking means performs a loss processing. It has, The processing means is A determination means for determining whether the direction of movement of the tracked target immediately before the loss occurs can be determined with a first degree of accuracy or higher, If the determination means determines that the direction of movement can be derived with a first degree of accuracy or higher, a determination means determines the standby shooting direction, which is the shooting direction of the imaging device after the loss occurs, based on the direction of movement. A control device characterized by having the following features. (Item 2) The determination means derives the movement direction of the tracked target and the accuracy of that movement direction immediately before the loss occurs, based on the control history of the shooting direction in the tracking means. The control device according to item 1, characterized in that it is a control device. (Item 3) If the determination means determines that the direction of movement cannot be derived with a first degree of accuracy or higher, the determination means determines the current shooting direction of the imaging device or a predetermined initial direction as the standby shooting direction. A control device according to item 1 or 2, characterized in that it is a control device. (Item 4) The control device is configured to perform tracking imaging of the imaging device using pan-tilt (PT) control. If the determination means determines that the direction of movement can be derived with a first degree of accuracy or higher, the determination means PT controls the current shooting direction of the imaging device to the direction of movement within a range where the position of the tracking target immediately before the loss occurs is not obscured, and determines this as the standby shooting direction. A control device according to any one of items 1 to 3, characterized in that (Item 5) The tracking means controls the shooting direction of the imaging device to the standby shooting direction, and if the loss of the tracking target by the detection means continues for a predetermined time, it controls the shooting direction of the imaging device to a predetermined initial direction. A control device according to any one of items 1 to 4, characterized in that (Item 6) The determination means determines the current shooting direction of the imaging device or a predetermined initial direction as the standby shooting direction if the standby shooting direction determined based on the movement direction is outside the control range of the shooting direction of the imaging device. A control device according to any one of items 1 to 5, characterized in that (Item 7) The control device is configured to perform tracking photography of the imaging device by pan-tilt-zoom (PTZ) control. The tracking means further controls the imaging angle of the imaging device based on the detection size of the target being tracked by the detection means. The determination means further determines whether it is possible to derive a second direction of movement in the depth direction of the image of the tracked object immediately before the occurrence of the loss with a second degree of accuracy or greater, If the determination means determines that the second direction of movement can be derived with a second degree of accuracy or higher, the determination means further determines the standby shooting angle, which is the shooting angle of the imaging device after the loss occurs, based on the second direction of movement. A control device according to any one of items 1 to 6, characterized in that (Item 8) The determination means calculates a control amount obtained by multiplying the PT direction control amount by the tracking means immediately before the loss occurs by a first coefficient as a control amount for controlling to the standby shooting direction, and calculates a control amount obtained by multiplying the zoom direction control amount by the tracking means immediately before the loss occurs by a second coefficient as a control amount for controlling to the standby shooting angle of view. The control device according to item 7, characterized in that it is a control device. (Item 9) A control device for controlling tracking photography by an imaging device, A detection means for detecting a target to be tracked from the image obtained by the aforementioned imaging device, A tracking means that tracks the target by controlling the shooting angle of the imaging device based on the detection size of the target to be tracked by the detection means, If the tracking means loses track of the target while the tracking means is tracking the target, the tracking means performs a loss processing. It has, The processing means is A determination means for determining whether the direction of movement in the depth direction in the video of the tracked target immediately before the occurrence of the loss can be determined with a predetermined accuracy or greater, If the determination means determines that the direction of movement can be derived with an accuracy of the predetermined or higher, the determination means determines the standby shooting angle, which is the shooting angle of the imaging device after the loss occurs, based on the direction of movement. A control device characterized by having the following features. (Item 10) A control method for a control device that controls tracking photography by an imaging device, A tracking step in which a target to be tracked is detected from the image obtained by the aforementioned imaging device, and the shooting direction of the imaging device is controlled based on the detected position of the target to track the target, If the tracking target is lost during the tracking process described above, a processing step is performed to execute a loss processing. Includes, The aforementioned processing step is: A determination step of determining whether the direction of movement of the tracked target immediately before the occurrence of the loss can be determined with a predetermined accuracy or higher, If the determination step determines that the direction of movement can be derived with an accuracy of the predetermined value or higher, a determination step is made to determine the standby shooting direction, which is the shooting direction of the imaging device after the loss occurs, based on the direction of movement. A control method characterized by including (Item 11) A control method for a control device that controls tracking photography by an imaging device, A tracking step in which a target to be tracked is detected from the image obtained by the aforementioned imaging device, and the imaging device's field of view is controlled based on the detected size of the target to track the target, If the tracking target is lost during the tracking process described above, a processing step is performed to execute a loss processing. Includes, The aforementioned processing step is: A determination step of determining whether the direction of movement in the depth direction in the video of the tracked object immediately before the loss occurs can be determined with a predetermined accuracy or higher, If the determination step determines that the direction of movement can be derived with an accuracy of at least the predetermined accuracy, a determination step is made to determine the standby shooting angle, which is the shooting angle of the imaging device after the loss occurs, based on the direction of movement. A control method characterized by including (Item 12) A program to enable a computer that controls tracking photography by an imaging device to function as a control device as described in any one of items 1 through 9.
[0091] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.
[0092] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]
[0093] A1100 Tracking device; A1200 Client device; A1101 Video input device; A1102 PT drive device; A1103 Setting information input unit; A1104 Tracking target detection unit; A1105 Tracking processing unit; A1106 Lost direction determination unit; A1107 Standby field of view determination unit; A1108 Imaging range control unit; A1109 Communication unit
Claims
1. A control device for controlling tracking photography by an imaging device, A detection means for detecting a target to be tracked from the image obtained by the aforementioned imaging device, A tracking means that tracks the target by controlling the shooting direction of the imaging device based on the detection position of the target to be tracked by the detection means, If the tracking means loses track of the target while the tracking means is tracking the target, the tracking means performs a loss processing. It has, The processing means is A determination means for determining whether the direction of movement of the tracked target immediately before the loss occurs can be determined with a first degree of accuracy or higher, If the determination means determines that the direction of movement can be derived with a first degree of accuracy or higher, a determination means determines the standby shooting direction, which is the shooting direction of the imaging device after the loss occurs, based on the direction of movement. It has, The determination means derives the movement direction of the tracked target and the accuracy of that movement direction immediately before the loss occurs, based on the control history of the shooting direction in the tracking means. The accuracy is derived such that, in the control history corresponding to a series of consecutive frames immediately before the loss occurs, the direction of movement of the tracked object is the same in the series of frames, and the positional difference between adjacent frames of the tracked object in the series of frames is greater than or equal to a threshold. A control device characterized by the following features.
2. If the determination means determines that the direction of movement cannot be derived with a first degree of accuracy or higher, the determination means determines the current shooting direction of the imaging device or a predetermined initial direction as the standby shooting direction. The control device according to feature 1.
3. The control device is configured to perform tracking imaging of the imaging device by pan-tilt (PT) control. If the determination means determines that the direction of movement can be derived with a first degree of accuracy or higher, the determination means PT controls the current shooting direction of the imaging device to the direction of movement within a range where the position of the tracking target immediately before the loss occurs is not obscured, and determines this as the standby shooting direction. The control device according to feature 1.
4. The tracking means controls the shooting direction of the imaging device to the standby shooting direction, and if the loss of the tracking target by the detection means continues for a predetermined time, it controls the shooting direction of the imaging device to a predetermined initial direction. The control device according to feature 1.
5. The determination means determines the current shooting direction of the imaging device or a predetermined initial direction as the standby shooting direction if the standby shooting direction determined based on the movement direction is outside the control range of the shooting direction of the imaging device. The control device according to feature 1.
6. The control device is configured to perform tracking photography of the imaging device by pan-tilt-zoom (PTZ) control. The tracking means further controls the imaging angle of the imaging device based on the detection size of the target being tracked by the detection means. The determination means further determines whether it is possible to derive a second direction of movement in the depth direction of the image of the tracked object immediately before the loss occurs with a second degree of accuracy or greater, If the determination means determines that the second direction of movement can be derived with a second degree of accuracy or higher, the determination means further determines the standby shooting angle, which is the shooting angle of the imaging device after the loss occurs, based on the second direction of movement. The control device according to feature 1.
7. The determination means calculates a control amount obtained by multiplying the PT direction control amount by the tracking means immediately before the loss occurs by a first coefficient as a control amount for controlling to the standby shooting direction, and calculates a control amount obtained by multiplying the zoom direction control amount by the tracking means immediately before the loss occurs by a second coefficient as a control amount for controlling to the standby shooting angle of view. The control device according to claim 6.
8. A control device for controlling tracking photography by an imaging device, A detection means for detecting a target to be tracked from the image obtained by the aforementioned imaging device, A tracking means that tracks the target by controlling the shooting angle of the imaging device based on the detection size of the target to be tracked by the detection means, If the tracking means loses track of the target while the tracking means is tracking the target, the tracking means performs a loss processing. It has, The processing means is A determination means for determining whether the direction of movement in the depth direction in the video of the tracked target immediately before the occurrence of the loss can be determined with a predetermined accuracy or greater, If the determination means determines that the direction of movement can be derived with an accuracy of the predetermined or higher, the determination means determines the standby shooting angle, which is the shooting angle of the imaging device after the loss occurs, based on the direction of movement. It has, The determination means derives the movement direction of the tracked target and the accuracy of that movement direction immediately before the loss occurs, based on the control history of the shooting direction in the tracking means. The accuracy is derived such that, in the control history corresponding to a series of consecutive frames immediately before the loss occurs, the direction of movement of the tracked object is the same in the series of frames, and the positional difference between adjacent frames of the tracked object in the series of frames is greater than or equal to a threshold. A control device characterized by the following features.
9. A control method for a control device that controls tracking photography by an imaging device, A tracking step in which a target to be tracked is detected from the image obtained by the aforementioned imaging device, and the shooting direction of the imaging device is controlled based on the detected position of the target to track the target, If the tracking target is lost during the tracking process described above, a processing step is performed to execute a loss processing. Includes, The aforementioned processing step is: A determination step of determining whether the direction of movement of the tracked target immediately before the occurrence of the loss can be determined with a predetermined accuracy or higher, If the determination step determines that the direction of movement can be derived with an accuracy of the predetermined value or higher, a determination step is made to determine the standby shooting direction, which is the shooting direction of the imaging device after the loss occurs, based on the direction of movement. Includes, In the determination step, based on the control history of the shooting direction in the tracking step, the movement direction of the tracked target immediately before the loss occurs and the accuracy of that movement direction are derived. The accuracy is derived such that, in the control history corresponding to a series of consecutive frames immediately before the loss occurs, the direction of movement of the tracked object is the same in the series of frames, and the positional difference between adjacent frames of the tracked object in the series of frames is greater than or equal to a threshold. A control method characterized by the following:
10. A control method for a control device that controls tracking photography by an imaging device, A tracking step in which a target to be tracked is detected from the image obtained by the aforementioned imaging device, and the imaging device's field of view is controlled based on the detected size of the target to track the target, If the tracking target is lost during the tracking process described above, a processing step is performed to execute a loss processing. Includes, The aforementioned processing step is: A determination step of determining whether the direction of movement in the depth direction in the video of the tracked object immediately before the loss occurs can be determined with a predetermined accuracy or higher, If the determination step determines that the direction of movement can be derived with an accuracy of at least the predetermined accuracy, a determination step is made to determine the standby shooting angle, which is the shooting angle of the imaging device after the loss occurs, based on the direction of movement. Includes, In the determination step, based on the control history of the shooting direction in the tracking step, the movement direction of the tracked target immediately before the loss occurs and the accuracy of that movement direction are derived. The accuracy is derived such that, in the control history corresponding to a series of consecutive frames immediately before the loss occurs, the direction of movement of the tracked object is the same in the series of frames, and the positional difference between adjacent frames of the tracked object in the series of frames is greater than or equal to a threshold. A control method characterized by the following:
11. A program for causing a computer that controls tracking photography by an imaging device to function as a control device according to any one of claims 1 to 8.