Photographing control device, photographing control method, and program

Abstract

To suppress occurrence of false tracking.SOLUTION: A photography control unit detects an object from an image picked up by an imaging apparatus, adjusts an angle of view of the imaging apparatus to track the object so that the object falls within the angle of view, and acquires related information on the object detected from the image. When receiving operation input from a user, the photography control unit determines that the object that the detection means has detected from the image is not an object to be tracked and holds the related information on the determined object. On the basis of, a result of comparison between the related information held and related information acquired after the related information is held, the photography control unit excludes an object corresponding to the related information acquired after the related information is held from the object to be tracked.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control technique for tracking and photographing an object. [Background technology]

[0002] When automatically capturing moving scenes such as sports scenes, there is a need to track specific objects such as players or balls so that they remain visible on the screen. A known technology for tracking a specific object involves selecting the object to be tracked, estimating the object's direction of movement between frames of a video based on the object's detection results, and associating the object between frames. However, depending on the presence of multiple objects or the lighting environment, the tracking target may be lost. Furthermore, with conventional tracking technologies, once a tracking target is selected, it is difficult to automatically change the tracking target. For this reason, for example, if the tracking target is lost, it is difficult to automatically select and track the next tracking target. In response to this issue, Patent Document 1 discloses a technology that enables tracking of a target by switching to manual operation when the tracking target is lost during automatic tracking, finding the desired target, and then switching back to automatic capture once the desired target is recaptured. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2004-519952 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the method disclosed in Patent Document 1, it is possible to switch to manual operation and then switch to automatic shooting after finding a target, but the tracking target captured in automatic shooting after switching may not be the tracking target intended by the user. In other words, with the method of Patent Document 1, the wrong object may be captured again as the tracking target and erroneously tracked, and the object intended by the user may not be tracked.

[0005] Therefore, an object of the present invention is to make it possible to suppress the occurrence of erroneous tracking. [Means for solving the problem]

[0006] The photographing control device of the present invention includes a detection unit that detects an object from an image captured by a PTZ-driveable imaging device, a tracking unit that adjusts the angle of view of the imaging device to track the object so that the object is within the angle of view of the imaging device, and a tracking unit that adjusts the angle of view of the object detected from the image. Feature quantity of an acquisition means for acquiring a PTZ operation input from a user; an acceptance means for accepting a PTZ operation input from the user; and an acceptance means for accepting a PTZ operation input from the user before the PTZ operation input. The features were acquired a determining means for determining that an object is not a target of tracking; and an object determined by the determining means to be not a target of tracking. The feature quantity of Objects not being tracked Feature quantity of a holding means for holding the object as the object; and the object held by the holding means. Features an exclusion means for excluding an object corresponding to the object from the tracking target; the exclusion means compares a first feature amount, which is a feature amount of an object that is not a tracking target and is stored by the storage means and is acquired before the PTZ operation input, with a second feature amount of an object that is acquired after the PTZ operation input, and if the first feature amount and the second feature amount are not similar, determines that the object corresponding to the second feature amount is the tracking target, and if the first feature amount and the second feature amount are similar, excludes the object corresponding to the second feature amount from the tracking target. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to suppress the occurrence of erroneous tracking. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a configuration diagram of a tracking system according to a first embodiment. [Figure 2] 10 is a flowchart illustrating a procedure for a view angle adjustment process according to the first embodiment. [Figure 3] FIG. 2 is an explanatory diagram of a detected object. [Figure 4] FIG. 10 is an explanatory diagram of an example of holding feature amounts of non-target objects; [Figure 5] FIG. 10 is an explanatory diagram of an example of excluding non-target objects. [Figure 6] FIG. 10 is an explanatory diagram of an example of selecting a target object. [Figure 7] FIG. 10 is a configuration diagram of a tracking system according to a second embodiment. [Figure 8] 10 is a flowchart showing the procedure of a view angle adjustment process according to a second embodiment. [Figure 9] FIG. 2 is an explanatory diagram of an image coordinate system. [Figure 10] FIG. 2 is an explanatory diagram of a PTZ coordinate system. [Figure 11] FIG. 1 is an explanatory diagram of a bird's-eye coordinate system. [Figure 12] FIG. 10 is a configuration diagram of a tracking system according to a third embodiment. [Figure 13] 10 is a flowchart showing the procedure of a field angle adjustment process according to a third embodiment. [Figure 14] FIG. 10 is an explanatory diagram of an example of presentation of a cut-out image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. Note that the present invention is not limited to the following embodiments. In the following description, the same components, processes, etc. will be denoted by the same reference numerals and will not be described again. First Embodiment Fig. 1 is a block diagram showing an example of the functional configuration of a tracking system A100 according to a first embodiment, which includes a field-of-view adjustment device A103 as an application example of an imaging control device according to the present invention. Fig. 2 is a flowchart showing the flow of a field-of-view adjustment process in the field-of-view adjustment device A103 according to the first embodiment. The flowchart in Fig. 2 will be described in detail later. The tracking system A100 includes an imaging device A101, an input device A102, a field-of-view adjustment device A103, and a monitor device A104.

[0010] The imaging device A101 is a device that captures moving images and generates moving images. It is configured to include at least a camera and a pan head on which the camera is mounted. The camera has an autofocus function, a zoom function, and a video capture function. The pan head is capable of mounting a camera and is capable of pan / tilt driving. The camera's zoom control and the pan / tilt driving of the pan head are controlled by a field-of-view adjustment device A103. In the following description, pan / tilt is abbreviated as "PT," and pan / tilt / zoom are collectively abbreviated as "PTZ." In this manner, the imaging device A101 adjusts the field angle during video capture by performing PTZ control using the field-of-view adjustment device A103. That is, in this embodiment, the field-of-view adjustment device A103 performs PTZ control to adjust the field angle determined by the shooting direction and zoom magnification during video capture by the imaging device A101. The imaging device A101 outputs image data of a moving image generated by imaging (hereinafter simply referred to as "image" unless otherwise specified) to the angle-of-view adjusting device A103.

[0011] The input device A102 is a device capable of acquiring operation input from a user, and is configured with a joystick, a touch panel, a mouse, etc., and acquires the operation input from the user as user operation information. The user operation information includes at least information instructing the shooting direction of the imaging device A101. Furthermore, when the user inputs information specifying the zoom magnification of the imaging device A101, the user operation information also includes information specifying the zoom magnification. The input device A102 outputs the user operation information to the angle-of-view adjustment device A103. At this time, the angle-of-view adjustment device A103 performs PTZ control of the imaging device A101 based on the user operation information input from the input device A102. That is, in this embodiment, the user operation information input to the input device A102 includes PTZ operation information for causing the imaging device A101 to perform a PTZ operation via the angle-of-view adjustment device A103.

[0012] The angle-of-view adjustment device A103 is a device that has a function of adjusting the angle of view of the imaging device A101 by PTZ control. When a captured image of a moving image is input from the imaging device A101, the angle-of-view adjustment device A103 detects an image area of ​​an object to be tracked from the image and performs PTZ control of the imaging device A101 so that the object always fits within the angle of view during shooting of the moving image. That is, the angle-of-view adjustment device A103 has an automatic tracking function that adjusts the angle of view of the imaging device A101 to track the object so that it fits within the image captured by the imaging device A101. In this embodiment, the object to be detected from the captured image may be a predetermined object, such as a person such as a player or a ball in a sports scene. Of course, the predetermined object is not limited to a person or a ball, and may be a target desired by the user, such as a vehicle or an animal. Hereinafter, in this embodiment, an object detected from an image and its image area will be simply referred to as an "object" unless otherwise specified. In this way, the angle-of-view adjustment device A103 detects an object from a video image captured by the imaging device A101, and adjusts the angle of view by performing PTZ control on the imaging device A101 to automatically track the object so that the object does not leave the angle of view of the imaging device A101 during video capture.

[0013] Furthermore, the angle-of-view adjustment device A103 of this embodiment has a function of automatically tracking an object using PTZ control, as well as a function of receiving user operation information input from the input device A102. For example, the angle-of-view adjustment device A103 also has a function of allowing the user to select a desired tracking target by receiving a PTZ operation from the user when the object to be tracked is not the object intended by the user. However, even if the user selects a tracking target and then performs object tracking using PTZ control, an object not intended by the user may be recaptured and recognized as the tracking target.

[0014] For this reason, the field-of-view adjustment device A103 of this embodiment also has a function for determining whether to treat an object detected from an image as a non-target object to be excluded from tracking targets, in other words, a function for determining whether to treat an object detected from an image as a tracking target. The field-of-view adjustment device A103 of this embodiment compares information related to an object detected from an image before a user operation input is received with information related to an object detected from an image after the user operation input is no longer received. Based on the comparison result, the field-of-view adjustment device A103 then determines whether to exclude an object detected from an image after the user operation input is no longer received from tracking targets. That is, when a user performs a PTZ operation input, the field-of-view adjustment device A103 compares an object detected from an image before the PTZ operation input is received with an object detected from an image after the PTZ operation input is no longer received. Based on the comparison result, the field-of-view adjustment device A103 then determines whether to treat an object detected from an image after the user PTZ operation input is no longer received as a non-target object to be excluded from tracking targets. In the angle-of-view adjusting device A103 of this embodiment, if the input of user operation information ceases after the angle of view adjustment based on user operation information, objects similar to objects detected before the input of the user operation information are excluded from the target objects to be tracked by the angle-of-view adjustment. Also, as will be described in detail later, in the first embodiment, feature amounts acquired from the image area of ​​the object detected from the image are used as object-related information, and it is determined whether the feature amounts are similar.

[0015] The angle-of-view adjustment device A103 then selects an object that is not a non-target object as a tracking target, and performs PTZ control so that the tracking target object falls within the angle of view of the imaging device A101, thereby automatically tracking the object. The angle-of-view adjustment device A103 outputs the image acquired through such angle-of-view adjustment to the monitor device A104. The monitor device A104 displays an image based on the image information input from the angle-of-view adjustment device A103.

[0016] As described above, in the tracking system A100 of this embodiment, when PTZ control based on user operation information is not performed, the imaging device A101 is automatically PTZ controlled to detect an object from a captured image, and the angle of view is adjusted to automatically track the object so that it does not leave the angle of view. On the other hand, when user operation information is input via the input device A102, the tracking system A100 adjusts the angle of view through PTZ control based on the user operation information. Then, when performing automatic PTZ control after PTZ control based on user operation information has ended, the tracking system A100 treats objects similar to the object that was the tracking target before PTZ control based on user operation information was performed as non-target objects and does not consider them as tracking targets.

[0017] The following describes in detail the configuration and operation of the angle-of-view adjusting device A103 of the first embodiment, which enables the angle-of-view adjustment and object tracking in the tracking system A100 as described above. The view angle adjustment device A103 includes an image acquisition unit A105, an object detection unit A106, a feature acquisition unit A107, an object exclusion unit A108, an object selection unit A113, an information recording unit A114, a direction estimation unit A115, and an association unit A116. The view angle adjustment device A103 further includes an input reception unit A109, an input storage unit A110, an astigmatism determination unit A111, an astigmatism storage unit A112, an operation determination unit A117, a view angle control unit A118, and an image output unit A119.

[0018] The image acquisition unit A105 sequentially acquires images of each frame of a moving image captured by the imaging device A101, and outputs the images of each frame to the object detection unit A106 and the angle of view control unit A118. The object detection unit A106 performs object detection processing on the image input to the image acquisition unit A105. Any method capable of detecting an object may be used for the object detection processing, such as template matching or semantic region segmentation. These object detection methods are well-known technologies, and therefore detailed description thereof will be omitted. The object detection unit A106 outputs information on the image area of ​​the detected object and coordinate information on the area of ​​the object within the image to the feature acquisition unit A107.

[0019] The feature acquisition unit A107 acquires feature amounts of an object detected by the object detection unit A106 from image information of the object. Any feature amount may be used as long as it is information that can be acquired to represent the characteristics of the object, such as color information, edge information, brightness information, contrast information, or size information of the image area of ​​the object. The feature acquisition unit A107 outputs the feature amount and coordinate information of the object to the target exclusion unit A108.

[0020] The object exclusion unit A108 compares the feature amount of the object input from the feature acquisition unit A107 with the feature amount of the object stored in a non-target storage unit A112 (described later) to determine whether or not to treat the object detected from the image by the object detection unit A106 as a non-target object. In this embodiment, a non-target object is an object that is excluded from targets to be tracked by automatic PTZ control in the angle of view adjustment device A103. Details of the process of excluding a non-target object from tracking targets will be described later.

[0021] The target selection unit A113 selects a target object when a target object has not been selected from the coordinate information and feature amounts of the object input from the target exclusion unit A108, or when a correspondence has not been established. Note that when there are multiple objects, the target selection unit A113 selects the object closest to the center of the angle of view as the target object. Any method may be used, such as selecting the object with the largest size as the target object. A detailed description of the target selection process in the target selection unit A113 will be given later. The target selection unit A113 outputs the coordinate information and feature amounts of the selected target object to the non-target holding unit A112 and the information recording unit A114.

[0022] The information recording unit A114 records the coordinate information of the object input from the target selection unit A113. In this embodiment, the information recording unit A114 records the coordinate information of the object corresponding to each frame of the video. Then, the information recording unit A114 outputs the coordinate information of the object in the current frame and the frame immediately before to the direction estimation unit A115.

[0023] The direction estimation unit A115 calculates a movement vector of the object from the coordinate information of the object in the current frame and the previous frame input from the information recording unit A114, and estimates the movement direction of the object based on the movement vector. Note that if the coordinate information of the object in the previous frame was not recorded, it is assumed that there is no movement direction. Then, the direction estimation unit A115 outputs information on the estimated movement direction of the object to the association unit A116.

[0024] The association unit A116 determines whether the object is the same between the frames based on the object's movement direction and the object's coordinate information in the current and previous frames, which are input from the direction estimation unit A115. If the association unit A116 determines that the object is the same, it updates the object's coordinate information and associates the object. Then, the association unit A116 outputs the updated object's coordinate information and movement direction information to the operation determination unit A117.

[0025] The operation determination unit A117 determines whether to adjust the angle of view based on the coordinate information of the object and the moving direction of the object input from the association unit A116, or whether to adjust the angle of view based on the user operation information input from the input reception unit A109.

[0026] When a user operation input (user operation information) is input from the input device A102, the input acceptance unit A109 accepts the user operation input. In the present embodiment, the input acceptance unit A109 accepts a PTZ operation input from the user as user operation information. The input acceptance unit A109 then outputs the accepted user operation information to the operation determination unit A117 and the input storage unit A110. The configuration of the input storage unit A110 and subsequent units will be described later.

[0027] When user operation information is input, the operation determination unit A117 determines to adjust the angle of view using PTZ coordinate values ​​corresponding to the user operation information.The operation determination unit A117 then outputs the PTZ coordinate values ​​corresponding to the user operation information to the angle of view control unit A118 as angle of view operation information.On the other hand, when user operation information is not input, the operation determination unit A117 determines to adjust the angle of view based on the object coordinate information and object movement direction input from the association unit A116.When user operation information is not input, the operation determination unit A117 outputs the object coordinate information and object movement direction information input from the association unit A116 to the angle of view control unit A118.

[0028] The angle-of-view control unit A118 controls the angle of view of the imaging device A101 based on the image input from the image acquisition unit A105 and the information input from the operation determination unit A117. When the operation determination unit A117 inputs object coordinate information and movement direction information from the association unit A116, the angle-of-view control unit A118 calculates PTZ coordinate values ​​for adjusting the angle of view so that the center of gravity of the object obtained from the object coordinate information becomes the center of the angle of view. The angle-of-view control unit A118 then controls the angle of view of the imaging device A101 based on the PTZ coordinate values. As a result, the imaging device A101 performs PT control corresponding to the object movement direction input from the association unit A116, thereby automatically tracking the object. However, if only coordinate information is input from the association unit A116 without movement direction information, the angle-of-view control unit A118 does not perform PT operation of the imaging device A101. Furthermore, when neither coordinate information nor movement direction information is input from the association unit A116, the angle-of-view control unit A118 controls the imaging device A101 to perform PT operation in the same direction as the previous movement direction. The angle-of-view control unit A118 selects a zoom magnification corresponding to the amount of movement of the object from among predetermined zoom magnifications, and performs zoom control of the imaging device A101 according to the selected zoom magnification. While performing this angle-of-view control, the angle-of-view control unit A118 also outputs images input from the image acquisition unit A105 to the image output unit A119. When PTZ coordinate values ​​corresponding to user operation information are input, the angle-of-view control unit A118 adjusts the angle of view of the imaging device A101 according to the PTZ coordinate values ​​corresponding to the user operation information.

[0029] The image output unit A119 outputs the images (moving images) acquired by the image acquisition unit A105 and input from the angle of view control unit A118 to the monitor device A104. As a result, a video based on the moving images is displayed on the monitor device A104.

[0030] Furthermore, when the input receiving unit A109 receives a user operation input, for example, a PTZ operation input, as user operation information, the input storage unit A110 stores the user operation information. In this embodiment, the input storage unit A110 stores user operation information corresponding to each frame of the video. The input storage unit A110 then outputs the user operation information for the current frame and the user operation information for the frame immediately preceding it to the non-eligibility determination unit A111.

[0031] The non-target determination unit A111 calculates the difference between the user operation information for the current frame input from the input storage unit A110 and the user operation information for the previous frame. Then, based on the difference, the non-target determination unit A111 determines whether to retain the feature amount of the object detected by the object detection unit A106 in the current frame. The non-target determination unit A111 determines to retain the feature amount of the object as the feature amount of a non-target object only when there is no user operation information for the previous frame and only user operation information for the current frame has been input. Then, the non-target determination unit A111 outputs the determination result to the non-target storage unit A112.

[0032] When the non-target holding unit A112 receives a determination result from the non-target determination unit A111 indicating that the feature amounts of an object should be held as feature amounts of a non-target object, the non-target holding unit A112 holds the feature amount information of the object corresponding to the coordinate information input from the target selection unit A113. That is, the feature amounts of the object held by the non-target holding unit A112 are the feature amounts of the object that was being tracked by automatic PTZ control in the angle-of-view adjustment device A103 when a user's PTZ operation was input from the input device A102. The non-target holding unit A112 then outputs the held feature amount information of the object to the target exclusion unit A108. In this embodiment, when the input of user operation information is stopped and automatic PTZ control is performed in the angle-of-view adjustment device A103, the non-target holding unit A112 outputs the held feature amount information of the object to the target exclusion unit A108.

[0033] Therefore, after PTZ control has been performed based on user operation information, when the input of user operation information ceases and automatic PTZ control is performed, the target exclusion unit A108 determines whether to make the object detected from the image by the object detection unit A106 a tracking target. If the feature amount of an object newly detected after the input of user operation information ceases is similar to the feature amount of the object stored in the non-target storage unit A112, the target exclusion unit A108 excludes the newly detected object as a non-target object from the tracking targets. On the other hand, if the feature amount of an object newly detected after the input of user operation information ceases is not similar to the feature amount of the object stored in the non-target storage unit A112, the target exclusion unit A108 designates the detected object as a target object to be automatically tracked by PTZ control.

[0034] Below, an example will be described with reference to Figures 3, 4, and 5 in which the object exclusion unit A108 excludes an object newly detected from an image by the object detection unit A106 as a non-target object from the target objects to be automatically tracked by PTZ control. 3 is a diagram showing image D001 acquired by image acquisition unit A105, in which objects P000, P001, and P002 are shown. In addition, in Fig. 3, objects P000, P001, and P002 detected from image D001 by object detection unit A106 are represented by rectangles P003, P004, and P005, respectively.

[0035] Fig. 4(a) is a diagram showing a non-target object feature list D101 in which the object IDs (identification information) and features of objects determined to be non-target objects by the object exclusion unit A108 are described. The non-target object feature list D101 in Fig. 4(a) shows an example in which the object P001 shown in Fig. 3 has been determined to be a non-target object, and the object ID P100 of the object P01 and its features such as color information P101 and luminance information P102 are described.

[0036] Fig. 4(b) is a diagram showing a detected object feature list D102 that describes the object IDs and features of objects detected from the input image by the object detection unit A106. The detected object feature list D102 in Fig. 4(b) shows an example in which object IDs P103 of objects P001, P002, ... detected from image D001 in Fig. 3 are described, along with color information P104 and brightness information P105, which are the features of the detected objects.

[0037] FIG. 4(c) shows a similarity detection result list D103, which shows the results of the object exclusion unit A108 calculating the similarity between the feature amounts described in the non-target object feature amount list D101 and the feature amounts described in the detected object feature amount list D102 and comparing the similarities. The similarity detection result list D103 in FIG. 4(c) shows an example in which a similarity determination result P106 is listed as the result of the similarity comparison, and a comparison result P107 of color information and a comparison result P108 of brightness information, which are feature amounts for each object. In this embodiment, the object exclusion unit A108 represents each feature amount as a vector, calculates the difference between the feature amount vectors, and compares the difference with a predetermined threshold to determine whether the features are similar. The object exclusion unit A108 may use a method for comparing similarities using a histogram of the feature amounts, or any other method that can compare similarities may be used. In the example of the similarity detection result list D103 shown in Figure 4(c), the similarity comparison result for object P001 indicates a high value, so object P001 is determined to be a non-target object to be excluded from the target objects. If determined to be a non-target object, the rectangle information for object P001 is deleted.

[0038] FIG. 5 is a diagram showing an image D201 according to the result of similarity comparison such as the similarity detection result list D103 shown in FIG. 4(c). It is assumed that the objects P200, P201, and P202 in the image D201 respectively correspond to the objects P000, P001, and P002 shown in FIG. 3. In this example, since the objects whose feature amounts are respectively determined not to be similar are the objects P200 and P202, the objects P200 and P202 are respectively represented by rectangles P203 and P205. On the other hand, since the object P201 is made a non-target object and excluded, no rectangle is attached. Then, the non-target exclusion part A108 excludes the object P201 and outputs the rectangle information of the objects P200 and P202 determined to be targets to the target selection part A113.

[0039] FIG. 6 is a diagram used to explain the method of selecting a target object in the target selection part A113. As shown in FIG. 6, it is assumed that objects P301, P302, and P303 are shown in the image D301 acquired by the image acquisition part A105. Also shown in FIG. 6 are the imaging angle center P300, the center of gravity P304 of the object P301, the center of gravity P305 of the object P303, the distance P306 between the imaging angle center P300 and the center of gravity P304, and the distance P307 between the imaging angle center P300 and the center of gravity P305. And it is assumed that the magnitude relationship between the distance P306 and the distance P307 is P306 < P307. In the case of this example, the target selection part A113 selects the object P301 whose center of gravity position is closest to the imaging angle center P300 as the target object. On the other hand, since the object P302 is a non-target object already excluded from the target, it cannot be selected.

[0040] Hereinafter, referring to the flowchart of FIG. 2, the procedure of the imaging angle adjustment process in the tracking system A100 of the first embodiment will be described. In the flowchart of FIG. 2 and each flowchart described hereinafter, it is assumed that the symbol "S" represents a processing step. When the tracking system A100 is activated by a user operation, it starts the processing of the flowchart in Fig. 2. Then, in S001, the image acquisition unit A105 of the angle-of-view adjustment device A103 acquires image information from the imaging device A101 and outputs the image information to the object detection unit A106 and the angle-of-view control unit A118.

[0041] Next, in S002, the object detection unit A106 performs an object detection process to detect an object from the image input from the image acquisition unit A105. Then, the object detection unit A106 outputs image information and coordinate information of the detected object to the feature acquisition unit A107.

[0042] Next, in S003, the feature acquisition unit A107 acquires a feature amount of the object from the image information of the object input from the object detection unit A106. Then, the feature acquisition unit A107 outputs the acquired feature amount and coordinate information of the object to the target exclusion unit A108.

[0043] Next, in S004, the object exclusion unit A108 compares the feature amounts of the object input from the feature acquisition unit A107 with the feature amounts of the object stored in the non-object storage unit A112. If the feature amounts are similar, the object exclusion unit A108 determines the object detected by the object detection unit A106 to be a non-object object and excludes it from the target, and if they are not similar, it determines the object to be a target object.

[0044] Next, in S005, the target selection unit A113 selects a target object if the target object has not been selected from the object coordinate information and feature amounts input from the target exclusion unit A108, or if the object has not been associated with the object coordinate information and feature amounts. The target object selected is the object closest to the center of the angle of view. The target selection unit A113 then outputs the coordinate information and feature amounts of the selected object to the non-target storage unit A112 and the information recording unit A114.

[0045] In S006, the input receiving unit A109 determines whether or not user operation information has been input from the input device A102. If user operation information has been input (YES in S006), the input receiving unit A109 outputs the user operation information to the input storage unit A110 and the operation determination unit A117, and the view angle adjustment device A103 proceeds to the process of S007. On the other hand, if user operation information has not been input (NO in S006), the view angle adjustment device A103 proceeds to the process of S010.

[0046] If the process proceeds to S007, the input holding unit A110 holds the user operation information from the input receiving unit A109. Then, the input holding unit A110 outputs the user operation information of the current frame and the previous frame to the non-eligible determination unit A111.

[0047] Next, in S008, the non-target determination unit A111 determines whether to retain information about a non-target object based on the difference between the user operation information for the current frame and the previous frame input from the input storage unit A110. The non-target determination unit A111 determines to retain information about a non-target object only if there is no input information for the previous frame and there is input only for the current frame. Then, if there is no input information for the previous frame and there is input only for the current frame (YES in S008), the view angle adjustment device A103 proceeds to the processing of S013. On the other hand, if not (NO in S008), the non-target determination unit A111 outputs the determination result to the non-target storage unit A112, and then the view angle adjustment device A103 proceeds to the processing of S009.

[0048] Next, in S009, the non-target holding unit A112 holds the feature amount of the object corresponding to the coordinate information of the object input from the target selection unit A113 based on the determination result input from the non-target determination unit A111. Then, the non-target holding unit A112 outputs the held feature amount information of the object to the target exclusion unit A108. Thereafter, the view angle adjustment device A103 proceeds to the process of S013.

[0049] Also, if the process proceeds to S010, the information recording unit A114 records the coordinate information of the object input from the target selection unit A113. Then, the information recording unit A114 outputs the coordinate information of the object in the current frame and the frame immediately before to the direction estimation unit A115.

[0050] Next, in S011, the direction estimation unit A115 calculates a movement vector of the object from the coordinate information of the object in the current frame and the previous frame input from the information recording unit A114, and estimates the movement direction of the object based on the movement vector. Note that if the coordinate information of the object in the previous frame was not recorded, it is assumed that there is no movement direction. Then, the direction estimation unit A115 outputs information indicating the estimated movement direction of the object to the association unit A116.

[0051] Next, in S012, the association unit A116 determines whether the object is the same object between the frames based on the movement direction of the object estimated by the direction estimation unit A115 and the coordinate information of the object in the current and previous frames. If the association unit A116 determines that the object is the same object, it updates the coordinate information of the object to associate the object, and outputs the updated coordinate information and movement direction information of the object to the operation determination unit A117.

[0052] Next, in S013, the operation determination unit A117 determines whether to adjust the angle of view based on the object coordinate information and movement direction input from the association unit A116, or whether to adjust the angle of view based on user operation information input from the input reception unit A109. If user operation information has been input, the operation determination unit A117 determines to adjust the angle of view using PTZ coordinate values ​​corresponding to the user operation information, and outputs the PTZ coordinate values ​​corresponding to the user operation information as angle of view operation information to the angle of view control unit A118. On the other hand, if user operation information has not been input, the operation determination unit A117 determines to adjust the angle of view based on the object coordinate information and movement direction input from the association unit A116. Then, the operation determination unit A117 outputs the object coordinate information and movement direction information input from the association unit A116 to the angle of view control unit A118.

[0053] Next, in S014, the angle-of-view control unit A118 controls the angle of view of the imaging device A101 based on the image input from the image acquisition unit A105 and the information input from the operation determination unit A117. When the angle-of-view control unit A118 receives object coordinate information and movement direction information from the association unit A116, it controls the angle of view of the imaging device A101 using PTZ coordinate values ​​for adjusting the angle of view so that the center of gravity of the object obtained from the object coordinate information becomes the center of the angle of view. As described above, if only coordinate information is received from the association unit A116 but no movement direction information is received, the angle-of-view control unit A118 does not perform PT operation of the imaging device A101. Furthermore, if neither coordinate information nor movement direction information is received from the association unit A116, the angle-of-view control unit A118 controls the imaging device A101 to perform PT operation in the same direction as the previous movement direction. The angle-of-view control unit A118 selects a zoom magnification corresponding to the amount of movement of the object from among predetermined zoom magnifications, and performs zoom control of the imaging device A101. The angle-of-view control unit A118 then outputs the image input from the image acquisition unit A105 to the image output unit A119.

[0054] Next, in S015, the image output unit A119 outputs the image input from the angle of view control unit A118 to the monitor device A104. Next, in S016, the angle-of-view adjustment device A103 determines whether a tracking system ON / OFF switch (not shown) has been operated by the user to stop the angle-of-view adjustment device A103. If the stop operation has not been performed (NO in S016), the angle-of-view adjustment device A103 returns to S002, and if the stop operation has been performed (YES in S016), the process of the flowchart in FIG. 2 ends.

[0055] As described above, in the field of view adjustment device A103 of this embodiment, when an unintended object is tracked and a PTZ operation is input by the user, the field of view adjustment device A103 designates the object that was being tracked immediately before the PTZ operation was input by the user as a non-target object and retains its feature amounts. Then, when the PTZ operation by the user ceases, the field of view adjustment device A103 excludes from tracking targets any object with feature amounts similar to the object that was tracked and retained before the PTZ operation by the user during subsequent tracking. This makes it possible for the tracking system A100 of this embodiment to prevent erroneous tracking from being performed again, thereby suppressing the occurrence of erroneous tracking.

[0056] In this embodiment, the non-target object determination process is triggered by the presence or absence of a PTZ operation input from the user. Alternatively, the non-target object determination process may be triggered by other user operation input, such as an automatic tracking operation reset operation input or automatic tracking pause operation input, when erroneous tracking occurs in automatic tracking, for example.

[0057] Second Embodiment Next, a second embodiment will be described. Fig. 7 is a block diagram showing an example of the functional configuration of a tracking system B100 including a field-of-view adjustment device B103 according to the second embodiment. Fig. 8 is a flowchart showing the processing flow in the field-of-view adjustment device B103 according to the second embodiment. The flowchart in Fig. 8 will be described later. In the second embodiment, descriptions of the same configurations and processing as those in the first embodiment will be omitted as appropriate.

[0058] The tracking system B100 of the second embodiment has an overhead-view imaging device B121 that captures an overhead image of a tracking target object, and acquires position information of the tracking target object obtained by capturing an image of the tracking target object with the overhead-view imaging device B121. Furthermore, in the tracking system B100 of the second embodiment, the position of the object captured from the overhead-view image is associated with the position of the object in the image of the object captured and detected by the imaging device A101. After the angle of view is adjusted based on user operation information, when the tracking target object is selected again, any object located at a position similar to the position captured and held by the overhead-view imaging device B121 is designated as a non-target object and not a tracking target. That is, in the second embodiment, the position information (coordinate information) of the object is used as the related information of the object.

[0059] The tracking system B100 of the second embodiment includes an imaging device A101, an input device A102, a field angle adjustment device B103, a monitor device A104, an imaging position input device B120, and an overhead imaging device B121. The imaging position input device B120 is a device that receives input of position information of the imaging device A101 from the user, and the position information is input from the user via a touch panel, a mouse, etc. The imaging position input device B120 outputs the input position information to the angle of view adjustment device B103. The overhead image capturing device B121 is a device that captures an overhead image of the location captured by the image capturing device A101 and is configured with a camera, etc. The overhead image capturing device B121 outputs the captured overhead image to the angle of view adjustment device B103.

[0060] As described above, when an image is input from the imaging device A101, the angle-of-view adjustment device B103 detects an object in the image and adjusts the angle of view of the imaging device A101 by performing PTZ control so that the object does not leave the angle of view. The angle-of-view adjustment device B103 of the second embodiment acquires, from the overhead image captured by the overhead image capture device B121, position information of the object that was detected when user operation information was acquired from the input device A102. The angle-of-view adjustment device B103 then retains the object position information and controls the angle of view based on the object position information and the user operation information. When selecting a tracking target from newly detected objects after the input of user operation information has ceased, the angle-of-view adjustment device B103 of this embodiment excludes, as non-target objects, objects having position information similar to the object position information acquired and retained from the overhead image from being tracked.

[0061] The view angle adjustment device B103 of the second embodiment includes an image acquisition unit A105, an object detection unit A106, a distance acquisition unit B107, an object exclusion unit B108, an object selection unit B113, an information recording unit A114, a direction estimation unit A115, and an association unit A116. The view angle adjustment device A103 also includes an input acceptance unit A109, an input storage unit A110, an asymmetric determination unit A111, a position association unit B127, an asymmetric determination unit B112, an operation determination unit A117, a view angle control unit A118, and an image output unit A119. The view angle adjustment device B103 also includes a PT value acquisition unit B122, an imaging position acquisition unit B123, a coordinate conversion unit B124, an overhead image acquisition unit B125, and an overhead view object detection unit B126. The angle-of-view adjusting device A103 outputs the image acquired after adjusting the angle of view to the monitor device A104.

[0062] The distance acquisition unit B107 acquires distance information from the object detected by the object detection unit A106 to the image capture device A101. For example, if the image capture device A101 is configured to output distance information included in a captured image, the distance acquisition unit B107 acquires distance information from the object's region in the captured image. For example, the distance information can be distance information acquired by a distance measurement sensor using a phase difference method or distance information based on the phase difference between each pixel or region acquired from an image sensor having a light-receiving region with a different pupil position for each pixel. Other distance information may be obtained by a method of estimating the distance from the size of the object or by a distance sensor, or by any method that can acquire the distance to the object. The distance acquisition unit B107 outputs the acquired distance information to the object and the object's coordinate information input from the object detection unit A106 to the target exclusion unit B108.

[0063] The PT value acquisition unit B122 acquires the current PT value from the image capture device A101. Then, the PT value acquisition unit B122 outputs the acquired PT value to the coordinate conversion unit B124. The imaging position acquisition unit B123 outputs the position information of the imaging device A101 input from the imaging position input device B120 to the coordinate conversion unit B124.

[0064] The overhead image acquisition unit B125 acquires the overhead image captured by the overhead image capturing device B121, and outputs it to the overhead object detection unit B126. The overhead object detection unit B126 performs object detection processing from the overhead image input from the overhead image capturing device B121. Any method that can detect an object may be used for object detection processing, such as template matching or semantic region segmentation. These object detection methods are well-known technologies, so detailed explanations will be omitted. The overhead object detection unit B126 outputs coordinate information of the object detected from the overhead image to the position association unit B127.

[0065] The coordinate conversion unit B124 calculates PTZ coordinate values ​​based on the object's in-image coordinate information and object distance information input from the target selection unit B113, the PT value from the PT value acquisition unit B122, and the image capture device position information from the image capture position acquisition unit B123. The coordinate conversion unit B124 then converts the PTZ coordinate values ​​into bird's-eye coordinate values. When bird's-eye coordinate information of an object is input from the non-target holding unit B112, the coordinate conversion unit B124 converts the information into PTZ coordinate values ​​and then into in-image coordinate values. The coordinate conversion unit B124 then outputs the object's in-image coordinate information to the target exclusion unit B108, and outputs the object's bird's-eye coordinate information to the position association unit B127.

[0066] The target exclusion unit B108 compares the coordinate information and distance information of the object input from the distance acquisition unit B107 with the coordinate information and distance information of the non-target object input from the coordinate conversion unit B124, and excludes objects that are similar to the coordinate information and distance information as non-target objects.The target exclusion unit B108 then outputs the coordinate information of objects that are not similar to the non-target objects to the target selection unit B113.

[0067] The position association unit B127 receives the determination result from the asymmetric determination unit A111 and, based on the determination result, associates, as a non-target object, an object whose overhead coordinate information from the coordinate conversion unit B124 is similar to that from the overhead coordinate information from the overhead object detection unit B126. The position association unit B127 then outputs the overhead coordinate information of the associated non-target object to the asymmetric storage unit B112.

[0068] The asymmetric object holding unit B112 holds the overhead coordinate information of the asymmetric object input from the position associating unit B127. Then, the asymmetric object holding unit B112 outputs the held overhead coordinate information of the asymmetric object to the coordinate conversion unit B124.

[0069] An example of conversion from image coordinate information to PTZ coordinate values ​​and an example of conversion from bird's-eye coordinate information to PTZ coordinate values ​​will be described below with reference to FIGS. 9, 10, and 11. FIG. Fig. 9 is a diagram showing an in-image coordinate system D401 in an image acquired by the image acquisition unit A105, and also shows a detected object P400 and rectangle P401. The vertices of rectangle P401 are represented by coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4), and the center is represented by coordinates (x0, y0).

[0070] Fig. 10 shows a coordinate system D501 resulting from coordinate conversion by the coordinate conversion unit B124 on the vertex coordinates and center coordinates shown in Fig. 9. In Fig. 10, coordinate system D501 represents a PTZ coordinate system, and Fig. 10 also shows an object P500 and a rectangle P501. In the PTZ coordinate system, each vertex of rectangle P501 is represented by coordinates (p1, t1), (p2, t2), (p3, t3), and (p4, t4), and the central angle of view is represented by coordinates (p0, t0). The coordinate values ​​of the center and each vertex of this rectangle P501 are values ​​converted from the coordinates of the center and each vertex in coordinate system D401 of Fig. 9.

[0071] Fig. 11 is a diagram showing an overhead coordinate system D601 corresponding to an overhead image captured by the overhead image capturing device B121. Fig. 11 shows the central coordinate position (x0', y0') of an object P600, which shows an object detected by the object detection unit A106 in correspondence with the overhead coordinate system, and the coordinate position (x1', y1') indicating the position P601 of the image capturing device A101 input from the position input unit B120. Also in Fig. 11, a line P602 represents the distance from the position of the image capturing device A101 (coordinate position (x1', y1')) to the central position of the object (central coordinate position (x0', y0')). The central coordinate position (x0', y0') of the object can be calculated from the coordinate position (x1', y1') of the image capturing device A101, the coordinates (p0, t0) of the central angle of view of the rectangle P502, and the distance information of the line P602. These coordinate transformations can also be reversed.

[0072] Hereinafter, the procedure of the angle of view adjustment process in the tracking system B100 of the second embodiment will be described with reference to the flowchart of FIG. When the tracking system B100 is activated by a user operation, it starts the processing of the flowchart in Fig. 8. Then, in S101, the image acquisition unit A105 of the angle of view adjustment device B103 acquires image information from the imaging device A101 and outputs the image information to the object detection unit A106 and the angle of view control unit A118.

[0073] Next, in S102, the overhead image acquisition unit B125 acquires an overhead image captured by the overhead image capturing device B121, and outputs the overhead image to the overhead object detection unit B126. Next, in S103, the imaging position acquisition unit B123 determines whether the user has input position information of the imaging device A101 via the imaging position input device B120. If the user has input position information (YES in S103), the imaging position acquisition unit B123 outputs the input position information to the coordinate conversion unit B124, and then the angle of view adjustment device B103 proceeds to the processing of S104. On the other hand, if the user has not input position information (NO in S103), the imaging position acquisition unit B123 continues the determination processing of S103.

[0074] In S104, the object detection unit A106 performs object detection processing on the image input from the image acquisition unit A105. Then, the object detection unit A106 outputs coordinate information of the detected object to the distance acquisition unit B107. Also, in S105, the overhead object detection unit B126 performs object detection processing on the input overhead image, and outputs the overhead coordinate information of the detected object to the position association unit B127.

[0075] Next, in S106, the distance acquisition unit B107 acquires distance information of the object detected by the object detection unit A106. Then, the distance acquisition unit B107 outputs the distance information of the object and the coordinate information of the object acquired from the object detection unit A106 to the target exclusion unit B108.

[0076] Next, in S107, the coordinate conversion unit B124 converts the bird's-eye view coordinate information of the object input from the non-target holding unit B112 into PTZ coordinate values, and also converts the coordinates into in-image coordinate values ​​of the object. Then, the coordinate conversion unit B124 outputs the in-image coordinate information of the object to the target exclusion unit B108.

[0077] Next, in S108, the target exclusion unit B108 excludes non-target objects based on a comparison between the coordinate information and distance information of the object input from the distance acquisition unit B107 and the coordinate information and distance information of the non-target objects input from the coordinate conversion unit B124. Then, the target exclusion unit B108 outputs the coordinate information of the object that is not similar to the position information of the non-target object to the target selection unit B113.

[0078] Next, in S109, the target selection unit B113 selects a target object when a target object has not been selected or cannot be associated, based on the coordinate information and distance information of the object input from the target exclusion unit B108. The target selection unit B113 selects the object closest to the center of the angle of view as the target object. Then, the target selection unit B113 outputs the coordinate information and distance information of the selected object to the coordinate conversion unit B124 and the information recording unit A114.

[0079] Next, in S110, the input receiving unit A109 determines whether user operation information has been input from the input device A102. If user operation information has been input (YES in S110), the input receiving unit A109 outputs the user operation information to the input storage unit A110 and the operation determination unit A117, and the view angle adjustment device B103 proceeds to the processing of S111. On the other hand, if user operation information has not been input (NO in S110), the view angle adjustment device B103 proceeds to the processing of S117.

[0080] When the process proceeds to S111, the input storage unit A110 records the user operation information sent from the input reception unit A109. Then, the input storage unit A110 outputs the user operation information of the current frame and the frame immediately before to the non-eligible determination unit A111.

[0081] Next, in S112, the non-target determination unit A111 determines whether to retain information about a non-target object based on the difference between the user operation information for the current frame and the previous frame input from the input retention unit A110. The non-target determination unit A111 determines to retain information about a non-target object only if there was no input information in the previous frame and there was input only in the current frame. That is, if there was no input information in the previous frame and there was input only in the current frame (YES in S112), the non-target determination unit A111 determines to retain information about a non-target object, and the angle of view adjustment device B103 proceeds to the processing of S120. On the other hand, if this is not the case (NO in S112), the non-target determination unit A111 outputs the non-target object determination result to the position association unit B127, and the angle of view adjustment device B103 proceeds to the processing of S113.

[0082] If the process proceeds to S113, the PT value acquisition unit B122 acquires the current PT value from the image capture device A101, and outputs the acquired PT value to the coordinate conversion unit B124. Next, in S114, the coordinate conversion unit B124 converts the coordinates into PTZ coordinate values ​​based on the object's in-image coordinate information and distance information from the target selection unit B113, the PT value from the PT value acquisition unit B122, and the image capture device position information from the image capture position acquisition unit B123. The coordinate conversion unit B124 then converts the coordinates into bird's-eye coordinate values. The coordinate conversion unit B124 then outputs the object's bird's-eye coordinate information to the position association unit B127.

[0083] Next, in S115, the position association unit B127 associates, as a non-target object, an object whose overhead coordinate information of the non-target object from the coordinate conversion unit B124 is similar to that of the object from the overhead-view object detection unit B126, based on the determination result from the non-target determination unit A111. Then, the position association unit B127 outputs the overhead coordinate information of the associated non-target object to the non-target storage unit B112.

[0084] Next, in S116, the asymmetric object holding unit B112 holds the overhead coordinate information of the asymmetric object input from the position association unit B127. Then, the asymmetric object holding unit B112 outputs the held overhead coordinate information of the asymmetric object to the coordinate conversion unit B124. Thereafter, the angle of view adjustment device B103 proceeds to s120.

[0085] Also, if the process proceeds to S117, the information recording unit A114 records the coordinate information of the object input from the target selection unit B113. Then, the information recording unit A114 outputs the coordinate information of the object in the current frame and the frame immediately before to the direction estimation unit A115. Next, in S118, the direction estimation unit A115 calculates the object's movement vector from the object's coordinate information for the current and previous frames input from the information recording unit A114, and estimates the object's movement direction based on the calculated vector. Note that if the object's coordinate information for the previous frame was not recorded, it is assumed that there is no movement direction. The direction estimation unit A115 then outputs the estimated object's movement direction to the association unit A116.

[0086] Next, in S119, the association unit A116 determines whether the object is the same between the frames based on the object's movement direction estimated by the direction estimation unit A115 and the object's coordinate information in the current and previous frames. If the association unit A116 determines that the object is the same, it updates the object's coordinate information and associates the object. Then, the association unit A116 outputs the updated object's coordinate information and movement direction to the operation determination unit A117. Thereafter, the angle of view adjustment device B103 proceeds to the processing of S120.

[0087] In S120, the operation determination unit A117 determines whether to adjust the angle of view based on the object coordinate information and the object's movement direction input from the association unit A116, or whether to operate the angle of view based on user operation information input from the input reception unit A109. If there is user operation information, the operation determination unit A117 adjusts the angle of view based on the PTZ coordinate values. On the other hand, if there is no user operation information, the operation determination unit A117 determines to operate the angle of view based on the object's coordinate information so that the center of the object becomes the center of the angle of view. The operation determination unit A117 then performs a PT operation in the object's movement direction input from the association unit A116. However, if there is no input of the movement direction from the association unit A116, the operation determination unit A117 does not perform a PT operation. Furthermore, if there is no input from the association unit A116, the operation determination unit A117 performs a PT operation in the same direction as the previous movement direction. The operation determination unit A117 selects a zoom magnification from predetermined zoom magnifications according to the amount of movement of the object, determines to perform a zoom operation according to the selected zoom magnification, and outputs the view angle operation information to the view angle control unit A118.

[0088] Next, in S121, the angle-of-view control unit A118 controls the angle of view based on the image input from the image acquisition unit A105 and the angle-of-view operation information input from the operation determination unit A117. Then, the angle-of-view control unit A118 outputs the image input from the image acquisition unit A105 after the angle-of-view control has been performed to the image output unit A119.

[0089] Next, in S122, the image output unit A119 outputs the image input from the angle-of-view control unit A118 to the monitor device A104. Thereafter, the angle-of-view adjustment device B103 proceeds to the process of S123. In S123, the angle-of-view adjustment device B103 determines whether a tracking system ON / OFF switch (not shown) has been operated by the user to stop the angle-of-view adjustment device. If the angle-of-view adjustment device B103 determines that a stop operation has not been performed (NO in S123), it returns the process to S104, and if it determines that a stop operation has been performed (YES in S123), it ends the angle-of-view adjustment process.

[0090] As described above, in the tracking system B100 of the second embodiment, when an unintended object is tracked and a user operation is input, position information of the target object is acquired and stored as bird's-eye coordinates. As a result, according to the second embodiment, objects whose positions are similar to the erroneously tracked object can be excluded from tracking targets during subsequent automatic tracking, and control can be performed to prevent erroneous tracking from being performed again.

[0091] In the second embodiment, the angle-of-view adjustment device B103 does not detect the feature amounts of the object, but may perform a similarity determination using the feature amounts as in the first embodiment in addition to the position information of the object, and may exclude non-target objects based on the result of the similarity determination based on the feature amounts and the position.

[0092] <Third embodiment> Next, a third embodiment will be described. Fig. 12 is a block diagram showing an example of the functional configuration of a tracking system C100 including a field-of-view adjustment device C103 according to a third embodiment. Fig. 13 is a flowchart showing the processing flow in the field-of-view adjustment device C103 according to the third embodiment. The flowchart in Fig. 13 will be described later. In the third embodiment, descriptions of the same configurations and processing as those in the above-described embodiments will be omitted as appropriate.

[0093] As described above, the tracking system C100 of the third embodiment adjusts the angle of view so that an object detected from a captured image does not fall outside the angle of view when no user operation information is input, and adjusts the angle of view based on a PTZ value based on the user operation information when user operation information is input. Similarly to the above, the third embodiment also prevents an object that was a tracking target when user operation information was input from becoming a tracking target again. In the third embodiment, a clipped image including the object that was a tracking target is presented to the user, and when the user, upon viewing the presented clipped image, inputs an instruction to make the object a non-target object, the feature amount of the object is retained. That is, in the third embodiment, information about the clipped image and user input information regarding the image are also used as object-related information. A tracking system C100 of the third embodiment includes an imaging device A101, an input device A102, a field angle adjustment device C103, a monitor device A104, and a holding determination input device C120.

[0094] The hold determination input device C120 is a device that receives hold determination input from a user, and is configured to have a touch panel, a mouse, etc. The hold determination input device C120 outputs hold determination information input by the user to the view angle adjustment device C103. The hold determination information will be described in detail later.

[0095] As described above, when an image is input from the imaging device A101, the angle-of-view adjustment device C103 detects an object in the image and adjusts the angle of view by PTZ control so that the object does not leave the angle of view. Furthermore, when the angle-of-view adjustment device C103 acquires user operation information from the input device A102, it retains the feature amounts of the detected object and controls the angle of view based on the user operation information, as in the first embodiment. Then, when the input of user operation information ceases and the angle-of-view adjustment device C103 selects a target for angle-of-view manipulation from the detected objects again, if the angle-of-view adjustment device C103 finds an object with feature amounts similar to those of the previously retained object, it does not include the object in the tracking target. In the third embodiment, whether to retain the feature amounts of an object is determined based on retention determination information input by the user.

[0096] The view angle adjustment device C103 according to the third embodiment includes an image acquisition unit A105, an object detection unit A106, a feature acquisition unit A107, a target exclusion unit A108, a target selection unit C113, an information recording unit A114, a direction estimation unit A115, and a matching unit A116. The view angle adjustment device C103 also includes an input reception unit A109, an input retention unit A110, a non-target determination unit A111, an operation determination unit A117, a view angle control unit A118, and an image output unit A119. Furthermore, in the third embodiment, the view angle adjustment device C103 also includes a retention determination reception unit C121, a non-target retention unit C112, a target clipping unit C122, and a non-target candidate presentation unit C123. The view angle adjustment device C103 outputs an image acquired after the view angle adjustment to the monitor device A104.

[0097] The holding determination receiving unit C121 outputs the holding determination information input from the holding determination input device C120 to the non-target holding unit C112. The target clipping unit C122 generates an image clipped from the object region based on the coordinate information of the object input from the target selection unit C113. Then, the target clipping unit C122 outputs the image clipped from the object region, as well as the feature amount and coordinate information of the object, to the non-target candidate presentation unit C123.

[0098] The non-target candidate presentation unit C123 outputs the cut-out image input from the target cut-out unit C122 to the monitor device A104. As a result, the cut-out image is displayed on the monitor device A104, allowing the user to view the cut-out image. In addition, the non-target candidate presentation unit C123 outputs coordinate information of the object corresponding to the cut-out image to the non-target holding unit C112.

[0099] If the non-target holding unit C112 determines that the object corresponding to the object coordinate information input from the non-target candidate presentation unit C123 is a non-target object based on the holding determination information input from the holding determination receiving unit C121, the non-target holding unit C112 holds the feature amount of the object. Then, the non-target holding unit C112 outputs the feature amount information of the held object to the target exclusion unit A108.

[0100] FIG. 14 is a diagram used to explain an example of presentation of a cropped image, and shows an example of a display screen D701 of the monitor device A104. The display screen D701 displays a video P700 based on an image output from the image output unit A119, detected objects P701, P702, and P703, a rectangle P704 of a selected target object, and a cropped image presentation area P705. A cropped image P707 including an object P706 is displayed in the cropped image presentation area P705. When a user inputs an operation, the rectangle P704 of the current target object is cropped and displayed in the cropped image presentation area P705 as the cropped image P707. The non-target candidate presentation unit C123 outputs the coordinate information and feature amount of the input object to the non-target storage unit C112 and outputs the cropped image to the monitor device A104.

[0101] Hereinafter, the procedure of the angle of view adjustment process in the tracking system C100 of the third embodiment will be described with reference to the flowchart of FIG. When the tracking system C100 is activated by a user operation, it starts the processing of the flowchart in Fig. 13. Then, in S201, the image acquisition unit A105 of the angle of view adjustment device C103 acquires image information from the imaging device A101 and outputs the image information to the object detection unit A106 and the angle of view control unit A118.

[0102] Next, in S202, the object detection unit A106 performs object detection processing on the image input from the image acquisition unit A105. The object detection unit A106 outputs coordinate information of the detected object to the feature acquisition unit A107. Next, in S203, the feature acquisition unit A107 acquires feature amounts of the object detected by the object detection unit A106 from the coordinate information of the object. Then, the feature acquisition unit A107 outputs the acquired feature amounts and the coordinate information of the object to the target exclusion unit A108.

[0103] Next, in S204, the object exclusion unit A108 compares the feature amount of the object input from the feature acquisition unit A107 with the feature amount of the object input from the non-object storage unit A112 to exclude the non-target object. Next, in S205, the target selection unit C113 selects a target object if the target object has not been selected or if the correspondence has not been established, based on the coordinate information and feature amount of the object input from the target selection unit C113. Then, the target selection unit C113 outputs the coordinate information and feature amount of the selected object to the target cutout unit C122 and the information recording unit A114.

[0104] Next, in S206, the input receiving unit A109 determines whether or not user operation information has been input from the input device A102. If user operation information has been input (YES in S206), the input receiving unit A109 outputs the user operation information to the input storage unit A110 and the operation determination unit A117, and then the view angle adjustment device C103 proceeds to the processing of S207. On the other hand, if user operation information has not been input to the input receiving unit A109 (NO in S206), the view angle adjustment device C103 proceeds to the processing of S211.

[0105] In S207, the input storage unit A110 records the user operation information input from the input reception unit A109. Then, the input storage unit A110 outputs the user operation information of the current frame and the one frame before that to the non-eligible determination unit A111.

[0106] Next, in S208, the non-target determination unit A111 determines whether to retain information about a non-target object based on the difference between the user operation information for the current frame and the previous frame input from the input retention unit A110. The non-target determination unit A111 then determines to retain a non-target object only if there was no input information in the previous frame and there was input only in the current frame (YES in S208), and the view angle adjustment device C103 then proceeds to the processing of S214. On the other hand, if there was input information in the previous frame or there was no input in the current frame, the non-target determination unit A111 determines not to retain a non-target object (NO in S208) and outputs the input determination result to the non-target candidate presentation unit C123. The view angle adjustment device C103 then proceeds to the processing of S209.

[0107] In S209, the target clipping unit C122 generates an image of the object clipped out based on the coordinate information of the object input from the target selection unit C113. Then, the target clipping unit C122 outputs the generated image, the feature amount of the object, and the coordinate information to the non-target candidate presentation unit C123.

[0108] Next, in S210, the non-target candidate presentation unit C123 presents the cut-out image input from the target cut-out unit C122 to the monitor device A104 to the user. Then, the non-target candidate presentation unit C123 outputs the coordinate information and feature amount of the input object to the non-target storage unit C112. Thereafter, the angle of view adjustment device C103 proceeds to the processing of S214.

[0109] Also, if the process proceeds to S211, the information recording unit A114 records the coordinate information of the object input from the target selection unit C113. Then, the information recording unit A114 outputs the coordinate information of the object in the current frame and the frame immediately before to the direction estimation unit A115. Next, in S212, the direction estimation unit A115 calculates a movement vector of the object from the coordinate information of the object in the current frame and the previous frame input from the information recording unit A114, and estimates the movement direction of the object based on the calculated vector.The direction estimation unit A115 then outputs the estimated movement direction of the object to the association unit A116.

[0110] Next, in S213, the association unit A116 determines whether the object is the same between the frames based on the object's movement direction estimated by the direction estimation unit A115 and the object's coordinate information in the current and previous frames. If the association unit A116 determines that the object is the same, it updates the object's coordinate information and associates the object. Then, the association unit A116 outputs the updated object's coordinate information and movement direction to the operation determination unit A117. Thereafter, the angle of view adjustment device C103 proceeds to the process of S214.

[0111] In S214, the hold determination reception unit C121 determines whether or not the user's non-target object determination information has been input from the hold determination input device C120. If the user's non-target object determination information has been input (YES in S214), the hold determination reception unit C121 outputs the hold determination information to the non-target object retention unit C112, and then the view angle adjustment device C103 proceeds to the processing of S215. On the other hand, if the user's non-target object determination information has not been input (NO in S214), the view angle adjustment device C103 proceeds to the processing of S216.

[0112] In S215, the non-target holding unit C112 determines, based on the input determination result from the retention determination receiving unit C121, that the object corresponding to the object coordinate information input from the non-target candidate presentation unit C123 is a non-target object, holds the feature amount information of the object, and outputs the held feature amount information of the object to the target exclusion unit A108.

[0113] Next, in S216, the operation determination unit A117 determines whether to adjust the angle of view based on the object coordinate information and the object's movement direction input from the association unit A116, or to operate the angle of view based on user operation information input from the input reception unit A109. If user operation information is present, the operation determination unit A117 determines to adjust the angle of view using PTZ coordinate values ​​based on the user operation information. On the other hand, if there is no user operation information, the operation determination unit A117 adjusts the angle of view based on the object's coordinate information so that the center of the object becomes the center of the angle of view, and determines to perform PT control in the object's movement direction input from the association unit A116. However, if the association unit A116 inputs that there is no movement direction, the operation determination unit A117 does not perform PT operation. Furthermore, if there is no input from the association unit A116, the operation determination unit A117 performs PT operation in the same direction as the previous movement direction. The operation determination unit A117 selects a zoom magnification from predetermined zoom magnifications according to the amount of movement of the object, determines to perform a zoom operation according to the selected zoom magnification, and outputs the view angle operation information to the view angle control unit A118.

[0114] Next, in S217, the angle-of-view control unit A118 controls the angle-of-view adjustment based on the image input from the image acquisition unit A105 and the angle-of-view operation information input from the operation determination unit A117. Then, the angle-of-view control unit A118 outputs the image after the angle-of-view adjustment to the image output unit A119. Next, in S218, the image output unit A119 outputs the image input from the angle of view control unit A118 to the monitor device A104. Next, in S219, the field-angle adjustment device C103 determines whether a tracking system ON / OFF switch (not shown) has been operated by the user to stop the field-angle adjustment device. If the field-angle adjustment device C103 determines that a stop operation has not been performed (NO in S219), it returns the process to S202, but if it determines that a stop operation has been performed (YES in S219), it ends the field-angle adjustment process.

[0115] As described above, in the tracking system C100 of the third embodiment, when an unintended object is tracked and a user operation input is made, the feature amount of the target object is acquired and a cropped image is presented to the user. Therefore, the user can determine whether the object is a tracking target. In the third embodiment, based on the result of this determination, it is then determined whether or not to retain the feature amount information of the non-target object. This allows the non-target object to be correctly excluded from the tracking targets during subsequent tracking, enabling control to be performed to prevent erroneous tracking from being performed again.

[0116] In the third embodiment, the similarity determination may be performed by adding the object position information as described in the second embodiment, and non-target objects may be excluded based on the results of the similarity determination.

[0117] Although the embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof. For example, the object exclusion unit may be configured to exclude non-target people by person detection, rather than excluding non-target objects using object detection information from the object detection unit.

[0118] The angle-of-view adjustment device of the above-described embodiment may be realized by an information processing device such as a personal computer connected to the imaging device 100. The computer then performs the same angle-of-view adjustment process as described in the above-described embodiments. The computer in this example executes software program code that implements the angle-of-view adjustment process of this embodiment. While the hardware configuration is not illustrated, the computer implementing the angle-of-view adjustment device of this embodiment includes a CPU, ROM, RAM (random access memory), an auxiliary storage device, a display unit, an operation unit, a communication I / F, and a bus. The CPU controls the entire computer and executes the above-described angle-of-view adjustment process using computer programs and data stored in the ROM and RAM. The angle-of-view adjustment device of this embodiment may also include one or more dedicated hardware components separate from the CPU, and may be configured to perform at least some of the processing performed by the CPU. Examples of dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM stores programs that do not require modification. The RAM temporarily stores programs and data supplied from the auxiliary storage device and data supplied from the outside via the communication I / F. The auxiliary storage device is composed of a hard disk drive or the like and stores various data such as image data and control parameters. The display unit is composed of, for example, an LCD display or an LED display, and displays a GUI for the user to operate the field of view adjustment device. The operation unit is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and inputs various instructions to the CPU in response to user operations. The CPU also functions as a display control unit that controls the display unit and an operation control unit that controls the operation unit. The communication I / F is used for communication with devices external to the field of view adjustment device. For example, if the field of view adjustment device is further connected to an external device via a wired connection, a communication cable is connected to the communication I / F. If the field of view adjustment device has the function of wirelessly communicating with an external device, the communication I / F is equipped with an antenna. The bus connects each unit of the field of view adjustment device to transmit information.In this embodiment, the external device connected to the angle-of-view adjustment device is the above-mentioned imaging device, another information processing device, etc. Furthermore, although the display unit and the operation unit are assumed to exist inside the angle-of-view adjustment device, the display unit may exist as the above-mentioned monitor device and the operation unit may exist as an input device as separate devices outside the angle-of-view adjustment device.

[0119] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0120] A100: Tracking system, A101: Imaging device, A102: Input device, A103: View angle adjustment device, A106: Object detection unit, A107: Feature acquisition unit, A108: Target exclusion unit, A113: Target selection unit, A117: Operation determination unit, A118: View angle control unit

Claims

1. a detection means for detecting an object from an image captured by a PTZ-driveable imaging device; a tracking means for adjusting an angle of view of the imaging device to track the object so that the object is within the angle of view of the imaging device; an acquisition means for acquiring a feature amount of an object detected from the image; A receiving means for receiving a PTZ operation input from a user; a determination means for determining, when a PTZ operation input is received from the user, that an object whose feature amount has been acquired before the PTZ operation input is not a target of tracking; a storage means for storing a feature amount of an object determined by the determination means not to be a tracking target as a feature amount of the object not to be tracked; an exclusion means for excluding an object corresponding to the feature amount stored in the storage means from the tracking target, The exclusion means is a first feature amount that is a feature amount of an object that is not a tracking target and is stored by the storing means, and that is a feature amount of an object that is acquired before the PTZ operation input, is compared with a second feature amount of an object that is acquired after the PTZ operation input; if the first feature amount and the second feature amount are not similar, the object corresponding to the second feature amount is determined to be the tracking target; and if the first feature amount and the second feature amount are similar, the object corresponding to the second feature amount is excluded from the tracking targets.

2. 2. The imaging control device according to claim 1, wherein the PTZ operation input from the user is an operation input for adjusting the angle of view of the imaging device.

3. a detection step of detecting an object from an image captured by a PTZ-driveable imaging device; a tracking step of adjusting an angle of view of the imaging device to track the object so that the object is within the angle of view of the imaging device; an acquisition step of acquiring feature amounts of the object detected from the image; a receiving step of receiving a PTZ operation input from a user; a determining step of determining, when a PTZ operation input is received from the user, that an object whose feature amount has been acquired before the PTZ operation input is not a target of tracking; a storing step of storing a feature amount of the object determined not to be a tracking target in the determining step as a feature amount of the object not to be tracked; an exclusion step of excluding an object corresponding to the feature amount held in the holding step from the tracking target, In the removing step, a first feature amount that is a feature amount of an object that is not a tracking target and that is retained by the retaining step and that is a feature amount of an object that is acquired before the PTZ operation input, and a second feature amount of an object that is acquired after the PTZ operation input; and if the first feature amount and the second feature amount are not similar, determining that the object corresponding to the second feature amount is the tracking target; and if the first feature amount and the second feature amount are similar, excluding the object corresponding to the second feature amount from the tracking target.

4. A program for causing a computer to function as each of the means of the imaging control device according to claim 1 or 2.

Images