Control device, control method, and program

Abstract

To enhance the possibility of rediscovering a tracking target even when the tracking target cannot be tracked.SOLUTION: A control device comprises: control means for controlling an imaging range of imaging means so as to track a tracking target; and setting means for setting a search range on the basis of a range that is set by a user, a user operation for controlling the imaging range, or a past position of the tracking target. When the tracking target can no longer be tracked, the control means controls the imaging range so as to search for the tracking target in the search range.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a method for controlling an imaging device. [Background technology]

[0002] Conventionally, imaging devices capable of pan / tilt control have been equipped with a person detection function and an automatic tracking function that controls pan / tilt so that the subject detected from the captured image is kept near the center of the imaging range.

[0003] Furthermore, the automatic tracking function has a function to search for the subject again if the subject becomes lost due to its movement and it is no longer possible to detect the subject.Patent Document 1 discloses a technology for searching for the tracking target by zooming out and increasing the angle of view when the tracking target is lost. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-46321 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, if the tracking target has moved away from the position where it was lost, simply zooming out may not allow the tracking target to be found again.

[0006] Therefore, an object of the present invention is to improve the possibility of rediscovering a tracking target even when tracking of the tracking target subject becomes impossible. [Means for solving the problem]

[0007] In order to solve the above problems, the control device of the present invention has the following configuration: Namely, it has a control means for controlling the imaging range of the imaging means so as to track a tracking target, and a setting means for setting a search range based on a range set by a user, a user operation for controlling the imaging range, or a past position of the tracking target, and when it becomes impossible to track the tracking target, the control means controls the imaging range so as to search for the tracking target within the search range. The search range is divided into a plurality of blocks, and the control means determines a route for controlling the imaging range in the search range based on the existence time of each block of the tracking target, and controls the imaging range according to the determined route. do. [Effects of the Invention]

[0008] According to the present invention, even when it becomes impossible to track a subject to be tracked, it is possible to improve the possibility of rediscovering the subject to be tracked. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a system. [Figure 2] FIG. 1 is a diagram illustrating an example of the appearance of an imaging device. [Figure 3] FIG. 2 is a diagram illustrating functional blocks of the imaging device. [Figure 4] FIG. 10 is a diagram for explaining a search process. [Figure 5] FIG. 10 is a diagram for explaining a search process. [Figure 6] 10 is a flowchart showing the flow of a tracking process. [Figure 7] 10 is a flowchart showing the flow of a search process. [Figure 8] 10 is a flowchart showing the flow of a search range setting process. [Figure 9] FIG. 10 is a diagram for explaining a search range setting process. [Figure 10] 10 is a flowchart showing the flow of a search range setting process. [Figure 11] FIG. 10 is a diagram for explaining a search range setting process. [Figure 12] FIG. 10 is a diagram for explaining the process of determining a search route. [Figure 13]FIG. 2 is a diagram illustrating the hardware configuration of each device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings.

[0011] (Embodiment 1) 1 is a diagram showing the system configuration of this embodiment. The system of this embodiment includes an image capturing device 100, an information processing device 200, a display 210, and a network 300.

[0012] The imaging device 100 and the information processing device 200 are connected to each other via a network 300. The network 300 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark).

[0013] The network 300 may be realized by the Internet, a wired local area network (LAN), a wireless LAN, a wide area network (WAN), or the like.

[0014] The imaging device 100 is a device that captures images and also functions as a control device capable of changing the imaging range by controlling at least one of panning, tilting, and zooming. The imaging device 100 transmits image data of the captured image, information on the date and time the image was captured, identification information for identifying the imaging device 100, and information on the imaging range of the imaging device 100 to an external device such as an information processing device 200 via a network 300. The information processing device 200 is, for example, a client device such as a personal computer on which a program for implementing the processing functions described below is installed. Note that although the system according to this embodiment uses one imaging device 100, multiple imaging devices 100 may also be used. That is, multiple imaging devices 100 may be connected to the information processing device 200 via the network 300. In this case, the information processing device 200 determines which of the multiple imaging devices 100 captured the transmitted image, for example, using identification information associated with the transmitted image.

[0015] The display 210 is configured with an LCD (Liquid Crystal Display) or the like, and displays images captured by the imaging device 100. The display 210 is connected to the information processing device 200 via a display cable that complies with a communication standard such as HDMI (High Definition Multimedia Interface) (registered trademark). The display 210 and the information processing device 200 may be provided in a single housing.

[0016] Next, an imaging device 100 according to this embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is an example of an external view of the imaging device 100 according to this embodiment. FIG. 3 is an example of functional blocks of the imaging device 100 and an information processing device 200 according to this embodiment. Among the functional blocks of the imaging device 100 shown in FIG. 3, the functions of the image processing unit 112, image analysis unit 113, system control unit 114, pan / tilt / zoom control unit 115, communication unit 116, etc. are realized as follows. That is, the functions are realized by a CPU (Central Processing Unit) 1300 of the imaging device 100 executing a computer program stored in a ROM (Read Only Memory) 1320 of the imaging device 100, which will be described later with reference to FIG. 13.

[0017] The direction in which the optical axis of the lens 101 faces is the imaging direction of the imaging device 100, and a light beam that passes through the lens 101 forms an image on an imaging element of an imaging unit 111 of the imaging device 100. In addition, the lens driving unit 102 is configured with a drive system that drives the lens 101 and changes the focal length of the lens 101. The lens driving unit 102 is controlled by a pan / tilt / zoom control unit 114.

[0018] The pan driving unit 103 is composed of a mechanical driving system that performs panning operations and a motor that is a driving source, and drives to control rotational driving for rotating the imaging direction of the imaging device 100 in a pan direction 105. The pan driving unit 103 is also controlled by a pan / tilt / zoom control unit 115.

[0019] The tilt driving unit 104 is composed of a mechanical drive that performs tilt operation and a motor that is a driving source, and drives to control rotational drive for rotating the imaging direction of the imaging device 100 in a tilt direction 106. The tilt driving unit 104 is controlled by a pan / tilt / zoom control unit 115.

[0020] The imaging unit 111 is composed of an imaging element (not shown), such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The imaging unit 111 photoelectrically converts the subject image formed through the lens 101 to generate an electrical signal. The image processing unit 112 performs image processing such as converting the electrical signal photoelectrically converted by the imaging unit 111 into a digital signal and compression encoding, and generates image data of the captured image.

[0021] The image analysis unit 113 performs processing to detect a specific subject to be tracked from the image of the image data generated by the image processing unit 102 (in other words, the image captured by the imaging unit 111) using well-known techniques such as deep learning. For example, a person is detected as the specific subject. The image analysis unit 110 may detect a specific person registered in advance as a tracking target by matching feature data of the image of the face portion. As a method for detecting the person to be tracked from the image, pattern matching or a learning model learned by machine learning may be used.

[0022] The pan-tilt-zoom control unit 114 controls the pan, tilt, and zoom of the imaging device 100 by controlling the pan driving unit 103, the tilt driving unit 104, and the lens driving unit 102 based on instructions transmitted from the system control unit 113.

[0023] The communication unit 116 communicates with the information processing device 200 via an I / F 1340, which will be described later with reference to Fig. 13. For example, the communication unit 116 transmits image data of an image captured by the imaging device 100 to the information processing device 200 via the network 300. The communication unit 116 also transmits information indicating the current imaging range of the imaging device 100. The communication unit 116 also receives control commands, which are commands for controlling the imaging device 100, transmitted from the information processing device 200, and transfers the control commands to the system control unit 113.

[0024] The system control unit 114 controls the entire imaging device 100 in accordance with processing executed by the CPU 900, which will be described later with reference to FIG. 9 , and performs, for example, the following processing. That is, the system control unit 113 analyzes a control command for controlling the imaging device 100 transmitted from the information processing device 200 and performs processing according to the control command. The system control unit 114 also calculates changes to the pan value, tilt value, and zoom value based on the position of the tracking target in the image detected by the image analysis unit 113 so that the tracking target appears at a specified size and in the center of the captured image. Note that it is not always necessary to calculate all of the changes to the pan value, tilt value, and zoom value; depending on the situation, the system control unit 114 may calculate a change amount for one of the values ​​or two of the values. Then, when the system control unit 114 calculates a change amount for the pan value, it instructs the pan operation control unit 114 to change the pan value of the imaging device 100 by the calculated change amount for the pan value. Furthermore, when the system control unit 114 calculates the amount of change in the tilt value, it instructs the pan / tilt / zoom control unit 115 to change the tilt value of the imaging device 100 by the calculated amount of change in the tilt value. Furthermore, when the system control unit 114 calculates the amount of change in the zoom value, it instructs the lens control unit 115 to change the zoom value of the imaging device 100 by the calculated amount of change in the zoom. This allows the imaging device to track and image the tracking target.

[0025] However, depending on the movement of the tracking target, the image analysis unit 113 may not be able to detect the tracking target from the captured image, making it impossible to track the tracking target. If the tracking target is lost (in other words, if it becomes impossible to track the tracking target), the system control unit 114 searches by controlling the pan / tilt / zoom so as to sequentially capture images within the set search range, and resumes tracking if the tracking target is found again.

[0026] It should be noted that the imaging range in this embodiment is determined by the pan value, tilt value, and zoom value of the imaging device 100. The pan value is the angle of the imaging direction (optical axis) in a pan direction 105 of the imaging device 100 when, for example, one of the drive ends of the pan drive unit 103 is set to 0°. The tilt value is the angle of the imaging direction (optical axis) in a tilt direction 106 of the imaging device 100 when, for example, one of the drive ends of the tilt drive unit 104 is set to 0°. The zoom value of the imaging device 100 when an image is captured by the imaging device 100 is calculated from the focal length of the lens 101.

[0027] Next, information processing of the information processing device 200 according to this embodiment will be described with reference to the functional blocks of the information processing device 200 shown in Fig. 3. Note that each function of the information processing device 200 is realized as follows using a ROM 1320 and a CPU 1300, which will be described later with reference to Fig. 13. That is, each function shown in Fig. 3 is realized by the CPU 1300 of the information processing device 100 executing a computer program stored in the ROM 1320 of the information processing device 100.

[0028] The display control unit 201 displays on the display 210 a GUI (Graphical User Interface) for setting images captured by the imaging device 100 and shot functions. The operation reception unit 202 receives information about user operations via an input device (not shown), such as a keyboard, mouse, or touch panel. The input unit may be a button, mouse, joystick, or the like, and receives various operations from users. For example, the display control unit 201 displays on the display 210 a GUI for setting a search range, and the operation reception unit 202 receives information about user operations on the GUI displayed on the display 210. The system control unit 203 transmits a control command to the remote camera via the communication unit 204 in response to the user's operation. An example of a method for setting a search range will now be described with reference to FIG. 4. The wide-angle image 401 shown in FIG. 4 is an image captured when the imaging device 100 is in a wide-angle state (in other words, when the zoom is at the WIDE end). Instead of wide-angle image 401, a panoramic image based on multiple images obtained by sequentially capturing images while changing the imaging range of imaging device 100 may be displayed. Wide-angle image 401 shown in FIG. 4 is displayed on display 210 by display control unit 201. Search range 402 displayed superimposed on wide-angle image 401 indicates the search range currently set based on a user operation. Icons 403 are displayed at the four corners of search range 402, and search range 402 can be changed by a user operation of dragging icon 403 with a mouse. Information about the set search range 402 is transmitted to imaging device 100 in response to the dragging operation of icon 403, and system control unit 114 of imaging device 100 sets search range 402 based on the transmitted information as the search range to be used in the search process described below.

[0029] The communication unit 204 transmits various setting commands and control commands for the imaging device 100 transmitted from the system control unit 203 to the imaging device 100 via an I / F 1340, which will be described later with reference to Fig. 13. The communication unit 204 also receives image data transmitted from the imaging device 100 and responses from the imaging device 100 in response to commands transmitted from the information processing device 200 to the imaging device 100, and transmits these to the system control unit 203. The communication unit 204 also transmits information on a currently set search range to the imaging device 100. The imaging device 100 acquires the information on the search range transmitted from the information processing device 200, and sets the search range as a search range to be used in a search process, which will be described later. The storage unit 205 also stores information related to the search range, image data of images acquired by the communication unit 204, and the like.

[0030] The system control unit 203 generates various setting commands and control commands based on the user operations received by the operation receiving unit 202 , and transmits them to the imaging device 100 via the communication unit 204 .

[0031] Here, referring to FIG. 5 , a process for calculating a route (search route) of a search process to be executed when it becomes impossible to track a subject will be described, based on the current imaging range and search range of the imaging device 100. The current imaging range 404 shown in FIG. 5 is the current imaging range of the imaging device 100. When it becomes impossible to track a tracking target, the system control unit 114 of the imaging device 100 calculates an initial search route for controlling pan, tilt, and zoom so that the imaging range is changed to an initial range 405, based on the search range 402 acquired from the information processing device 200. The initial range is determined by zooming out within a range in which the tracking target can be detected, and then the system control unit 114 moves the pan or tilt value so that the current imaging range 404 touches the closest side among the top, bottom, left, and right sides of the search range 402. In the example shown in FIG. 5 , the top side of the search range 402 is closest to the current imaging range 404, so the initial range 405 shown in FIG. 5 is selected. Route 411 is the route determined in this manner for moving from the current imaging range 404 to the initial range 405. Next, the system control unit 114 identifies the farthest side of the left or right side of the search range 402 from the initial range 405, and changes the pan value to control the imaging range so that the identified side borders the side of the imaging range. Route 406 is the route determined in this manner by the system control unit 114. Next, similarly, the system control unit 114 controls the imaging range along route 407, which is a route such that the right side of the imaging range borders the right side of the search range 402 and the bottom side of the imaging range reaches the bottom side of the search range 402. When the imaging range is controlled along route 407, the system control unit 114 changes the tilt value. Then, the system control unit 114 controls the imaging range along route 408, which is a route for moving the imaging range so that the bottom side of the search range 402 borders the bottom side of the imaging range and the left side of the imaging range reaches the left side of the search range 402. The system control unit 114 changes the pan value along route 408. Next, the system control unit 114 controls the imaging range along route 409, which is a route that moves the imaging range so that it contacts the left side of the search range 402, and in which the top side of the imaging range reaches the top side of the search range 402. At this time, the system control unit 114 changes the tilt value.Next, the system control unit 114 controls the imaging range along a route 410 that moves the imaging range so that the top edge of the imaging range contacts the top edge of the search range 402, and that brings the imaging range to the initial range 405. At this time, the system control unit 114 instructs the pan / tilt control unit 115 to change the pan value. The system control unit 114 controls the imaging range along the search routes 406 to 410 described above, and the image analysis unit 113 executes a process of detecting a tracking target from images captured sequentially during this process. Note that when a tracking target is detected, the search operation ends and tracking of the tracking target resumes. The image analysis unit 110 may be configured to match and detect a specific person registered as a tracking target from feature data of the image of the face portion, and resume the tracking operation only when a match is found.

[0032] Note that this search example is an example in which the vertical size of the imaging range at the zoom magnification when searching for the tracking target is equal to or greater than half the vertical size of the search range 402. If the vertical size of the search range 402 is even larger, the system control unit 114 determines a search route that searches the entire range of the search range 402 by repeatedly moving back and forth from the left end to the right end of the search range 402 while changing the tilt value.

[0033] Next, a process for tracking and capturing an image of a tracking target by the imaging device 100 will be described with reference to the flow shown in Fig. 6. It is assumed that none of the pan, tilt, and zoom of the imaging device is in operation at the start of the process according to the flow in Fig. 6. It is assumed that the process of the flowchart shown in Fig. 13 is executed by the functional blocks shown in Fig. 3 which are realized by the CPU 1300 of the imaging device 100 executing a computer program stored in the ROM 1320 of the imaging device 100.

[0034] First, in S601, the image analysis unit 113 detects a subject to be tracked from the image of the most recent frame. Next, in S602, the image analysis unit 113 determines whether the subject to be tracked is included in the image. If the subject to be tracked is detected by the image analysis unit 113 (Yes in S602), the process proceeds to S603, and the processing proceeds to step S403. If the subject to be tracked is not detected (No in S602), the process proceeds to S606. Note that the image analysis unit 113 may only treat subjects that appear within a predetermined area of ​​the image as the tracking target. Furthermore, the image analysis unit 113 may match a specific person registered as the tracking target based on feature data of the image of the face portion, and only treat subjects determined to be the specific person as the tracking target.

[0035] In S603, the system control unit 114 calculates the amount of change in the pan value and / or tilt value so that the tracking target appears at the center of the image, based on at least one of the current pan value and tilt value and the position of the tracking target in the image detected by the image analysis unit 110. The system control unit 103 then instructs the pan / tilt / zoom control unit 115 to change the current pan value and / or tilt value by the amount of change so that the current pan value and / or tilt value becomes the calculated pan value and / or tilt value. As a result, the pan / tilt / zoom control unit 115 controls the pan driving unit 103 and / or tilt driving unit 104 to change the current pan value and / or tilt value by the amount of change instructed by the system control unit 114.

[0036] Next, in S604, the system control unit 114 determines the amount of change in the zoom value so that the tracking target is displayed in the image at a predetermined size. For example, the system control unit 114 first determines whether the size of the tracking target in the image is appropriate for the size of the image. For example, the system control unit 114 determines whether the ratio of the number of vertical pixels of the circumscribing rectangle of the tracking target to the number of vertical pixels of the captured image exceeds a first ratio (e.g., 80%). As a result, if the number of vertical pixels of the circumscribing rectangle of the tracking target to the number of vertical pixels of the captured image is equal to or greater than the first ratio, the system control unit 114 determines that zooming out is necessary so that the ratio becomes less than a second ratio. The system control unit 114 also determines whether the ratio of the number of vertical pixels of the circumscribing rectangle of the tracking target to the number of vertical pixels of the captured image is less than a second ratio (e.g., 50%). If the ratio of the number of vertical pixels of the circumscribing rectangle of the tracking target to the number of vertical pixels of the captured image is less than a second ratio, the system control unit 114 determines that zooming in is necessary so that the ratio becomes equal to or greater than the second ratio. Here, the system control unit 103 may allow the user to set the first ratio and the second ratio in advance.

[0037] The system control unit 114 then instructs the pan-tilt-zoom control unit 115 to change the current zoom value of the imaging device by the determined change amount of the zoom value. The pan-tilt-zoom control unit 115 controls the lens driving unit 102 to change the zoom value by the change amount instructed by the system control unit 114.

[0038] Next, in S605, the system control unit 114 calculates the size of the tracking target in the captured image after zoom control from the size of the tracking target detected in S601 and the change in the zoom value in S604. For example, suppose zooming in is performed in S604 and the change in the zoom value is 1.5 times. If the vertical size of the tracking target before zooming in is 40% of the vertical size of the captured image, the height of the tracking target in the captured image after zooming in will be 60% of the vertical size of the image. Also, suppose zooming out is performed in step S604 and the change in the zoom value is to expand the angle of view vertically and horizontally by 150%. If the vertical size of the tracking target before zooming out is 90% of the vertical size of the captured image, the vertical size of the tracking target in the captured image after zooming out will be 60% of the vertical size of the captured image.

[0039] Then, the system control unit 114 stores the size of the tracking target in the image captured after zoom control, which is calculated in this manner, in a memory included in the system control unit 114 or in an external memory (not shown). After the process of S605, the process returns to S601.

[0040] In S606, when the state where the tracking target was detected changes to the state where it is not detected (when the tracking target cannot be tracked), the system control unit 114 executes a process of searching for the tracking target that has been lost (search process). Details of the process in S606 will be described later with reference to the flow in FIG. 7.

[0041] In S607, the system control unit 114 determines whether the image analysis unit 113 has been able to detect the tracking target from the image through the search process, and if the detection has been successful (Yes in S607), the process proceeds to S603. On the other hand, if the image analysis unit 113 has not been able to detect the tracking target from the captured image through the search process (No in S607), the process proceeds to S608. In S608, the system control unit 114 erases the size of the tracking target recorded in memory. In S114, the system control unit 114 instructs the pan / tilt / zoom control unit 115 to set the pan value, tilt value, and zoom value to the preset initial imaging range.

[0042] Details of the search process in S606 will now be described with reference to Fig. 7. Note that the process of the flowchart shown in Fig. 13 is executed by the functional blocks shown in Fig. 3 which are realized by the CPU 1300 of the imaging device 100 executing a computer program stored in the ROM 1320 of the imaging device 100.

[0043] In S701, the image analysis unit 113 executes a detection process for detecting a tracking target from a captured image for a predetermined period of time. The predetermined period of time may be a number of seconds set in advance by the user.

[0044] Next, in S702, if the image analysis unit 113 detects the tracking target (Yes in S702), the subject search process shown in Fig. 7 is terminated and the process proceeds to S607. On the other hand, if the image analysis unit 113 cannot detect the tracking target (No in S702), the process proceeds to S703.

[0045] In S703, the image analysis unit 113 reads the size recorded in memory in S605, i.e., the size of the tracking target on the image immediately before the tracking target was lost. Next, in S704, the image analysis unit 113 performs the following process. That is, assuming that a tracking target having the size of the tracking target read in S703 is captured in the image of the current frame, the image analysis unit 113 calculates how much zoom out can be performed within the zoom value range at which the tracking target can be detected from this state. For example, the image analysis unit 113 determines that detection is possible with high accuracy if the vertical size of the tracking target detected from the image is 30% or more of the vertical size of the image, and sets this as the minimum detection size. In this case, if the tracking target is zoomed out so that its height appears the same as the minimum detection size, if it moves away from the image capture device 100, the tracking target will become even smaller than the minimum detection size and will no longer be detectable. Therefore, the system control unit 114 calculates the amount of zoom out so that the tracking target is captured at a size that is 1.2 times the minimum detection size, that is, 36% of the minimum detection size. Furthermore, the system control unit 103 instructs the pan / tilt / zoom control unit 115 to zoom out by the calculated zoom-out amount, and causes the lens driving unit 102 to perform the zoom-out.

[0046] In S705, the system control unit 114 reads, from a memory or the like, the position information of the search range transmitted from the information processing device 200. It is assumed that the position information of the search range has been transmitted from the information processing device 200 to the imaging device 100 in advance and stored in the memory of the imaging device 100.

[0047] In S706, the system control unit 114 determines a search route based on the current imaging range and the set search range, for performing pan / tilt control so as to sequentially image the search range via the shortest route.

[0048] In S707, the system control unit 114 instructs the pan-tilt-zoom control unit 115 to control the imaging range along the search route determined in S706. As a result, the pan-tilt-zoom control unit 115 controls the pan driving unit 103 and / or the tilt driving unit 104 to control the imaging range along the search route. In the example shown in Fig. 5, the imaging range is controlled along the search route consisting of routes 406 to 410 starting from the initial range 405.

[0049] If the image analysis unit 113 detects the tracking target from the image captured in the process of controlling the imaging range along the search route in step S707 (Yes in S708), the process proceeds to S709. On the other hand, if the image analysis unit 113 cannot detect the tracking target from the image captured in the process of controlling the imaging range along the search route (No in S708), the process proceeds to S710.

[0050] In S709, the system control unit 114 instructs the pan / tilt / zoom control unit 115 to stop controlling the imaging range along the search route in S707. After the processing of S709, the process proceeds to S607 shown in FIG. 6. In S710, the system control unit 114 determines whether the number of times the processing of S707 (control processing of the imaging range along the search route) has been performed is equal to or greater than a first threshold (a threshold for giving up the search processing and ending tracking). If the number of times the processing of S707 has been performed is equal to or greater than the first threshold (Yes in S710), the search processing is terminated and the process proceeds to step S607. If the number of times the processing of S707 (control processing of the imaging range along the search route) has been performed is less than the first threshold, the process proceeds to S711.

[0051] In S711, the system control unit 114 determines whether the number of times the process of S707 (control process of the image capture range along the searched route) has been performed is equal to or greater than a second threshold (threshold for temporarily suspending the search and returning to the home position). If it is determined that the number of times is equal to or greater than the second threshold (Yes in S711), the process proceeds to S712 to temporarily return the image capture range to the home position. On the other hand, if it is determined that the number of times is less than the second threshold (No in S711), the process transitions to S707, and control process of the image capture range along the searched route is performed again. At this time, the count value of the number of times the process of S707 has been performed is incremented by +1.

[0052] In S712, the system control unit 114 instructs the pan / tilt / zoom control unit 115 to move to a home position (initial imaging range) determined by pan values, tilt values, and zoom values ​​previously set by the user. Here, the home position may be set in advance by the user, or a range where the tracking target is likely to exist may be calculated based on the movement history of the tracking target up to now, and the angle of view capturing that range may be set as the home position.

[0053] In S713, the image analysis unit 113 executes a detection process for detecting a tracking target from images captured sequentially over a predetermined time period. If the image analysis unit 113 detects a tracking target (Yes in S714), the subject search process ends and the process proceeds to S607 shown in Fig. 6. On the other hand, if the image analysis unit 113 cannot detect a tracking target (No in S714), the process proceeds to S707, where the control process for the imaging range along the search route is executed again in S707 to search for the tracking target again. At this time, the count value of the number of times the process of S707 has been executed is incremented by +1.

[0054] As described above, when the image capturing device 100 of this embodiment is unable to detect the tracking target, it executes a process of searching for the tracking target while controlling the imaging range over the entire search range set by the user as a range in which the tracking target may exist. By doing so, even if it becomes impossible to track the tracking target, it is possible to improve the possibility of finding the tracking target again.

[0055] (Embodiment 2) In this embodiment, if it becomes impossible to track the tracking target, similarly to the first embodiment, images are sequentially captured within the set search range using pan-tilt-zoom control to search for the tracking target. Subsequently, the searched tracking target is tracked and captured. Furthermore, in this embodiment, if the tracking target is tracked outside the set search range while being tracked, the search range is updated to include the position of the tracking target detected outside the search range. This allows the search range to include all locations where the tracking target has previously existed, making it easier to search for locations where the tracking target is likely to be present during subject search. Furthermore, even if the user is unable to accurately set the search range, the search range can be appropriately updated to reflect the range where the subject may be present. Note that differences from the first embodiment will be mainly described, and components and processes that are the same or equivalent to those in the first embodiment will be denoted by the same reference numerals, and redundant description will be omitted.

[0056] Here, the tracking and imaging operation of the tracking target by the imaging device 100 will be described with reference to the flowchart of Fig. 6. Note that the processing of the flowchart shown in Fig. 13 is executed by the functional blocks shown in Fig. 3 which are realized when the CPU 1300 of the imaging device 100 executes a computer program stored in the ROM 1320 of the imaging device 100. Furthermore, the processing of S601 to S609 in Fig. 8 is the same as the content described with reference to Fig. 6 in the first embodiment, and therefore description thereof will be omitted.

[0057] In S810, the system control unit 114 determines whether the position of the tracking target detected by the image analysis unit 113 is outside the currently set search range. If it is determined that the position of the detected tracking target is outside the range (Yes in S810), the system control unit 114 proceeds to step S611 to expand the search range. On the other hand, if the position of the detected tracking target is within the currently set search range (No in S810), the system control unit 114 transitions to S601.

[0058] In S811, the system control unit 114 updates the setting of the search range to include the current position of the tracking target. Here, the search range update process will be described with reference to FIG. 9. An imaging range 901 is the current imaging range of the imaging device 100, and a search range 902 is the currently set search range. A circumscribing rectangle 903 is a circumscribing rectangle of a person to be tracked detected by the image analysis unit 113 and indicates the position of the person. In the example shown in FIG. 9, the circumscribing rectangle 903 of the tracking target is located outside the search range 902. In this case, the system control unit 114 updates the search range 902 to include the circumscribing rectangle 903 located outside the search range 902. Specifically, the system control unit 114 determines that the search range 902 needs to be expanded rightward because side 905, which is the right side of the circumscribing rectangle 903, is located to the right of side 904, which is the right side of the search range 902. At this time, the system control unit 114 expands the position of the right side 904 of the search range 902 to the position of the right side 905 of the circumscribing rectangle 903 so that the search range 902 encompasses the circumscribing rectangle 903. At this time, the expanded range has a direction and distance indicated by arrow 906. Note that while the case of expanding the range to the right has been described as an example, the system control unit 114 expands the left side of the search range 902 to the left if the left side of the circumscribing rectangle 903 is located to the left of the left side of the search range 902. Similarly, the system control unit 114 expands the search range upward by comparing the top side and downward by comparing the bottom side. Through the above operations, the search range can be expanded to encompass the current position of the tracking target and the setting can be updated. Note that the system control unit 114 transmits information about the updated search range to the information processing device 200, and the display control unit 201 of the information processing device 200 may superimpose the updated search range on the wide-angle image 401 and display it on the display 210.

[0059] In this way, according to this embodiment, when the tracking target can no longer be detected and the entire range in which the target may exist is searched using pan-tilt-zoom drive, it is possible to include in the search range all areas in which the tracking target has previously existed, thereby increasing the possibility of successfully searching for the area in which the tracking target exists.

[0060] (Embodiment 3) In this embodiment, the setting of the search range is automatically determined from the trajectory of pan-tilt-zoom control performed manually by the user. This simplifies the setting of the search range. Differences from the first embodiment will be mainly described, and the same or equivalent components and processes as those in the first embodiment will be denoted by the same reference numerals, and redundant description will be omitted. In particular, the process of controlling the imaging range while tracking the tracking target and the search process when tracking of the tracking target becomes impossible will be considered the same as those in the first embodiment, and description thereof will be omitted.

[0061] In this embodiment, the search range setting process is executed in advance before the process of the flow shown in Fig. 6 of the first embodiment is started. The system control unit 114 starts the search range setting process when it receives a request command for setting a search range from the information processing device 200 based on a user operation. The search range setting process will now be described with reference to the flow shown in Fig. 10. Note that the process of the flowchart shown in Fig. 10 is executed by the functional blocks shown in Fig. 3 that are realized when the CPU 1300 of the imaging device 100 executes a computer program stored in the ROM 1320 of the imaging device 100.

[0062] First, in S1001, the system control unit 114 acquires a control command for controlling at least one of the pan value, tilt value, and zoom value (in other words, a control command for controlling the imaging range) transmitted from the information processing device 200. When the operation receiving unit 202 receives a user operation for changing the pan value, tilt value, and zoom value, the system control unit 203 of the information processing device 200 executes the following process. That is, the system control unit 114 generates a control command for setting the pan value, tilt value, and / or zoom value to the value specified by the user, and transmits the control command to the imaging device 100. In S1001, the system control unit 114 acquires the control command transmitted in this manner. Based on the control command, the system control unit 114 instructs the pan / tilt / zoom control unit 115 to set the pan value, tilt value, and / or zoom value to the value specified by the user. The pan / tilt / zoom control unit 115 controls the pan driving unit 103, the tilt driving unit 104, and / or the lens driving unit 102 in accordance with the instruction.

[0063] In S1002, the system control unit 114 generates history information indicating the time series changes in the pan value, tilt value, and zoom value of the image capturing apparatus 100 due to user operations, and stores the information in memory.

[0064] In S1003, the system control unit 114 determines whether or not a command to end the search range setting process has been received from the information processing device 200. If the command has been received (Yes in S1003), the process proceeds to S1004, and if the command has not been received (No in S1003), the process proceeds to S1001.

[0065] In S1004, the system control unit 114 sets, as a search range, a range that has been included at least once in the imaging range of the imaging device 100 controlled by user operation, based on the history information of the changes in the pan, tilt, and zoom values ​​stored in S1002. The process of setting the search range will now be described with reference to FIG. 11 . An area 1101 indicates the shooting location, and a range 1102 indicates the range that has been included at least once in the imaging range due to the user's pan, tilt, and zoom values. A trajectory 1103 is the trajectory of the imaging range traversed by the user's operation. If the range 1102 were set as the search range, it would have a complex shape, which would complicate the search route when searching for the tracking target. Therefore, a range 1104 (hereinafter referred to as the search range 1104), which is a circumscribing rectangle of the range 1102, is set as the search range. The left side of the search range 1104 is determined to pass through point 1105, which is the left end of the range 1102. Similarly, the right side of search range 1104 is determined to pass through point 1106, the right end of range 1102. The top side of search range 1104 is determined to pass through point 1107, the top end of range 1102, and the bottom side of search range 1104 is determined to pass through point 1108, the bottom end of range 1102. Note that, although the above example describes a rectangular search range, a search range of a complex shape may be set and a search performed. Furthermore, the search route may be determined based on the history information of changes in pan, tilt, and zoom values ​​stored in S1002, rather than the method shown in FIG. 5 of the first embodiment. In this case, the search process is performed using a search route that follows the trajectory of trajectory 1003.

[0066] In the above example, the user performs pan, tilt, and zoom operations for setting the search range after the imaging device 100 receives a search range setting request from the information processing device 200. However, history information on changes in pan, tilt, and zoom values ​​made by the user during normal imaging may be generated, and the search range may be determined based on the history information.

[0067] As described above, in the imaging device 100 of this embodiment, the search range in which the search process is executed is set by the user performing pan-tilt-zoom operations, thereby simplifying the setting of the search range.

[0068] (Embodiment 4) In this embodiment, when it becomes impossible to track the tracking target, similarly to the first embodiment, images of the set search range are sequentially captured by pan-tilt-zoom control to search for the tracking target. Thereafter, the tracked target is tracked and captured. In addition to the functions of the first embodiment, this embodiment also stores information on the history (movement history) of past locations where the tracking target was located. When searching for the tracking target after it becomes impossible to detect it, the possibility of finding the tracking target earlier can be increased by first searching positions where many tracking targets were located in the past. Note that differences from the first embodiment will be mainly described, and components and processes that are the same or equivalent to those of the first embodiment will be denoted by the same reference numerals, and redundant description will be omitted.

[0069] The system control unit 114 divides the set search range into a plurality of blocks at equal intervals. Based on the result of the tracking target detection process performed on the image by the image analysis unit 113, the system control unit 114 counts up the presence time of the tracking target for the block including the center position of the detected tracking target. In this way, the system control unit 114 can hold the cumulative presence time of the tracking target for each block as the movement history of the tracking target.

[0070] When determining a search route for a search process to be executed when it becomes impossible to track the tracking target, the system control unit 114 determines a search route that first searches blocks in which the tracking target has existed for a long time in the past, based on the time the tracking target has existed for each block in the search range. Here, the operation of determining the search route will be described with reference to FIG. 112. An area 1201 indicates the environment of the subject to be photographed, and a search range 1202 is the currently set search range. An imaging range 1203 is the imaging range at the start of the search process when it becomes impossible to track the tracking target. As shown in FIG. 12, the search range 1202 includes blocks 1204a to 1204e obtained by dividing it horizontally into five. The system control unit 114 keeps a cumulative total of the existence time, which is the time the tracking target has existed, for each of the blocks 1204a to 1204e. For example, if the tracking target has been present in the leftmost block 1204a for the longest time among blocks 1204a to 1204e, the system control unit 114 determines a search route so as to image block 1204a with priority. The movement route when controlling the imaging range at this time is route 1205. Thereafter, if the tracking target cannot be detected in the leftmost block, blocks on the right are also searched, and a search is performed while controlling the imaging range using the movement route of route 1206. Note that in the above example, a case has been described in which block division is performed horizontally, but this is not limiting and the block unit can be changed, such as division in the vertical direction or division in finer units.

[0071] In the above example, the search order is determined based on the time the tracking target existed in the past, but there is another method of determining the order based on the position where the tracking target was last lost. In that case, if the position where the tracking target was last detected was on the left side of the angle of view of the imaging device, the left side can be searched first. Furthermore, the search route may be determined by prioritizing the search in other ways using the history of the tracking target's past positions.

[0072] In this way, according to this embodiment, by estimating the area where the tracking target is likely to currently be located from the movement history of the tracking target and searching from the area with the highest probability, it is possible to increase the possibility of finding the tracking target in a short period of time.

[0073] (Other embodiments) Next, the hardware configuration of the imaging device 100 for realizing each function of each embodiment will be described with reference to Fig. 13. Note that, although the hardware configuration of the imaging device 100 will be described in the following explanation, it is assumed that the imaging device 100 is also realized by a similar hardware configuration.

[0074] The imaging device 100 in this embodiment includes a CPU 1300 , a RAM 1310 , a ROM 1320 , an HDD 1330 , and an I / F 1340 .

[0075] The CPU 1300 is a central processing unit that controls the imaging device 100. The RAM 1310 temporarily stores computer programs executed by the CPU 1300. The RAM 1310 also provides a work area used when the CPU 1300 executes processing. The RAM 1310 also functions as, for example, a frame memory or a buffer memory.

[0076] The ROM 1320 stores programs and the like used by the CPU 1300 to control the imaging device 100. The HDD 1330 is a storage device that records image data and the like. The I / F 1340 communicates with external devices via the network 300 in accordance with TCP / IP, HTTP, or the like.

[0077] Although the above-described embodiments have been described with reference to examples in which the CPU 1300 executes the processing, at least a portion of the processing by the CPU 1300 may be performed by dedicated hardware. For example, the processing of reading program code from the ROM 1320 and loading it into the RAM 1310 may be performed by a DMA (Direct Memory Access) that functions as a transfer device.

[0078] The present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-described embodiments. The program may be supplied to a system or device having a processor via a network or a storage medium. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions of the above-described embodiments. Each unit of the imaging device 100 may be realized by hardware shown in FIG. 13, or by software.

[0079] Note that one or more functions of the imaging device 100 according to each of the above-described embodiments may be provided in another device. For example, one or more functions of the imaging device 100 according to each of the above-described embodiments may be provided in the imaging device 100 itself. Note that the above-described embodiments may be combined, for example, any combination of the above-described embodiments may be implemented.

[0080] Although the present invention has been described above with reference to the embodiments, the above embodiments merely illustrate specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from the technical concept or main features of the present invention. For example, combinations of the embodiments are also included in the disclosure of this specification. [Explanation of symbols]

[0081] 100 Imaging device 200 Information processing device 113 Image Analysis Unit 114 System control section

Claims

1. a control means for controlling an imaging range of the imaging means so as to track the tracking target; a setting means for setting a search range based on a range set by a user, a user operation for controlling the imaging range, or a past position of a tracking target; When the tracking target cannot be tracked, the control means controls the imaging range so as to search for the tracking target in the search range; The search range is divided into a plurality of blocks, The control device is characterized in that the control means determines a route for controlling the imaging range in the search range based on the existence time of each block of the tracking target, and controls the imaging range according to the determined route.

2. 2. The control device according to claim 1, wherein the setting means acquires information about a range set by a user from an information processing device, and sets the range based on the acquired information as the search range.

3. The control device according to claim 1 , wherein the setting unit sets the search range based on a range included in the imaging range during the process of controlling the imaging range in accordance with the user operation.

4. 2. The control device according to claim 1, wherein, when the tracking target is detected outside the search range, the setting means updates the search range so as to include the position of the tracking target detected outside the search range.

5. a control step of controlling an imaging range of the imaging means so as to track the tracking target; a setting step of setting a search range based on a range set by a user, a user operation for controlling the imaging range, or a past position of a tracking target; When the tracking target can no longer be tracked, the control step controls the imaging range so as to search for the tracking target in the search range; The search range is divided into a plurality of blocks, A control method characterized in that, in the control step, a route for controlling the imaging range within the search range is determined based on the existence time of each block of the tracking target, and the imaging range is controlled according to the determined route.

6. A computer program for causing a computer to function as each of the means included in the control device according to any one of claims 1 to 4.

Images