Control device, control method, and program

Abstract

While continuing to track and shoot the subject being tracked, the display position of the subject being tracked can be changed to the display position (composition) intended by the operator. [Solution] The control device displays a GUI on the screen, which shows the speed and position of the movement of the display position of the target subject on the screen in response to the operator's operation to move the display position of the target subject on the screen during tracking shooting of the target subject, by compositing it onto the image captured by the camera with a pan / tilt head. The control device also has a tracking control means that controls tracking shooting of the target subject by the camera with a pan / tilt head based on the display position of the target subject on the screen set according to the operator's operation using the GUI and the detection result of the target subject appearing in the image captured by the camera with a pan / tilt head.

Description

Technical Field

[0005] , , ,

[0001] The present invention relates to a control technique during subject tracking photography.

Background Art

[0002] With the emergence of an AI (Artificial Intelligence) model that enables real-time object detection, object detection and tracking using AI technology have become possible. Utilizing this technology, an automatic tracking system has been developed that automatically pans a remote camera left and right to automatically track and photograph a moving subject. Also, there is a need to perform automatic tracking photography of a subject to be tracked while maintaining the composition intended by the operator by selecting the subject to be tracked and setting the display position of the subject to be tracked on the screen. In this case, the operator pre-sets the display position (hereafter referred to as the subject display position) for the subject to be tracked with respect to the automatic tracking system. Then, the automatic tracking system performs automatic tracking photography in a composition such that the subject to be tracked fits within the subject display position in accordance with the movement of the subject to be tracked during automatic tracking.

[0003] Patent Document 1 discloses a method of setting (i.e., setting the composition) the selection of the subject to be tracked and the output position (subject display position) of the subject to be tracked before the start of automatic tracking photography. According to the technique described in Patent Document 1, it is possible to track the subject to be tracked at the output position (subject display position) set before the start of automatic tracking photography (i.e., track while maintaining the composition).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Furthermore, during automatic tracking shooting, there is a need to change the position of the subject displayed on the screen to the position (i.e., composition) intended by the operator, in accordance with changes in the shooting scene or other situations. Changing the position of the subject displayed (changing the composition) can be achieved, for example, by selecting a preset position or tracing movement, but these are limited to changes to pre-registered compositions and often do not result in the smooth composition changes intended by the operator according to the shooting scene. There is also a method of changing the composition using touch panel operation, but in this case the composition change is linear or mechanical composition movement and cannot achieve the smooth composition changes or changes in movement speed that can be achieved with manual shooting. In addition, in the case of the technology disclosed in Patent Document 1, the position of the subject displayed (composition) is set before shooting starts, so if the composition is to be readjusted, automatic tracking shooting must be stopped and the setting must be readjusted again. In other words, even with the technology disclosed in Patent Document 1, it is not possible to change the position of the subject displayed to the position on the composition intended by the operator while continuing subject tracking shooting.

[0006] Therefore, the present invention aims to enable the operator to change the display position (composition) of a subject being tracked to a display position (composition) intended by the operator while continuing to track and photograph the subject being tracked. [Means for solving the problem]

[0007] The present invention relates to a control device for controlling a camera with a pan / tilt head to track and photograph a subject to be tracked, comprising: a GUI display means for displaying a GUI that shows the speed and position of movement of the display position of the subject to be tracked in response to an operator's operation to move the display position of the subject to be tracked on the screen during tracking and photography of the subject to be tracked, by compositing the GUI onto the image captured by the camera with the pan / tilt head; and a tracking control means for controlling tracking and photography of the subject to be tracked by the camera with the pan / tilt head based on the display position of the subject to be tracked on the screen set in response to the operator's operation using the GUI and the detection result of the subject to be tracked appearing in the image captured by the camera with the pan / tilt head. [Effects of the Invention]

[0008] According to the present invention, while continuing to track and photograph the subject being tracked, the operator can change the display position (composition) of the subject being tracked to a display position (composition) intended by the operator. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an example configuration of an automatic tracking camera system and control device. [Figure 2] This is a flowchart of the control process according to the first embodiment. [Figure 3] This is an explanatory diagram of the GUI for manipulating the display position. [Figure 4] This is an explanatory diagram of other GUI examples for manipulating display positions. [Figure 5] This is a flowchart of the control process according to the second embodiment. [Figure 6] This is a conceptual diagram illustrating the calculation of correction values ​​for PTZ control according to the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not limiting to the present invention, and not all of the features described in each embodiment are essential to the solution of the present invention; these features may be combined arbitrarily. The configuration of each embodiment may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (operating conditions, operating environment, etc.). In addition, redundant explanations of the same or similar configurations or processing steps will be omitted in the following embodiments.

[0011] <First Embodiment> Figure 1 shows an example configuration of an automatic tracking camera system according to the first embodiment. Figure 1(a) shows the main components of the camera with a pan / tilt head 101 and the control device 104, which are included in the automatic tracking camera system, and the monitor 113.

[0012] In the automatic tracking camera system shown in Figure 1(a), the camera with a tripod head 101 (hereinafter referred to as the tripod head camera 101) is configured to have a camera unit 102 mounted on a tripod head capable of panning (horizontal drive) and tilting (vertical drive) rotational movements. The camera unit 102 is a camera with a lens that has zooming and focusing functions. Hereinafter, panning will be referred to as "pan," tilting as "tilt," and zooming as "zoom," and as appropriate, pan-tilt will be referred to as "PT," zoom as "Z," and pan-tilt-zoom as "PTZ." PTZ control in the tripod head camera 101 is handled by the tripod head camera control unit 103, and the control signal when the tripod head camera control unit 103 performs PTZ control of the tripod head camera 101 is generated by the control device 104.

[0013] The control device 104 includes a signal processing unit 105, a control interface 106, a subject detection unit 107, an image synthesis unit 108, a control calculation unit 109, an operation GUI (graphical user interface) unit 110, an output unit 111, and an operation unit 112.

[0014] The video image signal captured by the camera unit 102 is sent to the signal processing unit 105 of the control device 104. Hereinafter, the image captured by the camera unit 102 will be referred to as the captured image. The signal processing unit 105 converts the image signal sent from the camera unit 102 to a digital signal if it is an analog signal. The signal processing unit 105 also performs image inversion processing to flip the image vertically if, for example, the pan-tilt camera 101 is installed on the ceiling, the camera unit 102 is suspended upside down relative to the ground or floor, and the image captured by the camera unit 102 is upside down. Transmission between the camera unit 102 and the control device 104 may be, for example, transmission using Media Over IP technology, which transmits image signals and audio signals over an IP network. In this case, the signal processing unit 105 also performs data conversion for the data transmitted using Media Over IP technology. If the control signals are implemented using IP communication, the connection between the pan-tilt camera 101 and the control device 104 can be configured with a single Ethernet® connection. The captured image after signal processing by the signal processing unit 105 is sent to the monitor 113 via the subject detection unit 107, image synthesis unit 108, and output unit 111, which will be described later. As a result, the captured image is displayed on the screen of the monitor 113, and the operator can view the image captured by the camera unit 102 by looking at the displayed image.

[0015] The control unit 112 includes a joystick for the operator to indicate directions such as pan and tilt, a seesaw key for the operator to indicate zoom in and out, a mode operation button for switching the control mode of the control device 104, and a select button. The control modes of the control device 104 include at least a PTZ operation mode, a tracking target subject setting mode for setting the subject to be tracked, and, in this embodiment, a display position operation mode that allows the display position of the subject to be tracked to be changed during automatic tracking shooting. For example, the operator can switch the control mode of the control device 104 to any of these modes by operating the mode operation button. Operator operation information for the joystick, seesaw key, mode selection button, etc. of the control unit 112 is sent to the control calculation unit 109.

[0016] For example, before the start of automatic tracking shooting, if the control device 104 is in PTZ operation mode, the control calculation unit 109 calculates a pan / tilt control value (PT control value) corresponding to the joystick operation information when the operator operates the joystick. Also, when the operator operates the seesaw key, the control calculation unit 109 calculates a zoom control value (Z control value) corresponding to the seesaw key operation information. These PT control values ​​and Z control values ​​are then sent as control signals to the pan / tilt camera control unit 103 of the pan / tilt camera 101 via the control I / F 106. The pan / tilt camera control unit 103 performs PT control of the pan / tilt camera 101 based on the PT control value and Z control of the camera unit 102 based on the Z control value. In this way, in PTZ operation mode before the start of automatic tracking shooting, the operator can perform PTZ operation of the pan / tilt camera 101 through operation of the operation unit 112.

[0017] Furthermore, before the start of automatic tracking shooting, if the control device 104 is in tracking target subject setting mode, the control calculation unit 109 generates a tracking target frame for the operator to select the tracking target subject in the captured image and further specify the display position of that tracking target subject on the screen. The image of the tracking target frame is combined with the captured image by the image synthesis unit 108 and sent from the output unit 111 to the monitor 113. This allows the operator to visually confirm the tracking target frame on the captured image. In addition, in tracking target subject setting mode, when the operator operates the joystick, the control calculation unit 109 moves the position of the tracking target frame in the captured image according to the joystick operation information. Also, when the operator operates the seesaw key, the control calculation unit 109 enlarges or reduces the tracking target frame in the captured image according to the seesaw key operation information. Thus, in the tracking target subject setting mode, the operator can specify the display position of the tracking target subject on the screen and set the desired composition by manipulating the position and size of the tracking target frame in the captured image through operations on the control unit 112.

[0018] After that, when the automatic tracking shooting is started in the control device 104, the subject detection unit 107 performs person detection using AI on the captured image received from the signal processing unit 105. In the subject detection unit 107 of the present embodiment, person detection using AI is realized by using a neural network model designed to specialize in face recognition and skeleton estimation. The neural network model used in the present embodiment is a model that has been pre-trained to output person detection results by face recognition and skeleton estimation for an input image. When the subject detection unit 107 detects a person of the tracking target subject from the captured image, it sends, for example, a rectangular subject detection frame surrounding the detected person, together with the captured image, to the image composition unit 108. In this case, the image composition unit 108 composes the subject detection frame on the person of the tracking target subject reflected in the captured image.

[0019] Also, during the automatic tracking shooting, the subject detection unit 107 outputs the result of the person detection of the tracking target subject to the control arithmetic unit 109 as person tags and coordinate axis information. The control arithmetic unit 109 refers to the person tags and coordinate axis information input from the subject detection unit 107 to identify the tracking target subject that is the target of the automatic tracking shooting. Then, while maintaining the display position of the tracking target subject on the screen, the control arithmetic unit 109 sends the PTZ control value for performing the automatic tracking shooting of the tracking target subject to the pan-tilt camera control unit 103 of the pan-tilt camera 101 via the control I / F 106. That is, the control arithmetic unit 109 performs tracking control by generating a PTZ control value for performing automatic tracking shooting while maintaining the display position (composition) of the tracking target subject based on the display position of the tracking target subject on the screen and the detection result of the tracking target subject by the subject detection unit 107.

[0020] The pan-tilt camera control unit 103 performs PTZ control of the pan-tilt camera 101 based on the PTZ control value sent from the control device 104. As a result, the pan-tilt camera 101 can perform automatic tracking shooting of the tracking target subject in a composition in which the tracking target subject is displayed at the display position on the screen set by the operator in the tracking target subject setting mode before the start of the automatic tracking shooting.

[0021] When automatic tracking shooting is being performed by the control device 104, the image synthesis unit 108 synthesizes a subject detection frame onto the captured image and sends it to the output unit 111. The output unit 111 converts the image input from the image synthesis unit 108 into an image signal that matches the connection interface with the monitor 113 and outputs it. The monitor 113 displays the image sent from the output unit 111. This allows the operator to visually confirm the status of automatic tracking of the target subject by viewing an image in which the subject detection frame has been superimposed on the image captured by the camera unit 102.

[0022] For example, if the control device 104 is set to PTZ operation mode during automatic tracking shooting, the control calculation unit 109 calculates a PT control value according to the operation information when the joystick is operated by the operator. Also, the control calculation unit 109 calculates a Z control value according to the operation information when the seesaw key is operated by the operator. The PT control value and Z control value calculated by the control calculation unit 109 are then sent to the pan / tilt camera control unit 103 of the pan / tilt camera 101 via the control I / F 106. In this way, the operator can perform PTZ operations on the pan / tilt camera 101 even during automatic tracking shooting. However, in this case, depending on the direction and amount of joystick operation by the operator, the display position of the subject being tracked may shift, making it impossible to maintain the composition.

[0023] Furthermore, the operator may want to change the display position of the subject being tracked on the screen (i.e., change the composition) in accordance with changes in the shooting scene, for example, during automatic tracking shooting. For this reason, the control device 104 according to this embodiment is provided with a display position operation mode that allows the display position of the subject being tracked to be changed during automatic tracking shooting. Even if the control device 104 is switched to display position operation mode during automatic tracking shooting, the subject detection unit 107 outputs the results of person detection to the control calculation unit 109 as a person tag and coordinate axis information, as described above. The control calculation unit 109 then refers to the person tag and coordinate axis information input from the subject detection unit 107 to identify the subject being tracked for automatic tracking shooting.

[0024] Here, if the control device 104 is in display position operation mode during automatic tracking shooting, the operation GUI unit 110 of the control calculation unit 109 generates a display position operation GUI to assist in specifying the display position and size of the subject to be tracked. The display position operation GUI moves, enlarges, shrinks, etc., the display position operation frame of the subject to be tracked according to the operation information obtained from the operation unit 112, and is a GUI that allows the operator to visually confirm the movement speed, movement position, and size of the display position operation frame. This display position operation GUI is sent to the image synthesis unit 108.

[0025] The image synthesis unit 108 synthesizes the display position operation GUI image with the captured image. The synthesized image is then sent from the output unit 111 to the monitor 113. In other words, when the display position operation mode is active during automatic tracking shooting, the image with the display position operation GUI synthesized with the captured image is displayed on the screen of the monitor 113.

[0026] Furthermore, in the display position operation mode during automatic tracking shooting, when the operator operates the operation unit 112, the operation GUI unit 110 of the control calculation unit 109 moves, enlarges, shrinks, etc., the display position operation frame on the screen according to the operation information from the operation unit 112. In other words, in the display position operation mode during automatic tracking shooting, the operator can specify the desired composition by moving, enlarging, shrinking, etc., the display position operation frame. In addition, the operator can visually confirm the status of the camera's captured image and the display position operation of the subject being tracked by viewing an image in which the display position operation GUI is superimposed on the captured image.

[0027] In this way, the control device 104 in display position operation mode implements a GUI display function that allows the user to move the display position of the subject being tracked on the screen during automatic tracking shooting. The control calculation unit 109 then generates PT control values ​​and Z control values ​​for displaying the subject to be tracked at the position of the display position control frame on the screen specified by the operator during automatic tracking shooting, that is, PTZ control values ​​for displaying the subject to be tracked at the target display position of the composition specified by the operator. These PT control values ​​and Z control values ​​are sent to the pan / tilt camera control unit 103 of the pan / tilt camera 101 via the control I / F 106.

[0028] Figure 1(b) shows an example of the hardware configuration of the control device 104 shown in Figure 1(a). The CPU 121 controls various devices connected to the bus 128 and performs control processing related to the control device 104 of this embodiment. In the case of the control device 104, the CPU 121 performs control processing related to each functional unit of the control device 104 shown in Figure 1(a) above. ROM122 stores the BIOS program and boot program. RAM123 is used as auxiliary storage for CPU121. The large-capacity memory 124 stores images sent from the camera unit 102, images synthesized by the image synthesis unit 108, and the control program according to this embodiment. The control program according to this embodiment stored in the large-capacity memory 124 is loaded into the RAM 123 and executed by the CPU 121. This enables the control processing by the control device 104 of this embodiment.

[0029] The control unit 125 corresponds to the control unit 112 in Figure 1(a) and is equipped with a joystick, seesaw key, mode operation button, select button, etc., which will be described later. The control unit 125 may also be equipped with a keyboard, mouse, or touch panel, and processes input from the operator. The output unit 126 corresponds to the output unit 111 in Figure 1(a) and outputs images and various processing results. I / O (Input / Output) 127 is connected to external devices such as the tripod camera 101 and monitor 113 via a network and communicates with them. Bus 128 connects the CPU 121, ROM 122, RAM 123, large-capacity memory 124, input unit 125, display unit 126, and I / O 127 in a manner that enables them to communicate with each other.

[0030] Figure 2 is a flowchart showing the control flow by the control device 104 when the control device 104 switches to display position operation mode during automatic tracking shooting, and a display position operation is performed to change the display position of the subject being tracked on the screen to the composition intended by the operator. Figure 3 is a diagram showing an example of the display image of the display position operation GUI. The operation and control using the display position operation GUI will be explained below with reference to Figures 2 and 3.

[0031] First, in step S201, when the control calculation unit 109 receives an instruction to switch to the display position operation mode via the operation unit 112 while the tracking target subject is being automatically tracked and shot, it switches the control mode of the control device 104 to the display position operation mode. In other words, the control device 104 switches to the display position operation mode while continuing to automatically track and shoot the tracking target subject.

[0032] Next, in step S202, the operation GUI unit 110 generates the aforementioned display position operation GUI and sends it to the image synthesis unit 108. The display position operation GUI includes the display position operation frame as described above. Also, since the control device 104 is performing automatic tracking and shooting of the subject to be tracked at this time, the image synthesis unit 108 also receives the subject detection frame, which is the subject detected by the subject detection unit 107. Therefore, the image synthesis unit 108 synthesizes the subject detection frame, the display position operation GUI, and the display position operation frame with the captured image. Here, in the display position operation mode during automatic tracking and shooting, the image synthesis unit 108 synthesizes the display position operation frame with the captured image as a display frame that is more emphasized than the subject detection frame. The image after synthesis by the image synthesis unit 108 is then sent from the output unit 111 to the monitor 113. As a result, the monitor 113 displays an image in which the display position operation GUI, the subject detection frame, and the display position operation frame that is more emphasized than the subject detection frame are synthesized with the captured image.

[0033] Figures 3(3a) to (3f) show examples of how the display position operation GUI displays an image where the subject detection frame 321 and the display position operation frame 320 are combined for the tracked subject 300. In the examples of Figures 3(3a) to (3f), the display position operation frame 320, which is more emphasized than the subject detection frame 321, is shown as a solid line frame, while the subject detection frame 321 is shown as a dashed line frame. In other words, when switched to display position operation mode, the display position operation frame 320 becomes the main frame display, and the subject detection frame 321 becomes a display frame for shooting assistance. Figure 3 also shows an example of the control device 104 and the operation unit 112, as well as the joystick 302 and seesaw key 301 of the operation unit 112. Furthermore, as shown in Figures 3(3a) to (3f), the display position operation GUI also includes an icon 312 representing the joystick 302 and an icon 311 representing the seesaw key 301.

[0034] Next, in step S203, the operation GUI unit 110 detects the operator's operation of the joystick 302, seesaw key 301, etc., based on the operation information from the operation unit 112. Figures 3(3a) to (3c) show examples of captured images and the display position operation GUI when operation of the joystick 302 is detected, and Figures 3(3d) to (3f) show examples of captured images and the display position operation GUI when operation of the seesaw key 301 is detected. As shown in Figures 3(3a) to (3f), regardless of whether the joystick 302 or the seesaw key 301 is operated, the display position operation frame 320 and the subject detection frame 321 are displayed in the display position operation GUI.

[0035] Next, in step S204, the operation GUI unit 110 is provided with operation display icons representing the direction and amount of operation of the operation unit 112, and movement display icons representing the movement speed and position of the display position of the subject being tracked, as components of the display position operation GUI. In this embodiment, the operation display icons are arrow-shaped icons that represent the direction of operation by the joystick 302 or seesaw key 301 with the direction of the arrow, and the magnitude of the amount of operation (movement speed) with the length of the arrow. The movement display icons are arrow-shaped icons that are different from the operation display icons, and represent the direction of movement of the display position operation frame with the direction of the arrow, and the movement speed (amount of movement) with the length of the arrow.

[0036] For example, when the operator tilts the joystick 302, the operation GUI unit 110 determines the direction of the arrows of the operation display icon and the movement display icon according to the tilt direction of the joystick 302. The operation GUI unit 110 also determines the length of the arrows of the operation display icon and the movement display icon according to the amount of tilt of the joystick 302. Here, when the joystick 302 is tilted, the movement speed (amount of movement) of the display position operation frame increases in proportion to the amount of tilt. Therefore, the operation GUI unit 110 makes the operation display icon longer as the amount of tilt of the joystick 302 increases, and shorter as the amount of tilt decreases. Also, the operation GUI unit 110 makes the movement display icon longer as the movement speed of the display position operation frame increases, and shorter as the movement speed decreases. When the amount of tilt of the joystick 302 is zero, that is, when the joystick 302 is not tilted, the operation GUI unit 110 removes the operation display icon and the movement display icon from the display position operation GUI. By including the aforementioned operation display icons and movement display icons in the display position operation GUI, the operator can visually recognize the direction and amount of operation of the joystick 302, as well as the direction and speed (amount of movement) of movement of the display position operation frame.

[0037] Figure 3(3b) shows an example of the display of the operation indicator icon 330 and the movement indicator icon 331 when the joystick 302 of the operation unit 112 is operated, for example, in the right pan direction (horizontal direction). The operation indicator icon 330 is an arrow-shaped icon that represents the tilt direction (operation direction) of the joystick 302 with the direction of the arrow, and the magnitude of the tilt amount (operation amount) with the length of the arrow. The movement indicator icon 331 is an arrow-shaped icon that represents the direction of movement of the display position operation frame 320 according to the tilt direction of the joystick 302 with the direction of the arrow, and the movement speed (amount of movement) of the display position operation frame 320 according to the amount of tilt with the length of the arrow. In Figure 3(3b), the joystick 302 is panned to the right, so the operation indicator icon 330 is placed on the right edge of the display position operation frame 320 and the movement indicator icon 331 is placed on the left edge of the frame, but this example is not the only one. Although not shown in the diagram, for example, when the joystick 302 is tilted upward, the operation indicator icon 330 and the movement indicator icon 331 are positioned, for example, on the upper and lower edges of the display position control frame 320. Similarly, although not shown in the diagram, for example, when the joystick 302 is panned and tilted diagonally, the operation indicator icon 330 and the movement indicator icon 331 are positioned on the display position control frame 320 according to the diagonal direction caused by the pan and tilt. The operation indicator icon 330 is also positioned near the icon 312 representing the joystick 302.

[0038] For example, when the operator operates the seesaw key 301, the operation GUI unit 110 places operation display icons and movement display icons, for example, at the four corners of the display position operation frame. In this embodiment, the seesaw key 301 corresponds to zoom operation, with one of the two keys of the seesaw key 301 corresponding to telephoto (i.e., magnification) zoom operation and the other corresponding to wide-angle (i.e., reduction) zoom operation. Therefore, when the telephoto key of the seesaw key 301 is operated, the operation GUI unit 110 enlarges the size of the display position operation frame (i.e., magnifies it), while when the wide-angle key is operated, the size of the display position operation frame is reduced (i.e., reduced). Furthermore, when the telephoto key of the seesaw key 301 is operated, the operation GUI unit 110 directs the arrows of the operation display icons and movement display icons outward (outside the display position operation frame), while when the wide-angle key is operated, the arrows direct inward. Furthermore, the operation GUI unit 110 lengthens the operation display icon and movement display icon as the pressing time of the telescopic key of the seesaw key 301 increases, or as the number of presses increases. Conversely, the operation GUI unit 110 shortens the operation display icon and movement display icon as the pressing time of the wide-side key decreases, or as the number of presses decreases. In this way, by including the operation display icon and movement display icon in the display position operation GUI even when the seesaw key 301 is operated, the operator can visually recognize the direction and amount of operation of the seesaw key 301.

[0039] Figure 3(3e) shows an example of how the operation GUI unit 110 displays the operation indicator icon 330 and the movement indicator icon 331 when the telescopic key of the seesaw key 301 is operated. As shown in Figure 3(3e), when the seesaw key 301 is operated, the operation indicator icon 330 and the movement indicator icon 331 are placed at the four corners of the display position operation frame 320. When the telescopic key of the seesaw key 301 is operated, the operation GUI unit 110 directs the arrows of the operation indicator icon 330 and the movement indicator icon 331 outwards (outside the display position operation frame 320). Although not shown in the illustration, when the wide-side key of the seesaw key 301 is operated, the operation GUI unit 110 directs the arrows of the operation indicator icon 330 and the movement indicator icon 331 inwards. Furthermore, the operation GUI section 110 sets the length of the arrows of the operation display icon 330 and the movement display icon 331 to a length corresponding to the press time or number of presses of the telescopic or wide-angle key of the seesaw key 301. The operation GUI section 110 also places a circle or the like to indicate the pressed key next to the icon 311 representing the seesaw key 301.

[0040] Let's return to the explanation of the flowchart in Figure 2. When the program proceeds to the next step S205 from step S204, the operation GUI unit 110 changes the position and size of the display position operation frame according to the direction and amount of operation on the joystick 302 and seesaw key 301. Figure 3(3c) shows an example of the display position control frame 320 after it has moved to the right in response to the operation of the joystick 302. Since the control device 104 continues to automatically track and photograph the subject being tracked, the subject detection frame 321 also moves in accordance with the subject being tracked 300. As in the example in Figure 3(3c), when a pan operation to the right is performed on the joystick 302, the displayed image transitions in the order of Figure 3(3a),(3b), and(3c), directly responding to the amount of tilt of the joystick 302, and the position of the display position control frame also moves. Figure 3(3f) shows an example of the display when the size of the display position control frame is enlarged in response to the operation of the telephoto key on the seesaw key 301. In this case as well, the control device 104 continues to automatically track and photograph the subject being tracked, so the subject detection frame 321 also enlarges in accordance with the subject being tracked 300. Thus, when the seesaw key 301 is operated to the telephoto side, the displayed image transitions in the order of Figure 3(3d), (3e), and (3f), directly responding to the operation of the seesaw key 301 to the telephoto side, and the size of the display position control frame changes. Thus, in the control device 104 according to this embodiment, while continuing automatic tracking and shooting of the subject being tracked, the display position of the subject being tracked can be operated using the display position operation mode, allowing the operator to change the composition to their desired one.

[0041] Next, in step S206, the control calculation unit 109 adjusts the display position of the subject being tracked in real time to match the composition after changing the display position of the subject being tracked as described above. That is, the control calculation unit 109, upon receiving the information that the display position (target position) and size (zoom) of the subject being tracked have been changed in the display position operation mode during automatic tracking shooting described above, performs PTZ control of the pan / tilt camera 101 during automatic tracking shooting to match the changed composition. Note that a detailed explanation of the specific control changes of the PTZ control due to the change in composition will be omitted as this is a technology that has already been put into practical use.

[0042] Next, in step S207, it is determined whether to terminate the control processing based on the display position operation mode during automatic tracking shooting. For example, the control calculation unit 109 terminates the processing shown in the flowchart of Figure 2 when the confirmation button on the operation unit 112 is pressed, and the composition change due to changing the display position and size of the subject to be tracked is confirmed. Also, for example, the control calculation unit 109 terminates the processing shown in the flowchart of Figure 2 when the system switches from the display position operation mode to, for example, the PTZ operation mode without fixing the change in the position of the display position operation frame. On the other hand, if the control calculation unit 109 does not determine to terminate in step S207, it returns to step S203 and continues the processing shown in this flowchart.

[0043] As described above, according to the first embodiment, while continuing automatic tracking shooting, the operator can accurately recognize the movement speed and position of the display position of the subject being tracked and perform display position operations. In other words, according to this embodiment, while continuing automatic tracking shooting, the operator can change the display position of the subject being tracked to the composition they intend.

[0044] In the first embodiment, an example was given in which the movement speed of the display position operation frame was displayed by the length of the movement display icon as a display method for the display position operation GUI, but the method is not limited to this example. For example, as shown in the transition example from (4a) to (4b) in Figure 4, afterimages may be added to the movement of the display position operation frame 420. That is, the movement of the display position operation frame 420 may be represented as a frame movement with an afterimage effect, such as an afterimage frame 422, and the movement speed may be expressed by the magnitude of the afterimage in the afterimage frame 422. Alternatively, as shown in the transition example from (4c) to (4d) in Figure 4, the movement display of the display position operation frame 420 may be represented by the amount of frame displacement between the two frames: the display position operation frame 420 after movement and the display position operation frame 423 before movement that is paired with it. As shown in these transition examples in Figure 4, any method of representation that can visually convey the movement speed and movement position of the display position operation frame to the operator may be used.

[0045] <Second Embodiment> In the first embodiment, the display position is moved in response to the operator's operation, and then this display position is reflected in the automatic tracking control of the subject being tracked. This results in latency until the change in display position is reflected. To improve this, in the second embodiment, the operation information when the display position operation frame is moved is reflected in the PTZ control of the automatic tracking control in real time, thereby realizing automatic tracking shooting of the subject being tracked that takes into account the movement of the display position. In the second embodiment, the configuration of the automatic shooting system, the pan / tilt camera 101, the control device 104, the monitor 113, etc., are the same as in Figures 1 and 2, so their illustration and description are omitted. The control processing by the control device 104 according to the second embodiment will be described below, focusing on the differences from the first embodiment.

[0046] Figure 5 is a flowchart showing the control flow of the control device 104 when the control device 104 according to the second embodiment is switched to display position operation mode during automatic tracking shooting, and a display position operation is performed to change the display position of the subject being tracked to a composition intended by the operator. Note that the processing of steps S501 to S507 is the same as the processing of steps S201 to S207 explained using the flowchart in Figure 2 in the first embodiment, so the explanation is omitted.

[0047] In the second embodiment, in order to reflect the display position movement operation information in real time to the PTZ control of the automatic tracking control, step S508 is added between steps S505 and S506 as processing by the control calculation unit 109. The control calculation unit 109 of the control device 104 is responsible for the PTZ control of the pan-tilt camera 101, and therefore understands the pan, tilt, and zoom state, and knows the angle of view during automatic tracking shooting. Therefore, in step S508, the control calculation unit 109 of the control device 104 according to the second embodiment converts the amount of operation when the operator moves the display position operation frame through the operation unit 112 into a PTZ movement amount, and adds it to the PTZ control amount for automatic tracking.

[0048] As an example of PTZ control, movement in the pan axis direction will be explained with reference to Figure 6. In the example in Figure 6, the case where the operator moves the display position control frame (subject display position) from movement amount A to movement amount B in the horizontal direction, which is the pan direction, will be explained. Here, if we let Hl be the horizontal (horizontal) magnitude determined by the field of view, Hm be the horizontal movement amount of the display position control frame, Sl be the horizontal field of view, and Sc be the pan correction angle, then the pan correction angle Sc applied to the PTZ control of the automatic tracking control can be calculated using the following equation (1).

[0049] Sc=Sl×Hl / Hm Formula (1)

[0050] Furthermore, the correction of the PTZ control of the automatic tracking control based on the operation information when the display position operation frame is moved does not necessarily need to be a precise correction. That is, since tracking processing is performed in step S506 in accordance with the movement of the display position operation frame, even if there is a small error, it will be corrected in the processing in step S506. Also, for example, if precise correction is performed, there is a risk of the response becoming too quick, so in practice, the amount of correction may be adjusted according to the characteristics of the automatic tracking camera system and the shooting application. Regarding the correction of the zoom direction, it is conceivable to directly reflect the operation information of enlarging and reducing the display size by operating the seesaw key of the operation unit 112 into the zoom control and correct it there.

[0051] As described above, in the control device 104 according to the second embodiment, while continuing automatic tracking and shooting of the subject being tracked, the operator can accurately recognize the movement speed and position when the display position of the subject being tracked is changed. In other words, according to the second embodiment, the latency until the change in the display position of the subject being tracked is reflected can be improved, and the display position of the subject being tracked can be changed to the composition intended by the operator.

[0052] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more of the functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0053] A processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gateway (FPGA). It may also include a digital signal processor (DSP), a dataflow processor (DFP), or a neural processing unit (NPU).

[0054] This embodiment includes the following configurations, methods, and programs. (Composition 1) A control device that controls a camera with a pan / tilt head to track and photograph a target subject, A GUI display means that, during tracking and shooting of the subject to be tracked, displays on the screen a GUI that shows the speed and position of movement of the display position of the subject to be tracked, in response to the operator's operation to move the display position of the subject to be tracked on the screen, by compositing it onto the image captured by the camera with a pan / tilt head. A tracking control means controls the tracking and shooting of the subject to be tracked by the camera with a pan / tilt head, based on the display position of the subject to be tracked on the screen set according to the operation of the operator using the GUI and the detection result of the subject to be tracked that appears in the image captured by the camera with a pan / tilt head. A control device characterized by having the following features. (Configuration 2) The control device according to Configuration 1, characterized in that the tracking control means corrects the amount of pan, tilt, and zoom control for tracking and photographing the subject to be tracked with the camera with a pan / tilt head based on operation information of the operator's operation to move the display position of the subject to be tracked. (Composition 3) The control device according to configuration 1 or 2, characterized in that the GUI display means generates a display position operation frame representing the display position of the subject to be tracked, and moves the display position operation frame in response to an operation by the operator. (Composition 4) The control device according to configuration 3, characterized in that the GUI includes an operation display icon corresponding to an operation by the operator and a movement display icon corresponding to the movement of the display position of the subject being tracked. (Composition 5) The aforementioned operation display icon is an arrow-shaped icon that represents the direction and amount of operation used by the operator to move the display position of the subject being tracked. The control device according to configuration 4, characterized in that the movement display icon is an arrow-shaped icon representing the direction and amount of movement of the display position operation frame. (Composition 6) The GUI display means is The length of the operation display icon is changed according to the amount of movement performed by the operator to move the display position of the subject being tracked. The control device according to configuration 5, characterized in that the length of the moving display icon is changed according to the amount of movement of the display position operation frame. (Composition 7) The control device according to configuration 3, characterized in that the GUI display means adds an afterimage to the movement of the display position operation frame in response to an operation by the operator to move the display position of the subject to be tracked. (Composition 8) The control device according to configuration 7, characterized in that the GUI display means changes the amount of afterimage according to the movement speed of the display position operation frame. (Composition 9) The control device according to configuration 3, characterized in that the GUI display means displays two display position operation frames: one before the operator moves the display position of the subject to be tracked, and another after the operator moves the display position. (Composition 10) The control device according to configuration 9, characterized in that the GUI display means changes the amount of displacement of the frame position between the display position operation frame before the display position is moved and the display position operation frame after the display position is moved, according to the movement speed of the display position operation frame. (Method 1) A control method for controlling a camera with a tripod head to track and photograph a target subject, A GUI display step is performed in which a GUI indicating the movement speed and position of the display position of the target subject on the screen, in response to the operator's operation to move the display position of the target subject on the screen during tracking shooting, is superimposed on the image captured by the camera with a pan / tilt head and displayed on the screen. A tracking control step that controls tracking and shooting of the subject to be tracked by the camera with a pan / tilt head based on the display position of the subject to be tracked on the screen set according to the operation of the operator using the GUI and the detection result of the subject to be tracked that appears in the image captured by the camera with a pan / tilt head, A control method characterized by having the following features. (Program 1) A program that causes a computer to function as a control device described in any one of configurations 1 to 10. [Explanation of Symbols]

[0055] 101: Pan / tilt camera, 102: Camera unit, 103: Pan / tilt camera control unit, 104: Control device, 105: Signal processing unit, 106: Control I / F, 107: Subject detection unit, 108: Image synthesis unit, 109: Control calculation unit, 110: Operation GUI unit, 111: Output unit, 112: Operation unit, 113: Monitor

Claims

1. A control device that controls a camera with a pan / tilt head to track and photograph a target subject, A GUI display means that, during tracking and shooting of the subject to be tracked, displays on the screen a GUI indicating the speed and position of the movement of the display position of the subject to be tracked, in response to the operator's operation to move the display position of the subject to be tracked on the screen, by compositing it onto the image captured by the camera with a pan / tilt head. A tracking control means controls the tracking and shooting of the subject to be tracked by the camera with a pan / tilt head, based on the display position of the subject to be tracked on the screen set according to the operator's operation using the GUI and the detection result of the subject to be tracked that appears in the image captured by the camera with a pan / tilt head. A control device characterized by having the following features.

2. The control device according to claim 1, characterized in that the tracking control means corrects the amount of pan, tilt, and zoom control for tracking and photographing the subject to be tracked with the camera with a pan / tilt head based on operation information of the operator's operation to move the display position of the subject to be tracked.

3. The control device according to claim 1, characterized in that the GUI display means generates a display position operation frame representing the display position of the subject to be tracked, and moves the display position operation frame in response to an operation by the operator.

4. The control device according to claim 3, characterized in that the GUI includes an operation display icon corresponding to an operation by the operator and a movement display icon corresponding to a movement of the display position of the subject being tracked.

5. The aforementioned operation display icon is an arrow-shaped icon that represents the direction and amount of operation used by the operator to move the display position of the subject being tracked. The control device according to claim 4, characterized in that the movement display icon is an arrow-shaped icon representing the direction and amount of movement of the display position operation frame.

6. The GUI display means is The length of the operation display icon is changed according to the amount of movement performed by the operator to move the display position of the subject being tracked. The control device according to claim 5, characterized in that the length of the moving display icon is changed according to the amount of movement of the display position operation frame.

7. The control device according to claim 3, characterized in that the GUI display means adds an afterimage to the movement of the display position operation frame in response to an operation by the operator to move the display position of the subject to be tracked.

8. The control device according to claim 7, characterized in that the GUI display means changes the amount of afterimage according to the movement speed of the display position operation frame.

9. The control device according to claim 3, characterized in that the GUI display means displays two display position operation frames: a display position operation frame before the operator moves the display position of the subject to be tracked, and a display position operation frame after the operator moves the display position.

10. The control device according to claim 9, characterized in that the GUI display means changes the amount of displacement of the frame position between the display position operation frame before the display position is moved and the display position operation frame after the display position is moved, according to the movement speed of the display position operation frame.

11. A control method for controlling a camera with a tripod head to track and photograph a target subject, A GUI display step is performed in which, during tracking and shooting of the subject to be tracked, a GUI indicating the movement speed and position of the display position corresponding to the operator's operation to move the display position of the subject to be tracked on the screen is superimposed on the image captured by the camera with a pan / tilt head and displayed on the screen; A tracking control step that controls tracking and shooting of the subject to be tracked by the camera with a pan / tilt head based on the display position of the subject to be tracked on the screen set according to the operation of the operator using the GUI and the detection result of the subject to be tracked that appears in the image captured by the camera with a pan / tilt head, A control method characterized by having the following features.

12. Computers, A control device that controls a camera with a pan / tilt head to track and photograph a target subject, A GUI display means that, during tracking and shooting of the subject to be tracked, displays on the screen a GUI indicating the speed and position of the movement of the display position of the subject to be tracked, in response to the operator's operation to move the display position of the subject to be tracked on the screen, by compositing it onto the image captured by the camera with a pan / tilt head. A tracking control means controls the tracking and shooting of the subject to be tracked by the camera with a pan / tilt head, based on the display position of the subject to be tracked on the screen set according to the operator's operation using the GUI and the detection result of the subject to be tracked that appears in the image captured by the camera with a pan / tilt head. A program that causes a control device to function as such.

Images