Controller and control method for the same

Abstract

To reduce complexity of setting in automatic tracking.SOLUTION: An imaging apparatus has a tracking function of tracking a subject to perform photographing, and a cropping function of generating a cropped video obtained by cutting a partial area from a photographed video. A controller controlling the imaging apparatus comprises: first setting means that sets in the imaging apparatus, a target position in the photographed video in which a subject stays in the tracking function; second setting means that sets, in the imaging apparatus, a cropped area in the photographed video cut out with the cropping function; and display control means that displays, on a display, a graphical user interface (GUI) receiving at least one of the setting of the target position and the setting of the cropped area. The display control means displays, on the display, a first GUI component indicating the target position and a second GUI component indicating the cropped area.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to subject tracking control. [Background technology]

[0002] There is a known technology for a camera capable of controlling pan-tilt-zoom (PTZ camera) that detects a subject to be tracked specified by a user from a captured image and automatically tracks the subject. Patent Document 1 discloses a method that enables a target position for automatically tracking a subject to be specified. With this method, the camera operator can adjust the target position, allowing the subject to be continuously captured at a desired position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-108425 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when performing automatic tracking during shooting at a close-up angle of view (an angle of view in which the subject occupies a high proportion of the shooting angle of view), the subject may be lost if the subject moves. It is conceivable that the loss of the subject may be reduced by shooting at an angle of view wider than the close-up angle described above and performing automatic tracking control. However, if the angle of view desired by the user is a close-up angle of view, it becomes necessary to separately crop a range equivalent to the close-up angle of view. In this case, when setting the target position for automatic tracking, the user must perform the setting operation while being conscious of the crop frame, which is complicated and places a significant burden on the user.

[0005] The present invention has been made in view of the above problems, and aims to provide a technique for reducing the complexity of settings in automatic tracking control. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, a control device according to the present invention has the following configuration: That is, the control device controls an imaging device, The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, The control device a first setting means for setting a target position in the captured image where the subject stays in the tracking function in the imaging device; a second setting means for setting a crop area in the captured image cut out by the crop function in the imaging device; a display control means for displaying, on a display unit, a graphical user interface (GUI) that accepts at least one of the setting of the target position and the setting of the crop region; Equipped with The display control means displays a first GUI component indicating the target position and a second GUI component indicating the crop region on the display unit. Alternatively, a control device that controls an imaging device, The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, The control device a first setting means for selecting a subject to be tracked in the tracking function from among a plurality of subjects in the captured video and setting the selected subject in the imaging device; a second setting means for setting a crop area in the captured image cut out by the crop function in the imaging device; a display control means for displaying a graphical user interface (GUI) on a display unit, the GUI accepting at least one of the selection of the subject and the setting of the crop region; Equipped with The display control means displays, on the display unit, a first GUI component indicating the selected subject and a second GUI component indicating the crop region. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique for reducing the complexity of settings in automatic tracking control. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating the overall configuration of a system. [Figure 2] FIG. 2 is a diagram illustrating the hardware configuration of each device. [Figure 3] 10 is a flowchart of a tracking operation. [Figure 4] FIG. 1 is a diagram illustrating a problem with automatic tracking. [Figure 5] FIG. 10 is a diagram illustrating generation of a wipe screen. [Figure 6] 10 is a flowchart of automatic tracking setting (first embodiment). [Figure 7] FIG. 2 is a diagram illustrating a display in a GUI (first embodiment). [Figure 8] 10 is a flowchart of automatic tracking setting (second embodiment). [Figure 9] FIG. 10 is a diagram illustrating a display in a GUI (second embodiment). DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) As a first embodiment of a control device according to the present invention, a controller for controlling a PTZ camera will be described below as an example.

[0011] <Overall system configuration and configuration of each device> FIG. 1 is a diagram showing the overall configuration of an automatic tracking system 10. The automatic tracking system 10 includes a camera 100 and a controller 200. The camera 100 is an imaging device for capturing an image of a subject 20, and the controller 200 is a control device for remotely controlling the camera 100. The camera 100 and the controller 200 are configured to be able to communicate data with each other via a network 300. The network 300 includes networks such as a local area network (LAN) and the Internet. Note that any connection format or communication protocol may be used as long as mutual communication is possible. For example, devices may be directly connected using a serial communication cable without using a network.

[0012] Fig. 2 is a diagram showing the hardware configuration of the camera 100 and the controller 200. Note that the configuration shown in Fig. 2 is merely an example of the hardware configuration of the camera 100 and the controller 200, and can be changed / modified as appropriate.

[0013] Camera 100 has a mechanism capable of pan-tilt-zoom (PTZ) control for changing the shooting direction and shooting angle of view. Furthermore, camera 100 has a tracking function that detects a subject from a captured image and autonomously changes the shooting direction based on the detection results to track the subject. In the following description, camera 100 is assumed to be a camera that optically controls pan-tilt-zoom and also performs crop processing (crop function) by image processing unit 106. However, camera 100 may also perform pan-tilt-zoom digitally by crop processing by image processing unit 106.

[0014] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the overall operation of the camera 100, and also executes or controls various processes, which will be described later, as processes performed by the camera 100.

[0015] The RAM 102 is a high-speed storage device such as a DRAM. The RAM 102 has an area for storing computer programs and data loaded from the ROM 103, and an area for storing captured images output from the image processing unit 106. The RAM 102 also has an area for storing various information received from the controller 200 via a network interface (I / F) 105, and a work area used by the CPU 101 and the inference unit 104 when executing various processes. In this way, the RAM 102 can provide areas for storing various types of data as needed.

[0016] ROM 103 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. ROM 103 stores setting data for camera 100, computer programs and data related to the startup of camera 100, and computer programs and data related to basic operations of camera 100. ROM 103 also stores computer programs and data for causing CPU 101 and inference unit 104 to execute or control various processes, which will be described later as processes performed by camera 100.

[0017] The inference unit 104 performs inference processing to estimate the presence or absence, position, etc. of a subject from a captured image. The inference unit 104 is, for example, a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). While a GPU is generally effective for use in inference processing, equivalent functions may be realized by a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Also, the inference unit 104 may be configured so that part or all of its processing is performed by the CPU 101.

[0018] The network I / F 105 is an interface for connecting to the network 300, and is responsible for communication with an external device such as the controller 200 via a communication medium such as Ethernet (registered trademark). Note that a separate serial communication I / F may be prepared and used for communication.

[0019] Image processing unit 106 generates a captured image, which is data in a predetermined format, based on the video signal output from image sensor 107. The generated captured image is compressed as necessary and then output to RAM 102. Note that image processing unit 106 may perform various types of processing on the image represented by the video signal acquired from image sensor 107, such as image quality adjustments such as color correction, exposure correction, and sharpness correction, and cropping to cut out only a predetermined area. These processes may also be performed in accordance with instructions received from controller 200 via network I / F 105.

[0020] The image sensor 107 outputs a video signal based on an optical image of a subject formed by an imaging optical system. The image sensor 107 may be, for example, a photodiode, a CCD (Charge Coupled Device) sensor, or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0021] The drive I / F 108 is an interface for transmitting and receiving instruction signals such as control signals between the CPU 101 and the drive unit 109. The drive unit 109 is a drive mechanism for changing the shooting direction of the camera 100, and includes a mechanical drive system, a drive motor, and the like. The drive unit 109 performs pan (P) control for changing the shooting direction in the horizontal (sideways) direction and tilt (T) control for changing the shooting direction in the vertical (lengthways) direction in accordance with instructions received from the CPU 101 via the drive I / F 108. The drive unit 109 also performs optical zoom (Z) control for optically changing the shooting angle of view.

[0022] The video output I / F 110 is an interface for externally outputting the captured image generated by the image processing unit 106. For example, it is configured as an interface conforming to SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark).

[0023] The above-mentioned CPU 101 , RAM 102 , ROM 103 , inference unit 104 , network I / F 105 , image processing unit 106 , drive I / F 108 , and video output I / F 110 are connected to a system bus 111 .

[0024] The image processing unit 106 is also configured to be able to output a cropped image obtained by cutting out a predetermined area (crop area) of the captured image based on crop processing instructed by the CPU 101. For example, the CPU 101 reads out the crop setting stored in the RAM 102 and instructs the image processing unit 106 to perform crop processing.

[0025] Controller 200 receives captured images transmitted from camera 100 and transmits control signals to camera 100 via network 300. For example, based on an operation received from a user, controller 200 can transmit a target position (position in a captured image) of a tracking subject to camera 100 as a control signal. In other words, the user can operate controller 200 to instruct camera 100 to perform tracking and capturing.

[0026] The CPU 201 executes various processes using computer programs and data stored in the RAM 202. As a result, the CPU 201 controls the overall operation of the controller 200, and also executes or controls various processes, which will be described later, as processes performed by the controller 200.

[0027] The RAM 202 is a high-speed storage device such as a DRAM. The RAM 202 has an area for storing computer programs and data loaded from the ROM 203, and an area for storing various data received from the camera 100 via the network I / F 204. The RAM 202 also has a work area used by the CPU 201 when executing various processes. In this way, the RAM 202 can provide areas for storing various types of data as needed.

[0028] The ROM 203 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The ROM 203 stores setting data for the controller 200, computer programs and data related to the startup of the controller 200, and computer programs and data related to the basic operation of the controller 200. The ROM 203 also stores computer programs and data for causing the CPU 201 to execute or control various processes, which will be described later as processes performed by the controller 200.

[0029] The network I / F 204 is an interface for connecting to the network 300, and is responsible for communication with external devices such as the camera 100 via a communication medium such as Ethernet (registered trademark). For example, communication between the controller 200 and the camera 100 includes sending control commands to the camera 100, receiving captured images from the camera 100, and the like.

[0030] Display unit 205 is a display unit having a screen such as a liquid crystal display, and displays captured images received from camera 100, a setting screen for controller 200, etc. In the following explanation, a case will be described in which display unit 205 is a touch panel screen that can accept operations from a user. Note that display unit 205 may be configured as an external device rather than being built into controller 200. For example, an external display device may be connected to controller 200, and a display control unit in controller 200 may display captured images, a setting screen, etc. on the external display device.

[0031] The user input I / F 206 is an interface for accepting user operations on the controller 200. For example, it includes a mouse, a keyboard, a button, a dial, a joystick, a touch panel, and the like.

[0032] The above-mentioned CPU 201, RAM 202, ROM 203, network I / F 204, display unit 205, and user input I / F 206 are connected to a system bus 207. The controller 200 can also be configured by a PC (personal computer).

[0033] <Camera 100 Operation> Fig. 3 is a flowchart of the tracking operation in the camera. Specifically, it is a diagram explaining the control by which camera 100 detects a subject and tracks the subject. Fig. 3 shows a loop process of capturing an image, identifying the position of the subject from the captured image, and tracking the subject.

[0034] In S301, the CPU 101 of the camera 100 acquires a captured image (frame image) from the image processing unit 106 and stores it in the RAM 102.

[0035] In S302, CPU 101 detects a subject included in the captured video acquired in S301. Specifically, CPU 101 reads the captured image from RAM 102, inputs it to inference unit 104, and stores the type of subject and position information (subject position) in the captured video inferred by inference unit 104 in RAM 102. Inference unit 104 has a trained model created using a machine learning method such as deep learning, receives an image as input data, and outputs the type of subject such as a person, position information, and a score indicating likelihood as output data. Here, the position information will be described as the position coordinates of a target (a person's face) in the captured image.

[0036] In S303, CPU 101 determines whether the subject position of the subject included in the captured video acquired in S301 matches the position (target position) at which the subject is to be stopped in the captured video. Specifically, CPU 101 determines whether the subject position stored in RAM 102 in S302 matches a pre-specified target position. If it is determined that the position information matches, steps S304 to S305 are skipped and the process proceeds to loop processing for the next captured image. On the other hand, if it is determined that the position information does not match, the process proceeds to S304. Note that it is preferable to configure the camera 100 so that if the difference between the subject position and the target position is within a predetermined range, they are considered to match. This makes it possible to prevent excessive drive control from being performed on camera 100.

[0037] Here, the target position is assumed to be stored in RAM 102 by CPU 101 when the system is started up. Furthermore, the target position may include information indicating the size of the subject in the captured image (absolute size / relative size) in addition to information indicating the coordinates (horizontal and vertical directions) on the captured image, as in target position 404 described later with reference to FIG. 4. For example, by setting a larger target size of the subject in the captured image, it is possible to obtain a captured image in which the subject appears larger. In this way, it is possible to obtain a captured image in which the subject appears at a size desired by the user, depending on the target size of the subject. Note that the target position on the captured image at the time of startup can be any position, such as the center of the captured image, the position at the time the system was last shut down, or a position previously designated by the user.

[0038] In S304, CPU 101 calculates driving parameters for pan / tilt (PT) control required to match the subject position with the target position. Specifically, CPU 101 derives the difference between the current subject position and the target position, and calculates driving parameters according to the difference. Here, the driving parameters refer to parameters for controlling motors (not shown) in the pan direction and tilt direction included in driving unit 109. Note that if a subject size target is set, parameters for controlling a zoom motor (not shown) are also calculated. CPU 101 stores the calculated driving parameters in RAM 202.

[0039] In S305, CPU 101 reads out drive parameters from RAM 202 and controls drive unit 109 via drive I / F 108. As a result, the shooting direction (pan / tilt direction) of camera 100 is changed so that the subject position matches the target position. Also, the shooting angle of view (zoom magnification) of camera 100 is changed so that the size of the subject matches the target subject size.

[0040] <Challenges with tracking at close angles of view> First, the issues that arise when controlling the target position and performing automatic tracking at a "close-up angle of view" will be described with reference to Fig. 4. Note that a close-up angle of view means an angle of view where the subject occupies a high proportion of the shooting angle of view 402, as shown in Fig. 4(c), for example.

[0041] 4(a) shows a frontal shot of three people watching a recorded video 403 (such as a movie). The image also shows the angle of view 402 of the camera 100 shooting the subject 401 (one person).

[0042] 4(b), (d), and (f) are diagrams simulating a superimposed image in which a captured image of a subject 401 (image at a shooting angle of view 402) is superimposed as a wipe screen on a recorded image 403. A wipe screen refers to a screen in which an image is superimposed on a partial area of ​​a display screen.

[0043] 4(c) and (e) are diagrams simulating a graphical user interface (GUI) for setting a target position 404. Here, the target position 404 indicates a target area of ​​a "person's face" in a shooting angle of view 402. As shown in FIG. 4(c), by setting the target position 404 so that its proportion in the shooting angle of view 402 is high, it is possible to perform automatic tracking so that the face of the subject 401 is captured in a large image. Then, by superimposing the captured image as a wipe screen as shown in FIG. 4(d), the superimposed image shown in FIG. 4(b) can be generated.

[0044] However, if the subject 401 makes a sudden movement while automatic tracking is being performed, the face of the subject 401 may move out of the imaging angle of view 402. For example, if the subject 401 stands up from the sitting position shown in FIG. 4(a), the face of the subject 401 moves out of (is lost from) the imaging angle of view 402. As a result, the image of the imaging angle of view 402 captured by the camera 100 will be in a subject-lost state as shown in FIG. 4(e). In this case, the image will be a superimposed image as shown in FIG. 4(f). In particular, the subject-lost state is likely to occur in the case of a close-up angle of view (an angle of view in which the subject occupies a high proportion of the imaging angle of view 402) as shown in FIG. 4(c).

[0045] <Automatic tracking combined with wide-angle shooting and cropping> Therefore, in the first embodiment, a subject is photographed at a relatively wider angle than the above-mentioned "close-up angle of view," and a wipe screen is generated by performing a crop process on the image photographed at that wide angle. That is, by performing a crop process on the image photographed at the wide angle, a wipe screen that is an image equivalent to the "close-up angle of view" is generated.

[0046] 5(a) and (d) show images of three people viewed from the front of a recorded video 403. Also shown are a tracking angle of view 502 and a crop area 501 of the camera 100 capturing the subject 401 (one person).

[0047] 5(b) and (e) are diagrams simulating a GUI for setting a target position 404 within a captured image. Also, Fig. 5(c) and (f) are diagrams simulating a superimposed image in which a captured image of a subject 401 (image of a crop area 501) is superimposed on a recorded image 403 as a wipe screen.

[0048] 5(a), automatic tracking and shooting of subject 401 is performed at tracking angle of view 502, and a wipe screen is generated by cutting out crop area 501 from tracking angle of view 502. By setting crop area 501 to be equivalent to shooting angle of view 402, it is possible to generate a wipe screen with an angle of view equivalent to shooting angle of view 402 (close-up angle of view) in FIG.

[0049] Because the tracking angle of view 502 is wider than the shooting angle of view 402, the tracking target (here, a person's face) is less likely to be lost. That is, even if the subject makes a sudden movement (standing up) as shown in FIG. 5(d), tracking can be continued normally as shown in FIG. 5(e). As a result, as shown in FIG. 5(f), a superimposed image similar to that of FIG. 4(b) can be created.

[0050] <Operation of the controller 200> 6 is a flowchart of the automatic tracking setting in the first embodiment. Specifically, it is a diagram for explaining the operation when the controller 200 sets the crop or the target position of the camera 100. In particular, the automatic tracking setting described below is characterized in that the crop setting and the target position setting are not performed independently, but can be set in conjunction with each other.

[0051] In S601, the CPU 201 of the controller 200 determines whether or not to continue the processing (control flow) of Fig. 6. For example, the CPU 201 determines whether or not to continue the processing by checking whether or not a command indicating the end of this control flow has been received via the network I / F 204 or the user input I / F 206. If the processing is to be continued, the process proceeds to S602, and if the processing is not to be continued, the process ends.

[0052] In S602, the CPU 201 acquires camera information of the camera 100. Here, the camera information is information including a captured image, a target position, and a crop setting. Specifically, the CPU 201 transmits a camera information acquisition request to the camera 100 via the network I / F 204. Upon receiving the camera information acquisition request, the CPU 101 of the camera 100 reads out the captured image, the target position, and the crop setting stored in the RAM 102 and transmits them to the controller 200 via the network I / F 105. The controller 200 stores the received camera information (captured image, the target position, and the crop setting) in the RAM 202.

[0053] In S603, the CPU 201 determines whether the setting mode is the target position adjustment mode. Specifically, the CPU 201 obtains the setting mode from the RAM 202, and proceeds to S604 if the setting mode is the target position adjustment mode; otherwise, the CPU 201 proceeds to S607. The setting mode may include a target position adjustment mode for setting a target position, a crop adjustment mode for setting a crop frame, or other modes. When the setting mode is changed via the network I / F 204 or the user input I / F 206, the CPU 201 stores the changed setting mode in the RAM 202.

[0054] In S604, the CPU 201 displays a graphical user interface (GUI) for adjusting the target position. Details of the GUI will be described later with reference to Fig. 7. Specifically, the CPU 201 reads out the captured image and the target position from the RAM 202, and displays the GUI for adjusting the target position on the display unit 205.

[0055] The GUI for adjusting the target position may be any GUI that allows the user to visually understand the target position. For example, a frame indicating coordinates on the captured image or an icon resembling a human body may be displayed. Also, a subject detection frame created based on the subject position acquired from camera 100 may be displayed. In this case, information including the subject position is also received from camera 100 in advance in S602.

[0056] In S605, the CPU 201 displays a crop frame (a GUI component indicating the range in which cropping is performed) on the GUI. Specifically, the CPU 201 reads the crop setting from the RAM 202 and displays the crop frame superimposed on the captured image displayed on the display unit 205.

[0057] This makes it easy for the user to understand the range in which cropping is performed and the target position or subject position in the camera 100. In other words, the user can set automatic tracking while viewing the crop state.

[0058] In S606, CPU 201 executes target position setting. Specifically, CPU 201 receives a target position change request from the user via user input I / F 206. The user changes the target position based on the GUI displayed in S604 and S605. CPU 201 transmits the changed target position received via user input I / F 206 to camera 100 via network I / F 204. Upon receiving the instruction to change the target position, camera 100 stores the received target position in RAM 102. This allows the settings of camera 100 to be changed so that the subject is captured at the changed target position.

[0059] Here, the GUI displayed on the display unit 205 and user operations will be described with reference to Figures 7(a) to 7(d). Figure 7 is a diagram for explaining the display of the GUI in the first embodiment. The GUI is a user interface that is displayed on the display unit 205, provides information to the user, and accepts operations from the user.

[0060] 7(a) and (c) show exemplary GUIs in which GUI components (a target position icon 702 and a crop frame 703) are superimposed on a video 700 in which three people are photographed from the front. The target position icon 702 is a GUI component that indicates the target position of the tracking target (the face of the subject 701) in the captured image 700. The crop frame 703 is a GUI component that indicates the range in which cropping is performed. FIGS. 7(b) and (d) show images cut out by cropping from FIGS. 7(a) and (c), respectively.

[0061] In FIG. 7(a), the target position icon 702 (size of the target position) occupies a small proportion of the crop frame 703. Furthermore, the target position icon 702 is set below the crop frame 703. In this case, the image cut out by the cropping process (cropped image) is as shown in FIG. 7(b). That is, the subject 701 is too small and only a portion of the subject 701 is captured in the cropped image, allowing the user to understand that the crop range and target position have not been set appropriately.

[0062] Therefore, while viewing the GUI displayed in S606, the user operates to move the target position icon 702 to the desired position within the cropping frame 703. For example, as shown in FIG. 7(c), the target position icon 702 is placed above the cropping frame 703 and its size is increased. This arrangement results in the desired cropped image as shown in FIG. 7(d). In other words, by providing a GUI that allows the user to change the target position while simultaneously displaying the cropping frame and the target position icon, it is possible to reduce the need for complicated setting operations.

[0063] In S607, the CPU 201 determines whether the setting mode is the crop adjustment mode. If it is the crop adjustment mode, the process proceeds to S608; if not, the process proceeds to S601. In S608, the CPU 201 displays a GUI for crop adjustment. Details of the GUI will be described later with reference to FIG. 7. Specifically, the CPU 201 reads the captured image and crop settings from the RAM 202, and displays the GUI for crop adjustment on the display unit 205. Note that the GUI for crop adjustment may be any GUI that allows the user to visually understand the crop area. For example, a rectangular frame indicating the crop area or characters indicating the coordinates may be displayed on the captured image.

[0064] In S609, CPU 201 displays the target position (GUI component indicating the target of automatic tracking) on ​​the GUI. Specifically, CPU 201 reads the target position from RAM 202 and displays the target position superimposed on the captured image displayed on display unit 205. In addition, a subject detection frame created based on the subject position acquired from camera 100 may also be displayed. In this case, information including the subject position is also received from camera 100 in advance in S602.

[0065] This makes it easy for the user to understand the range in which cropping is performed and the target position or subject position in camera 100. In other words, the user can set cropping while viewing the setting status of auto-tracking.

[0066] In S610, CPU 201 executes cropping frame setting. Specifically, CPU 201 receives a cropping frame change request from the user via user input I / F 206. The user changes the cropping frame setting based on the GUI displayed in S608 and S609. CPU 201 transmits the changed cropping frame setting received via user input I / F 206 to camera 100 via network I / F 204. Upon receiving the instruction to change the cropping frame setting, camera 100 stores the received cropping frame setting in RAM 102. This allows the settings of camera 100 to be changed so that cropping processing is performed using the changed cropping frame setting.

[0067] Here, the GUI displayed on the display unit 205 and user operations will be described with reference to Figs. 7(c) to (f). Figs. 7(c) and (e) exemplify GUIs in which GUI components (a target position icon 702 and a crop frame 703) are superimposed on an image 700 in which a person is photographed from the front. The target position icon 702 is a GUI component that indicates the target position of the tracking target (the face of the subject 701) in the captured image 700. The crop frame 703 is a GUI component that indicates the range in which cropping processing is performed. Figs. 7(d) and (f) are diagrams showing images cut out by cropping processing from Figs. 7(c) and (e), respectively.

[0068] In FIG. 7(e), the crop frame 703 is small. Furthermore, the crop frame 703 is positioned offset from the target position icon 702. In this case, the image cut out by the cropping process (cropped image) is as shown in FIG. 7(f). That is, the subject 701 is too large and only a portion of the subject 701 is captured in the cropped image, allowing the user to understand that the crop range and target position have not been set appropriately.

[0069] Therefore, while viewing the GUI displayed in S610, the user operates the cropping frame 703 so that the target position icon 702 is positioned at the desired position within the cropping frame 703. For example, as shown in FIG. 7(c), the cropping frame 703 is moved and enlarged so that the target position icon 702 is positioned at the top of the cropping frame 703. This arrangement results in the desired cropped image as shown in FIG. 7(d). In other words, by providing a GUI that allows the user to change the target position while simultaneously displaying the cropping frame and the target position icon, it is possible to reduce the need for complicated setting operations.

[0070] As described above, according to the first embodiment, crop frame setting and target position setting in automatic tracking that combines "wide-angle shooting + crop" are performed in conjunction with each other. In particular, a GUI is provided that allows the target position to be changed while simultaneously displaying the crop frame and target position icon. By using this GUI, it is possible to reduce the amount of complicated setting operations.

[0071] (Variation) In the first embodiment described above, the complexity of setting the target position for automatic tracking within the area cut out by cropping is reduced, but this is not the only option. For example, the present invention can also be applied to settings when selecting a tracking subject for automatic tracking.

[0072] For example, if the user wishes to realize the wipe screen described with reference to Fig. 5 with a different subject, the user performs an operation to select the different subject as a tracking subject. At that time, if camera 100 cannot detect the different subject, the different subject cannot be selected as a tracking subject. Therefore, when performing an operation to select the different subject as a tracking subject, a frame imitating the subject position is displayed so that the user can identify whether the different subject can be selected as a tracking subject.

[0073] By displaying a frame that resembles the subject position on the same screen as the crop frame, the user can determine whether another subject can be selected as a tracking subject while looking at the crop frame, which is the wipe screen. This allows the user to select another subject immediately when they want to. This reduces the complexity of the setting operation for selecting a tracking subject to be automatically tracked.

[0074] In this case, in S602, the CPU 201 receives information including all detected subject positions from the camera 100 and stores it in the RAM 202. Next, in S604, the CPU 201 reads out all detected subject positions and, based on that, displays a GUI for specifying a subject that displays a frame or the like that resembles the subject position. Then, in S606, the CPU 201 accepts a request from the user to change the subject to be tracked, and instructs the camera 100 to change the tracking target to the subject to be tracked specified by the user.

[0075] In the first embodiment, a GUI displaying GUI components for both cropping frame setting and target position setting is described as being displayed on the display unit 205. However, each GUI component may be individually displayable. In this case, the CPU 201 stores a target position display flag and a cropping frame display flag in the RAM 202 according to a user-specified display / non-display setting. When each flag is on (or 1), it indicates "display enabled," and when it is off (or 0), it indicates "display disabled." In S604 or S609, the CPU 201 displays the target position on the display unit 205 if the target position display flag is on (or 1). In addition, in S605 or S608, the CPU 201 displays the cropping frame on the display unit 205 if the cropping frame display flag is on (or 1). This allows the user to specify whether to display the GUI components for cropping frame setting and target position setting.

[0076] For simplicity of explanation, the first embodiment has been described as having one crop area processed by the image processing unit 106 of the camera 100, but the number is not limited to this. If multiple crop settings are possible, the CPU 201 displays multiple crop frames on the display unit 205. This makes it possible to apply the present invention to cameras that have multiple crop settings and can generate multiple cropped images.

[0077] (Second embodiment) In the first embodiment described above, it was explained that a target position icon and a cropping frame are displayed in the GUI regardless of whether the cropped image is being output to the image output I / F 110. In the second embodiment, an embodiment will be described in which the GUI display is adjusted depending on whether the cropped image is being output to the image output I / F 110. In particular, an embodiment will be described in which a cropping frame is displayed only when the controller 200 is in the target position setting mode and the cropped image is being output to the image output I / F 110. Note that the overall configuration of the system, the configuration of each device, and the operation of the camera are the same as those in the first embodiment (FIGS. 1 to 3), and therefore description thereof will be omitted.

[0078] <Operation of the controller 200> 8 is a flowchart of the automatic tracking setting in the second embodiment. Note that S801, S803, S804, and S806 to S810 are the same as S601, S603, S604, and S606 to S610 in the first embodiment, and therefore the description thereof will be omitted.

[0079] In S802, the CPU 201 acquires camera information of the camera 100. Here, the camera information refers to information including a captured image, a target position, and a crop setting. However, the crop setting differs from the first embodiment in that it includes information indicating whether a cropped image is being output to the video output I / F 110, which is an output unit. For simplicity's sake, the following description will be given assuming that there is one video output I / F 110 and one crop setting; however, a configuration may be adopted in which there are multiple video output I / Fs 110 and multiple crop settings. If there are multiple I / Fs 110 and multiple crop settings, the configuration is such that output information indicating which crop setting is used to apply the cropped image to each video output I / F 110 is included.

[0080] In S805, the CPU 201 displays a crop frame on the GUI as necessary. Specifically, the CPU 201 reads the crop setting from the RAM 202 and determines, based on the read crop setting, whether a cropped image is being output to the video output I / F 110. If a cropped image is being output to the video output I / F 110, a crop frame corresponding to the output cropped image is superimposed on the captured image displayed on the display unit 205.

[0081] 9(a) to 9(d), a description will be given of the GUI displayed on the display unit 205 (only when a cropped video is output to the video output I / F 110) and user operations. Fig. 9 is a diagram for explaining the display of the GUI in the second embodiment. The GUI is a user interface that is displayed on the display unit 205, provides information to the user, and accepts operations from the user.

[0082] For ease of explanation, a case will be described in which there are two video output I / Fs 110 (a) and (b) and three crop settings. The crop setting to be assigned to each video output I / F 110 is managed by the output setting. FIG. 9(a) shows an example of a GUI in which GUI components (a target position icon 902 and three crop frames 903a to 903c) are superimposed on a video 900 in which three people are captured from the front. The target position icon 902 is a GUI component that indicates the target position of the tracking target (the face of subject 901) in the captured image 900.

[0083] In Fig. 9(a), the output setting is "no crop" for both video output I / F(a) and (b). However, in Fig. 9(a), a crop frame 903 is drawn regardless of the output of the video output I / F 110, so three crop frames 903a to 903c and a target position icon 902 are drawn on the GUI. As a result, the visibility of the target position icon 902 is reduced.

[0084] Therefore, in the second embodiment, as shown in FIGS. 9(b) to 9(d), a cropping frame 903 is drawn in accordance with the output of the video output I / F 110, thereby improving the visibility of the target position icon 902.

[0085] 9(b), like in FIG. 9(a), the output setting is "no crop" for both the video output I / F(a) and (b). In this case, none of the three cropping frames 903a to 903c are drawn, and only the target position icon 902 is drawn on the GUI.

[0086] In Figure 9(c), the output setting for video output I / F(a) is "crop 903a," and for (b) is "no crop." In this case, a crop frame 903a is drawn in addition to the target position icon 902. This reduces the complexity of setting the target position icon 902 relative to the crop frame 903a.

[0087] In Figure 9(d), the output setting for video output I / F (a) is "crop 903a," and for (b) is "crop 903b." In this case, crop frames 903a and 903b are drawn in addition to the target position icon 902. This reduces the complexity of setting the target position icon 902 relative to the crop frames 903a and 903b.

[0088] As described above, according to the second embodiment, the GUI display is adjusted depending on whether the cropped video is being output to the video output I / F 110. By displaying only the cropping frame for the cropped video being output to the video output I / F 110 on the GUI, it becomes easier to set the target position icon 902 for the cropping frame.

[0089] In the second embodiment, a GUI displaying GUI components for both crop frame setting and target position setting is displayed on the display unit 205. However, similar to the configuration described in the modified example of the first embodiment, each GUI component may be displayed individually.

[0090] The disclosure of this specification includes the following control device, control method, and program. (Item 1) A control device for controlling an imaging device, The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, The control device a first setting means for setting a target position in the captured image where the subject stays in the tracking function in the imaging device; a second setting means for setting a crop area in the captured image cut out by the crop function in the imaging device; a display control means for displaying, on a display unit, a graphical user interface (GUI) that accepts at least one of the setting of the target position and the setting of the crop region; Equipped with The display control means displays a first GUI component indicating the target position and a second GUI component indicating the crop region on the display unit. A control device characterized by: (Item 2) The display control means displays the first GUI component and the second GUI component in a superimposed manner on the captured image. 2. The control device according to item 1, (Item 3) an acquisition unit that acquires from the imaging device a captured image captured by the imaging device, a setting of the target position that is set in the imaging device, and a setting of the crop area that is set in the imaging device; The display control means displays the first GUI component based on the setting of the target position acquired by the acquisition means, and displays the second GUI component based on the setting of the crop area acquired by the acquisition means. 3. The control device according to item 1 or 2. (Item 4) The display control means, when in a first setting mode for accepting the setting of the target position, accepts movement and / or deformation of the first GUI component, and, when in a second setting mode for accepting the setting of the crop region, accepts movement and / or deformation of the second GUI component. 4. The control device according to item 3, (Item 5) the crop function can generate a plurality of cropped images by cutting out different partial areas from the captured image; the imaging device has an output unit that outputs one or more cropped images included in the plurality of cropped images; the acquisition means further acquires output information regarding the cropped image being output from the output unit; The display control means displays the second GUI component based only on the setting of the crop area corresponding to the cropped image indicated by the output information. 5. The control device according to item 3 or 4. (Item 6) further comprising a receiving means for receiving a first designation regarding whether or not the first GUI component is to be displayed on the GUI and a second designation regarding whether or not the second GUI component is to be displayed on the GUI; The display control means determines whether or not to display the first GUI component on the GUI based on the first designation, and determines whether or not to display the second GUI component on the GUI based on the second designation. 6. The control device according to any one of items 3 to 5, (Item 7) A control device for controlling an imaging device, The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, The control device a first setting means for selecting a subject to be tracked in the tracking function from among a plurality of subjects in the captured video and setting the selected subject in the imaging device; a second setting means for setting a crop area in the captured image cut out by the crop function in the imaging device; a display control means for displaying a graphical user interface (GUI) on a display unit that accepts at least one of the selection of the subject and the setting of the crop region; Equipped with The display control means displays a first GUI component indicating the selected subject and a second GUI component indicating the crop region on the display unit. A control device characterized by: (Item 8) A control method for a control device that controls an imaging device, comprising: The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, the control device is configured to be able to set, in the imaging device, a target position in the captured video where the subject stays in the tracking function and a crop area in the captured video that is cut out by the crop function; The control method includes: a display control step of displaying a graphical user interface (GUI) on a display unit that accepts at least one of the setting of the target position and the setting of the crop region; a setting step of setting the target position setting and the crop region setting received via the GUI in the imaging device; Including, In the display control step, a first GUI component indicating the target position and a second GUI component indicating the crop region are displayed on the display unit. A control method comprising: (Item 9) A control method for a control device that controls an imaging device, comprising: The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, the control device is configured to be able to set, in the imaging device, a selection of a subject to be tracked in the tracking function from a plurality of subjects in the captured video and a crop area to be cut out in the captured video by the crop function; The control method includes: a display control step of displaying a graphical user interface (GUI) on a display unit that accepts at least one of the selection of the subject and the setting of the crop region; a setting step of setting the selection of the subject and the setting of the crop region received via the GUI in the imaging device; Including, In the display control step, a first GUI component indicating the selected subject and a second GUI component indicating the crop region are displayed on the display unit. A control method comprising: (Item 10) 10. A program for causing a computer to execute the control method according to item 8 or 9.

[0091] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0092] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0093] 101 CPU; 102 RAM; 103 ROM; 104 Inference unit; 105 Network I / F; 106 Image processing unit; 107 Image sensor; 108 Drive I / F; 109 Drive unit; 110 Video output I / F; 111 Bus; 201 CPU; 202 RAM; 203 ROM; 204 Network I / F; 205 Display unit; 206 User input I / F; 207 Bus

Claims

1. A control device for controlling an imaging device, The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, The control device a first setting means for setting a target position in the captured image where the subject will stay in the tracking function in the imaging device; a second setting means for setting a crop area in the captured image cut out by the crop function in the imaging device; a display control means for displaying a graphical user interface (GUI) on a display unit, the GUI accepting at least one of the setting of the target position and the setting of the crop region; Equipped with The display control means displays a first GUI component indicating the target position and a second GUI component indicating the crop region on the display unit. A control device characterized by:

2. The display control means displays the first GUI component and the second GUI component in a superimposed manner on the captured image.

2. The control device according to claim 1.

3. an acquisition unit that acquires from the imaging device a captured image captured by the imaging device, a setting of the target position that is set in the imaging device, and a setting of the crop area that is set in the imaging device; The display control means displays the first GUI component based on the setting of the target position acquired by the acquisition means, and displays the second GUI component based on the setting of the crop area acquired by the acquisition means.

2. The control device according to claim 1.

4. The display control means, when in a first setting mode for accepting the setting of the target position, accepts movement and / or deformation of the first GUI component, and, when in a second setting mode for accepting the setting of the crop region, accepts movement and / or deformation of the second GUI component.

4. The control device according to claim 3.

5. the crop function can generate a plurality of cropped images by cutting out different partial areas from the captured image; the imaging device has an output unit that outputs one or more cropped images included in the plurality of cropped images; the acquisition means further acquires output information regarding the cropped image being output from the output unit; The display control means displays the second GUI component based only on the setting of a crop area corresponding to the cropped image indicated by the output information.

4. The control device according to claim 3.

6. a receiving unit configured to receive a first specification regarding whether or not the first GUI component is to be displayed on the GUI and a second specification regarding whether or not the second GUI component is to be displayed on the GUI; The display control means determines whether or not to display the first GUI component on the GUI based on the first designation, and determines whether or not to display the second GUI component on the GUI based on the second designation.

4. The control device according to claim 3.

7. A control device for controlling an imaging device, The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, The control device a first setting means for selecting a subject to be tracked in the tracking function from among a plurality of subjects in the captured video and setting the selected subject in the imaging device; a second setting means for setting a crop area in the captured image cut out by the crop function in the imaging device; a display control means for displaying a graphical user interface (GUI) on a display unit, the GUI accepting at least one of the selection of the subject and the setting of the crop region; Equipped with The display control means displays a first GUI component indicating the selected subject and a second GUI component indicating the crop region on the display unit. A control device characterized by:

8. A control method for a control device that controls an imaging device, comprising: The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, the control device is configured to be able to set, in the imaging device, a target position in the captured video where the subject stays in the tracking function and a crop area in the captured video that is cut out by the crop function; The control method includes: a display control step of displaying a graphical user interface (GUI) on a display unit that accepts at least one of the setting of the target position and the setting of the crop region; a setting step of setting the target position setting and the crop area setting received via the GUI in the imaging device; Including, In the display control step, a first GUI component indicating the target position and a second GUI component indicating the crop region are displayed on the display unit. A control method comprising:

9. A control method for a control device that controls an imaging device, comprising: The imaging device has a tracking function for tracking and photographing a subject, and a crop function for generating a cropped image by cutting out a partial area from the photographed image, the control device is configured to be able to set, in the imaging device, a selection of a subject to be tracked in the tracking function from a plurality of subjects in the captured video and a crop area to be cut out in the captured video by the crop function; The control method includes: a display control step of displaying a graphical user interface (GUI) on a display unit that accepts at least one of the selection of the subject and the setting of the crop region; a setting step of setting the selection of the subject and the setting of the crop region received via the GUI in the imaging device; Including, In the display control step, a first GUI component indicating the selected subject and a second GUI component indicating the crop region are displayed on the display unit. A control method comprising:

10. A program for causing a computer to execute the control method according to claim 8 or 9.

Images