Field of view control device, imaging device, imaging system, control method and program

Abstract

To achieve appropriate framing in accordance with a subject.SOLUTION: An angle-of-view control apparatus for determining settings for angle of view to be applied to an imaging apparatus includes: determination means which determines settings for angle of view to be applied to the imaging apparatus; output means which configures control information related to modification of the settings for angle of view, based on the settings for angle of view determined by the determination means, and output the information to the imaging apparatus; acquisition means which acquires an image captured by the imaging apparatus; and storage means which stores, regarding predetermined settings for angle of view, a template in which a distribution of images of predetermined objects in the angle of view is defined. The determination means is configured to determine the settings for angle of view to be applied to the imaging apparatus on the basis of a difference between the distribution of the predetermined objects in a first captured image captured by the acquisition means for the predetermined settings for angle of view and a distribution of the predetermined objects in the template stored in the storage means for the settings for angle of view.SELECTED DRAWING: Figure 14

Description

Technical Field

[0001] The present invention relates to an angle-of-view control device, an imaging device, an imaging system, a control method, and a program, and particularly to a technique for remotely adjusting the angle-of-view setting of an imaging device.

Background Art

[0002] There is an imaging technique for obtaining an imaging image with a desired angle of view by remotely controlling pan, tilt, and zoom (PTZ) for an imaging device arranged at a predetermined position. As a method for automating the angle-of-view setting of such an imaging device, Patent Document 1 discloses a method for adjusting the angle-of-view setting so as to include all the faces of persons detected in the imaging range and a point-of-interest area included in the background or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the technique described in Patent Document 1, when an unintended person is included in the imaging range, the angle-of-view setting is adjusted to include the person, and as a result, there is a possibility that a suitable imaging image cannot be obtained. For example, in imaging at a wedding ceremony venue, the main subjects to be framed are the bride and groom, but there may be attendants or guests included in the imaging range, so in the technique described in Patent Document 1, there is a possibility that an angle-of-view setting for realizing suitable imaging is not selected.

[0005] In contrast, by setting the imaging device to a fixed field of view without PTZ movement, it is possible to capture a predetermined area within the venue at all times. However, due to differences in the height, build, and standing position of the subjects, such as the bride and groom, the overall composition may not always be satisfactory. In such cases, the operator in charge of photography had to manually adjust the field of view setting of the imaging device each time while looking at the captured image, which was cumbersome.

[0006] This invention has been made in view of the above-mentioned problems, and aims to provide a field of view control device, imaging device, imaging system, control method, and program that realize suitable framing according to the subject. [Means for solving the problem]

[0007] To achieve the aforementioned objectives, the field of view control device of the present invention includes: a determination means for determining a field of view setting to be applied to an imaging device; an output means for configuring control information relating to changing the field of view setting based on the field of view setting determined by the determination means and outputting it to the imaging device; an acquisition means for acquiring an image captured by the imaging device; and a storage means for storing a template that defines the distribution of images of predetermined objects within the field of view for a predetermined field of view setting. A means for identifying the position of an image of a predetermined object included in the captured image, It has, The determination of the field of view setting to be applied to the imaging device based on the template by the determination means includes a first stage and a second stage following the first stage, The means of decision are, In the first stage, the field of view setting to be applied to the imaging device is determined so as to reach a predetermined field of view setting, and in the second stage, With a predetermined field of view setting Responding The first captured image obtained by the acquisition means In this case, an end subject appearing at one end in at least one of the horizontal and vertical directions is captured at one end of the field of view in that direction, and a predetermined object is captured. In the template To capture the designated corresponding position The method is characterized by determining the field of view setting to be applied to the imaging device. [Effects of the Invention]

[0008] With this configuration, the present invention makes it possible to achieve suitable framing according to the subject. [Brief explanation of the drawing]

[0009] [Figure 1] Figure illustrating the configuration of an imaging system according to embodiments and modified examples of the present invention. [Figure 2]This figure shows an example of the arrangement of the remote camera 110 according to an embodiment of the present invention. [Figure 3] A diagram illustrating the hardware configuration of a remote camera 110 according to embodiments and modified examples of the present invention. [Figure 4] A diagram illustrating the functional configuration of a remote camera 110 according to embodiments and modifications of the present invention. [Figure 5] A diagram illustrating the hardware configuration of a controller 120 according to embodiments and modified examples of the present invention. [Figure 6] A diagram illustrating the functional configuration of a controller 120 according to embodiments and modified examples of the present invention. [Figure 7] A diagram illustrating the hardware configuration of PC170 according to embodiments and modified examples of the present invention. [Figure 8] Figure illustrating the functional configuration of PC170 according to embodiments and modified examples of the present invention. [Figure 9] A diagram illustrating a GUI used for registering preset information according to an embodiment of the present invention. [Figure 10] A diagram illustrating a GUI for receiving the specification of a preset registration number according to an embodiment of the present invention. [Figure 11] A diagram illustrating the GUI used in defining a template according to an embodiment of the present invention. [Figure 12] A flowchart illustrating the registration process performed in PC170 according to an embodiment of the present invention. [Figure 13] A diagram illustrating one aspect of determining the second field of view setting according to an embodiment of the present invention. [Figure 14] This figure illustrates the field of view control process performed in a remote camera 110 according to an embodiment of the present invention. [Figure 15] This figure illustrates a GUI used for registering additional preset information according to a modified example of the present invention (Modification 2). [Modes for carrying out the invention]

[0010] [Embodiment] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] One embodiment described below is an example of an imaging system, which is a system including a remote camera having a function of controlling the change of the angle of view as an imaging device, and a controller for giving an instruction to change the angle of view of the remote camera, to which the present invention is applied. However, the present invention is applicable to any device capable of determining the angle-of-view setting to be applied to the imaging device.

[0012] 《Configuration of Imaging System》 FIG. 1 shows the device configuration of the imaging system according to the present embodiment. The imaging system of the present embodiment is composed of a plurality of remote cameras 110, a controller 120, a switcher 140, a video storage 150, a PC 170, and a PoE hub 180.

[0013] Each remote camera 110 is an imaging device that captures subject light through a lens 306 and outputs video information. The remote camera 110 includes a mechanism for changing the angle of view such as pan, tilt, and zoom, and performs an operation related to the change of the angle of view based on a control signal received from the controller 120 connected via an RS-422 serial communication cable 130. The video information output from the remote camera 110 is transmitted to the switcher 140 via an SDI cable 160. In the following description, for ease of understanding of the invention, the video information transmitted from the remote camera 110 to the switcher 140 is described as information composed of a series of intermittently acquired captured images, but in addition to this, it may include audio information.

[0014] The switcher 140 displays video information acquired from multiple remote cameras 110 and selects video information to record. The switcher 140 is connected to a display 142 that displays a list of multiple acquired video information and a display 144 that displays one video information selected by a selection operation or the like. In the imaging system of this embodiment, the switcher 140 is used to select video information to be stored (recorded) in the video storage 150, and outputs one selected video signal from the multiple received video information to the video storage 150 for recording. That is, the video information displayed on the display 144 is transmitted to the video storage 150 for recording.

[0015] The controller 120 receives operation inputs related to changing the field of view setting of each remote camera 110 and sends control signals related to the change in field of view setting to the remote camera 110. The controller 120 is equipped with an LCD 126 that displays the current status and setting information of the controller 120. The controller 120 is also equipped with a seesaw switch 127 for operating the zoom state of the controlled remote camera 110 and a joystick 128 for operating the pan and tilt state. The controller 120 is also equipped with a button 129 for receiving operation inputs related to changing to a preset field of view setting or calling up functions related to field of view settings.

[0016] The PC170 primarily receives operational inputs related to the settings of the controller 120 and the remote camera 110, and configures the information necessary to reflect those settings and transmits that information to each device.

[0017] The PC170, remote camera 110, controller 120, and switcher 140 are connected by a UTP cable 190 (or LAN cable) via a PoE hub 180. The remote camera 110 receives power from the PoE hub 180 via the UTP cable 190 and is powered by it.

[0018] For example, in an embodiment where the imaging system is used to video record a church wedding ceremony, the remote cameras 110 may be positioned on both sides of the altar (110a and b) and at the back of the church (ceremony hall) (110c), as shown in Figure 2. The operator can select the remote cameras 110a to c to be controlled using the controller 120 and change the field of view setting (field of view adjustment) of the corresponding remote camera 110 by operating the seesaw switch 127 or joystick 128. The video information obtained from imaging by each remote camera 110 is transmitted to the switcher 140 via the PoE hub 180 and displayed on the switcher 140's display 142. When the operator selects one video information (or one remote camera 110) using the switcher 140, that video information is displayed on the display 144 and recorded in the video storage 150.

[0019] <Hardware configuration of remote camera 110> Next, the hardware configuration of the remote camera 110 will be explained using Figure 3.

[0020] As shown in the figure, the remote camera 110 has a CPU 301 that performs overall control and various calculations for the remote camera 110. The CPU 301 controls the remote camera 110 by reading the OS and programs stored in the ROM 302, loading them into the RAM 303, and executing them. Here, the ROM 302 is non-volatile memory, and the RAM 303 is volatile random-access memory.

[0021] The photoelectric conversion sensor 307 is an image sensor that converts the light beam incident through the lens 306 into an electrical signal (analog image signal) and outputs it. The output analog image signal is then processed by the image processing device 308, which applies development and image processing to convert it into a digital image signal or video signal. In addition, the image processing device 308 also performs various compression processes to generate video information (video data) in a predetermined encoding format. Device I / O 310 is an interface for various devices connected to the remote camera 110, such as a gyroscope for vibration damping.

[0022] The network controller 305 controls the transmission and reception of information with external devices. The network controller 305 can control the transmission and reception of commands with external devices and transmit video and audio information. Compressed video in formats such as JPEG, H.264, HEVC, and VVC, generated by the image processing device 308 in the remote camera 110, can be transmitted to the network via UTP cable through the network controller 305. In addition, in the imaging system of this embodiment, uncompressed video can also be output from the remote camera 110 to the switcher 140 via SDI 312.

[0023] The remote camera 110 also includes a motor 309 for controlling various pan, tilt, and zoom states related to the angle of view setting, as well as the aperture state. When the motor 309 receives a control signal from the controller 120 via the RS-422 interface 304, it performs pan, tilt, and zoom operations based on the control signal.

[0024] These hardware components are connected via bus 311 and configured to communicate with each other.

[0025] <Functional Configuration of Remote Camera 110> The functional configuration of each of these remote cameras 110 will be further explained with reference to the block diagram in Figure 4.

[0026] The control unit 401 controls the operation of each block of the remote camera 110. The control unit 401 implements its control function by having the CPU 301 read the program stored in the ROM 302, load it into the RAM 303, and execute it.

[0027] The memory unit 402 stores preset information (preset information) of the field of view settings registered in the remote camera 110. The memory unit 402 may be composed of, for example, a ROM 302. As will be described in detail later, in one embodiment the preset information includes information on the pan, tilt, and zoom field of view settings of the remote camera 110. In one embodiment the preset information may be composed of, for example, numerical values ​​such as angles and magnifications of each adjustment item from a reference state, or numerical values ​​of encoders provided on the motor 309 for each adjustment item.

[0028] The imaging unit 403 captures an image of a subject within the imaging range and outputs the obtained image (or a group of images, video information). The imaging unit 403 is realized when the image processing device 308 develops the analog image signal output by the photoelectric conversion sensor 307.

[0029] The detection unit 404 detects the image of a predetermined object included in the imaging range based on the image output by the imaging unit 403. In the imaging system of this embodiment, the predetermined object to be detected by the detection unit 404 is assumed to be a human body, and the detection unit 404 is capable of detecting the position or region in the image where an image of a human body appears. Any known method may be used for the human body image detection algorithm. For example, a region in the image corresponding to a person's face may be identified by pattern matching, and a region of a specific shape may be detected as a human body image based on that region. The detection unit 404 is realized by the image processing device 308 analyzing the image.

[0030] The determination unit 405 determines the field of view setting that the remote camera 110 should adopt. As will be described in detail later, in the imaging system of this embodiment, if a template that defines the distribution of object images is defined for a field of view setting for which preset information has been set, the field of view setting is adjusted based on the template. For this reason, the determination unit 405 determines the field of view setting after adjustment when field of view adjustment based on the template is necessary. That is, the field of view setting performed by the determination unit 405 is not determined based on control signals from the controller 120 or the like, but is determined by the remote camera 110 based on the current field of view setting and the template. The determination unit 405 may be a functional configuration realized by the CPU 301 executing predetermined processing.

[0031] The field of view control unit 406 controls the operation related to changing the field of view setting of the remote camera 110. In this embodiment, the field of view setting of the remote camera 110 is changed by changing the pan, tilt, and zoom state of the remote camera 110. The field of view control unit 406 controls the change to the field of view setting based on the control signal received from an external device (controller 120 or PC 170), and controls the change to the field of view setting determined by the determination unit 405. More specifically, the field of view control unit 406 derives the drive amount of the drive unit 407, described later, based on the difference between the current field of view setting (pan value, tilt value, zoom value) and the field of view setting based on the received control signal or the field of view setting determined by the determination unit 405. The field of view control unit 406 then configures control information to perform a drive operation equal to the derived drive amount and outputs it to the drive unit 407. The field of view control unit 406 may be a functional configuration realized by the CPU 301 executing predetermined processing.

[0032] The drive unit 407 performs the driving related to the angle of view setting of the remote camera 110. The drive unit 407 is a motor 309 provided for changing pan, tilt, and zoom, respectively. Based on the control information received from the angle of view control unit 406, the drive unit 407 performs the driving to change at least one of the pan, tilt, and zoom states of the remote camera 110.

[0033] The communication unit 408 enables communication between the remote camera 110 and external devices. The communication unit 408 receives control signals and transmits information indicating the current field of view setting to the controller 120 via the RS-422 interface 304. The communication unit 408 also transmits uncompressed video to the switcher 140 via SDI 312 and compressed video to the PC 170 via the network controller 305. The communication unit 408 also receives preset information configured on the PC 170.

[0034] <Hardware configuration of Controller 120> Next, the hardware configuration of the controller 120 will be explained using Figure 5.

[0035] As shown in the figure, the controller 120 has a CPU 501 that performs overall control and various calculations of the controller 120. The CPU 501 controls the controller 120 by reading the OS and programs stored in the ROM 502, loading them into the RAM 503, and executing them. Here, the ROM 502 is a non-volatile memory, and the RAM 503 is a volatile random-access memory.

[0036] The RS-422 interface 504 is an interface for sending control signals (or control commands) related to changing the field of view setting to the remote camera 110, which is controlled by the controller 120. The network controller 505 controls the transmission and reception of information with external devices. The transmission and reception of control signals may also be performed via the network controller 505.

[0037] In addition, the controller 120 includes an LCD 126 for displaying information related to the controller 120's settings, a user interface (UI) for receiving various operation inputs, and an I / O 510 interface for enabling external information communication. The UI includes a dial 507 for receiving operation inputs related to focus control and aperture control of the remote camera 110. The UI also includes a seesaw switch 127 and a joystick 128 for receiving operation inputs related to changing the field of view setting of the remote camera 110, and a button 129 for receiving operation inputs such as recalling presets related to the field of view setting and switching the white balance.

[0038] These hardware components are connected via bus 511 and configured to communicate with each other.

[0039] <Functional Configuration of Controller 120> The functional configuration of this controller 120 will be further explained with reference to the block diagram in Figure 6.

[0040] The control unit 601 controls the operation of each block in the controller 120. The control unit 601 implements its control function by having the CPU 501 read the program stored in the ROM 502, load it into the RAM 503, and execute it.

[0041] The management unit 602 manages the status of preset information settings for each remote camera 110. In this embodiment, the preset information registered for each remote camera 110 is stored in the memory unit 402 of the remote camera 110, so the controller 120 does not need to hold information indicating its specific contents. As will be described in detail later, the preset information is configured to be identified by a preset registration number, so the controller 120 shares information on which preset registration number corresponds to the preset information that has been registered for each remote camera 110. The preset registration number may correspond to a number assigned to, for example, a button 129 of the controller 120, and the operator can change the field of view setting related to the preset information by selecting the desired remote camera 110 and inputting an operation to the button 129. Therefore, the management unit 602 manages the status of preset information registration for each remote camera 110 as information for determining whether or not the operation input to the button 129 is valid. The management unit 602 may have a functional configuration implemented by, for example, a ROM 502.

[0042] The status acquisition unit 603 acquires information on the current field of view settings (e.g., pan value, tilt value, and zoom value) for the remote camera 110 to be controlled via the communication unit 605 described later. The status acquisition unit 603 may be configured to function by the CPU 501 detecting, via the RS-422 interface 504, that it has received information indicating the current pan, tilt, and zoom status from each remote camera 110.

[0043] The operation input unit 604 detects operation inputs made to various user interfaces and sends control signals corresponding to the operation inputs to the control unit 601. The operation input unit 605 includes, for example, a dial 507, a seesaw switch 127, a joystick 128, and a button 129.

[0044] The communication unit 605 implements communication control functions between the controller 120 and external devices. The communication unit 605 transmits control signals related to changes in the field of view setting and receives information on the current field of view setting via the RS-422 interface 504 with the remote camera 110. The communication control unit 606 also transmits and receives various types of information with the PC 170 via the network controller 505.

[0045] <Hardware configuration of PC170> Next, the hardware configuration of the PC170 will be explained using Figure 7.

[0046] As shown in the figure, the PC170 has a CPU 701 that controls the entire PC170 and performs various calculations. The CPU 701 controls the PC170 by reading the OS and programs stored in the ROM 702, loading them into the RAM 703, and executing them. Here, the ROM 702 is non-volatile memory, and the RAM 703 is volatile random-access memory.

[0047] The external storage controller 704 connects to other removable storage devices such as HDDs and controls the reading and writing of data to them. The network controller 705 controls the transmission and reception of information with external devices. The GPU 706 is a display control device of the PC 170 and performs various controls related to the configuration and display of the display screen, including the drawing of graphics to be displayed on the display 707. The PC 170 also includes a keyboard 708 and a mouse 709 as a UI that accepts various operation inputs.

[0048] The PC170 also includes an I / O710, which is an expansion interface such as USB. By connecting a converter 712, which converts to RS-422, an interface for controlling the remote camera 110, to the I / O710, it is also possible to control the remote camera 110 from the PC170.

[0049] These hardware components are connected via bus 711 and configured to communicate with each other.

[0050] <PC170 Functional Configuration> The functional configuration of the PC170 will be further explained with reference to the block diagram in Figure 8.

[0051] The control unit 801 controls the operation of each block in the PC 170. The control unit 801 implements its control function by having the CPU 701 read the program stored in the ROM 702, load it into the RAM 703, and execute it.

[0052] The memory unit 802 stores preset information relating to the field of view settings of each remote camera 110, and, if any, templates that define the distribution of objects defined for the preset field of view settings. The memory unit 802 may consist of, for example, a ROM 702 or an HDD connected via an external storage controller 704. As will be described in detail later, the template may define the distribution of main subject images, such as the bride and groom, in the captured image taken by the remote camera 110 with the field of view settings determined by the preset information, that is, where these images are located within the field of view.

[0053] The image acquisition unit 803 acquires video information output by the remote cameras 110 that are registered as preset information, i.e., a series of captured images, via the communication unit 806 described later. The acquisition of captured images by the image acquisition unit 803 can be carried out in any manner. For example, the image acquisition unit 803 may receive captured images output by all remote cameras 110 as they are sequentially transmitted. Alternatively, the image acquisition unit 803 may sequentially acquire captured images output by one remote camera 110 selected by the switcher 140, for example. The captured images may be, for example, JPEG compressed video, or compressed video such as H.264, HEVC, or VVC. The function of the image acquisition unit 803 may be realized by the CPU 701 detecting, via the network controller 705, that it has received captured images (video information) from each remote camera 110.

[0054] The registration unit 804 performs various processes related to the registration of preset information for each remote camera 110. As will be described in detail later, the registration unit 804 configures preset information based on the operation input related to the field of view setting to be preset, and transmits the preset information to the target remote camera 110 via the communication unit 806 (described later) for storage. If necessary, the registration unit 804 also configures a template for the field of view setting to be preset, which defines the distribution of object images in the captured image obtained with that field of view setting, and transmits it to the remote camera 110. When the registration unit 804 has configured preset information, it also transmits information to the controller 120 via the communication unit 806 indicating that the preset information has been registered to the remote camera 110. The registration unit 804 may be configured to function by having the CPU 701 perform the corresponding processing.

[0055] The operation input unit 805 detects operation inputs made to various user interfaces and sends control signals corresponding to the operation inputs to the control unit 801. The operation input unit 805 is, for example, a keyboard 708 and a mouse 709.

[0056] The communication unit 806 implements communication control functions between the PC 170 and external devices. The communication unit 806 transmits and receives information between the remote camera 110 or controller 120 via the network controller 705. For example, the communication unit 806 receives video information from the remote camera 110 via the network controller 705. Also, for example, when preset information is configured by the registration unit 804, the communication unit 806 transmits information corresponding to the remote camera 110 or controller 120 via the network controller 705. The communication unit 806 also transmits control signals related to changing the field of view setting of the remote camera 110 via the I / O 710 and converter 712.

[0057] Preset and template functions related to field of view settings In the imaging system of this embodiment, a function is provided to remotely adjust the field of view setting of at least each remote camera 110 so that video information with the desired framing is acquired by each remote camera 110 for video recording performed using the system. The operator can adjust the field of view setting of the relevant remote camera 110 by, for example, operating various UIs provided on the controller 120.

[0058] On the other hand, in situations where the placement of the remote camera 110 and the framing suitable for video recording are predetermined, such as in the video recording of a wedding ceremony, it is cumbersome to require an operator to perform operations each time to set the field of view of the remote camera 110. For this reason, the imaging system of this embodiment is configured to allow fixed field of view settings to be registered in advance as preset information for each remote camera 110, as described above. More specifically, in the imaging system, a registrant such as an operator sets the pan, tilt, and zoom state of the remote camera 110 to the desired state, and then performs an operation on the PC 170 to register the field of view setting for that state, thereby configuring the preset information. The configured preset information is transmitted from the PC 170 to the remote camera 110, and is used to control the operation of the remote camera 110 when an instruction to change to the field of view setting related to the preset information is received. Furthermore, the fact that the preset information has been registered is shared with the controller 120, and the controller 120 can receive operation inputs for changing to the field of view setting related to the preset information, for example, by a predetermined button 129. Accordingly, upon detection of an operation input to the button 129, the controller 120 sends a control signal containing information that uniquely identifies the preset information to the corresponding remote camera 110. The operation of the remote camera 110 is then controlled based on this control signal, thereby achieving a change in the field of view.

[0059] Incidentally, even if the field of view setting determined by such preset information is applied to the remote camera 110 and imaging is performed, whether the distribution of the main subject image in the captured image is in a suitable manner may change depending on the appearance and position of the main subject. More specifically, in the case of video recording of a wedding ceremony as shown in Figure 2, the main subjects are the bride and groom, and if their standing positions on the altar differ from the positions assumed by the preset information registrant, the captured image may not be framed in a suitable manner. Also, if the height or build of at least one of the bride and groom differs from the assumption made when registering the preset information, the captured image may not be framed in a suitable manner, for example, the head of the main subject may be out of frame. In other words, whether or not an captured image with the framing assumed by the preset information registrant is obtained is not determined solely by the field of view setting of the remote camera 110, but also by the distribution of the subject image within the field of view. Therefore, in imaging systems equipped with preset information for field of view settings, after changing the field of view of the remote camera 110 based on the preset information, the operator had to manually adjust it while viewing the image captured by the remote camera 110.

[0060] The imaging system of this embodiment has a function that allows the operator to define a template that specifies the distribution of images of predetermined objects in the captured image obtained with the corresponding field of view setting, in order to reduce the operational burden on the operator. That is, it provides the registrant with a function that allows them to further define the distribution of images of predetermined objects that are preferable to appear in the captured image for the field of view setting to be used as preset information. In other words, if the registrant of the preset information wishes to further define a preferred framing for the corresponding field of view setting, a template corresponding to that framing is associated with and stored in the preset information. Alternatively, the preset information is configured in a manner that includes a template. In other words, if such preset information is selected to be used, it is configured to change the corresponding field of view setting and to define the distribution of predetermined objects that serve as the basis for adjustment for the captured image obtained with that field of view setting. On the other hand, if the registrant does not wish to define a framing, the preset information is not associated with a template and is configured to simply change the field of view setting of the remote camera 110.

[0061] <GUI for information registration> The graphical user interface (GUI) for registering preset information and defining templates may be, for example, the one shown in Figure 9. The GUI 900 shown in Figure 9 is displayed on the display 707 when a request for registration of preset information is made on the PC 170, and triggers the corresponding process by input on the GUI 900, for example, by the mouse 709.

[0062] As shown in the figure, the GUI 900 contains an image 901 of the remote camera 110 (hereinafter referred to as the target camera) that is to be registered, and the registrant can check what the framing will be on the PC 170. The image 901 is acquired by the image acquisition unit 803 via the communication unit 806.

[0063] The GUI 900 is also equipped with pan / tilt buttons 902 and a zoom slider 903, allowing the user to change the pan, tilt, and zoom states of the target camera by inputting commands to these buttons. At this time, the control unit 801 sends a control signal to the target camera via the communication unit 806 in response to the input command, causing the target camera to perform the corresponding operation related to changing the field of view setting. In other words, the user can determine the field of view setting to register as preset information for the target camera while checking the imaging pattern of the target camera using the captured images 901 sequentially sent from the target camera.

[0064] The GUI 900 includes a registration button 904, and in response to the input to this button, the field of view setting currently applied to the target camera is registered as preset information. In the imaging system of this embodiment, each preset information is registered in a way that allows identification by a preset registration number, and when an input is made to the registration button 904, a GUI that accepts the specification of a preset registration number, as shown in Figure 10, is further displayed. When a preset registration number is entered in the GUI, the registration unit 804 configures the preset information including the preset registration number. The registration unit 804 then transmits the configured preset information to the communication unit 806, which sends it to the target camera along with the registration request. At this time, the control unit 401 of the target camera, in response to the reception of the registration request by the communication unit 408, stores the preset information received along with the registration request in the storage unit 402, and completes the registration of the preset information. The registration unit 804 also transmits information to the controller 120 via the communication unit 806 indicating that the preset information has been registered for the preset registration number, and has the management unit 602 manage this information. The configured preset information is also stored in the memory unit 802.

[0065] Furthermore, if an operation is performed on the Cancel button 905 in GUI900, the function related to registering preset information will be terminated, and the display of GUI900 will also end.

[0066] The GUI900 also includes a template registration button 906 for accepting template registrations. By performing an operation on the template registration button 906, the user can define a template related to the field of view setting currently applied to the target camera and then register the preset information. In other words, if the preset information is registered by performing an operation on the template registration button 906 rather than the registration button 904, the preset information associated with the template will be registered on the remote camera 110.

[0067] When an operation is performed on the template registration button 906, the display on the display 707 switches from GUI 900 to GUI 1100 shown in Figure 11(a). Similar to GUI 900, GUI 1100 includes the captured image 1101 of the target camera. The registrant can define the distribution of images of predetermined objects to be used as adjustment criteria for the field of view related to the captured image 1101 by placing icons corresponding to objects within the captured image 1101.

[0068] The icons that can be placed are listed in the object list 1102, and the user can place the desired icon for the template by dragging and dropping it onto the captured image 1101 from the object list 1102. In this embodiment, the predetermined object is a human body, and the object list 1102 lists icons corresponding to different appearances of the human body. In the example shown in the figure, four types of icons are placed as appearances of the human body. These are different: a full-body front icon 1102a showing the whole body of a person facing forward, an upper-body front icon 1102b showing the upper body of a person facing forward, a full-body side icon 1102c showing the whole body of a person facing sideways, and a face icon 1102d showing the face of a person.

[0069] For example, if the user drags and drops the full-body side icon 1102c from the object list 1102 onto the captured image 1101, a new full-body side icon 1111 will be placed and displayed on the captured image 1101, as shown in Figure 11(b). The user can determine the initial placement position of the person's image to be used as a template through this operation. The full-body side icon 1111, once placed, can be moved further on the captured image 1101, allowing its position within the field of view of the corresponding subject image to be changed. Furthermore, by moving, for example, each vertex of the outer frame, the size, posture (tilt), and left / right / up / down of the full-body side icon 1111 within the field of view of the corresponding subject image can be changed.

[0070] The registrant may place icons corresponding to the desired number of subjects on the captured image 1101 (by dragging and dropping and other adjustments), and can define the distribution of human images (subject images) related to the field of view setting of the target camera on an icon basis. In other words, the GUI 1100 allows the registrant to define framing criteria, such as the selection of the type of human appearance and where and how large the human image should be placed within the field of view.

[0071] Each icon is placed and managed on a different layer above it according to its arrangement order, and is configured to allow for independent re-editing of its arrangement. By inputting an operation to the layer adjustment button 1103, the user can change the layer hierarchy and thus change the vertical relationship between icons on the captured image 1101, that is, the front-to-back relationship in the depth direction of the person corresponding to the icon.

[0072] Similar to GUI900, GUI1100 includes a registration button 1104 and a cancel button 1105. When an operation is performed on either of these buttons, preset information is registered, or the function related to preset registration is terminated. When an operation is performed on the registration button 1104, the input of a preset registration number is also accepted. Furthermore, the preset information registered when an operation is performed on the registration button 1104 is associated with a template defined by the distribution of icons placed on the captured image 1101. More specifically, when a preset registration number is input, the registration unit 804 configures preset information that includes the preset registration number and associates it with a template constructed based on the icons placed on the captured image 1101. The registration unit 804 then transmits the configured preset information to the communication unit 806, which sends it to the target camera along with the registration request. At this time, the control unit 401 of the target camera, in response to receiving the registration request from the communication unit 408, stores the preset information received along with the registration request in the storage unit 402, and completes the registration of the preset information. The registration unit 804 also transmits information to the controller 120 via the communication unit 806 indicating that the preset information has been registered for the preset registration number, and instructs the management unit 602 to manage this information. The configured preset information is also stored in the storage unit 802.

[0073] Thus, in the imaging system of this embodiment, the PC170 can configure and register preset information that defines the field of view settings to be applied to each remote camera 110, and, if necessary, a template that defines the distribution of the subject image in the captured image related to the field of view setting.

[0074] In this embodiment of the imaging system, each preset registration number corresponds to one of the multiple buttons 129 on the controller 120, and buttons 129 with registered preset information can provide the function of changing the field of view setting to that preset information. That is, by registering preset information related to an arbitrary preset registration number on each remote camera 110, the controller 120 can accept operation input for the button 129 associated with that number. In other words, by performing an operation input on the button 129 with the corresponding preset information registered, the operator can cause the selected remote camera 110 to perform an operation related to changing the field of view setting to that preset information.

[0075] At this time, the control unit 601, upon detection by the operation input unit 604 that an operation input has been made to the button 129, can output the preset registration number as a control signal to the selected remote camera 110 via the communication unit 605. The field of view control unit 406 of the selected remote camera 110 can read the corresponding preset information from the storage unit 402 based on the control signal and control the drive unit 407 to achieve the field of view setting indicated by the preset information.

[0076] <Registration Process> The following describes the registration process performed by the PC170 in this embodiment for registering such preset information, using the flowchart in Figure 12. The process corresponding to the flowchart can be realized by the CPU701 reading the corresponding processing program stored in, for example, the ROM702, loading it into the RAM703, and executing it. This registration process will be described as starting when, for example, an operation input related to registering preset information to a target camera is detected in an application related to the imaging system running on the PC170. Although a detailed explanation will be omitted below, it will be assumed that the processing related to the display of GUI900 or GUI1100 is also being executed in parallel during the execution of this registration process.

[0077] In S1201, the control unit 801 determines whether an operation input related to changing the field of view setting of the target camera has been made. That is, the control unit 801 determines whether an operation input has been made to either the pan / tilt button 902 or the zoom slider 903 of the GUI 900. If the control unit 801 determines that an operation input related to changing the field of view setting of the target camera has been made, it moves the process to S1202; otherwise, it moves the process to S1203.

[0078] In S1202, the control unit 801 transmits a control signal corresponding to the input operation to the target camera via the communication unit 806. As a result, the target camera executes drive control related to changing the field of view setting, and the field of view setting of the target camera is changed. At this time, the image acquisition unit 803 can acquire the captured image output by the target camera with the changed field of view setting.

[0079] In S1203, the control unit 801 determines whether or not an instruction has been given to register preset information with the field of view setting currently applied to the target camera. This determination may be based on whether or not an operation input has been made to the registration button 904. If the control unit 801 determines that an instruction has been given to register preset information, it moves the process to S1204; otherwise, it moves the process to S1206.

[0080] In S1204, the registration unit 804, under the control of the control unit 801, receives input related to the specification of a preset registration number. Once the registration unit 804 receives input related to the specification of a preset registration number, it proceeds to S1205.

[0081] In S1205, the registration unit 804, under the control of the control unit 801, registers preset information to the target camera and completes the registration process. Specifically, the registration unit 804 acquires information on the pan, tilt, and zoom values ​​currently applied to the target camera and constructs the preset information together with the preset registration number entered in S1204. The registration unit 804 then transmits the constructed preset information to the communication unit 806 and sends it to the target camera along with the registration request. The registration unit 804 also transmits information to the controller 120 via the communication unit 806 indicating that the preset information has been registered for the target camera's preset registration number. This makes the target camera and controller 120 ready for use (the field of view setting related to the preset information can be changed) with the preset information registered.

[0082] On the other hand, if the control unit 801 determines in S1203 that no instruction to register preset information has been given, it determines in S1206 whether or not an instruction to register preset information accompanied by a template definition has been given. This determination may be based on whether or not an operation input has been made to the template registration button 906. If the control unit 801 determines that an instruction to register preset information accompanied by a template definition has been given, it moves the process to S1207; otherwise, it returns the process to S1201.

[0083] In S1207, the control unit 801 performs various icon placement processes related to the definition of templates in the GUI 1100. The processes performed in this step are various information processing related to defining the distribution of subject images in the captured image in response to the detection of operation inputs such as dragging and dropping icons from the object list 1102, changing their position, changing their size, and changing their orientation.

[0084] In S1208, the control unit 801 determines whether an instruction to register preset information has been given, which involves defining a specified distribution of subject images as a template. This determination may be based on whether or not an operation input has been made to the registration button 1104. If the control unit 801 determines that an instruction to register preset information with a template definition has been given, it moves the process to S1209; otherwise, it returns the process to S1207.

[0085] In S1209, the registration unit 804, under the control of the control unit 801, receives input related to the specification of a preset registration number. Once the registration unit 804 receives input related to the specification of a preset registration number, it proceeds to S1210.

[0086] In S1210, the registration unit 804, under the control of the control unit 801, registers preset information to the target camera and completes the registration process. Specifically, the registration unit 804 first constructs a template based on the distribution of subject images (icons) in the captured image as defined in the GUI 1100. The template may be composed of information such as the type of subject image (whole body front view, whole body side view, etc.), the reference position, rotation angle, and size of the subject image in the captured image, and information defining the front-to-back relationship with other subject images, based on the state of each placed icon. Here, the reference position may indicate the coordinates in the captured image of any vertex of the outer frame provided to manage the placement of each icon, but in this embodiment, for convenience, it is assumed to be the centroid position of the subject image (region) shown by the icon. In short, the template may be configured to define the type of subject image, the position in the field of view, and the size in the field of view for each subject image included in the template.

[0087] The registration unit 804 also acquires information on the pan, tilt, and zoom values ​​currently applied to the target camera, and configures preset information with the preset registration number entered in S1209 and the configured template. The registration unit 804 then transmits the configured preset information to the communication unit 806 and sends it to the target camera along with the registration request. The registration unit 804 also transmits information to the controller 120 via the communication unit 806 indicating that the preset information has been registered for the target camera's preset registration number. This makes the target camera and controller 120 ready for use (the field of view setting related to the preset information can be changed) with the preset information registered. When the preset information is used, the target camera can refer to a template that defines the distribution of the subject image that serves as the basis for the field of view setting related to the preset information.

[0088] In this way, the imaging system of this embodiment can register preset information that defines the field of view setting to be used as a preset, and, if necessary, a template that defines the distribution of predetermined objects in that field of view setting, to each remote camera 110.

[0089] <Template-based field of view control> When the field of view setting of the remote camera 110 is changed by specifying preset information associated with a registered template (hereinafter referred to as additional preset information), the field of view control is performed differently from that of normal preset information.

[0090] When the field of view setting is changed by specifying normal preset information that is not associated with a template, the remote camera 110 is driven to achieve the field of view setting defined by that preset information. In other words, the image captured by the remote camera 110 after the field of view setting is changed captures a fixed imaging range, and as mentioned above, depending on the state of the subject, the subject image may not necessarily be framed in a way that is favorably distributed.

[0091] On the other hand, when the field of view setting is changed by specifying additional preset information, the remote camera 110 is driven to achieve that field of view setting. Subsequently, further drive control is performed to obtain an image with adjusted framing according to the state of the subject. That is, when the field of view setting is changed by specifying additional preset information, the change is performed in two stages: first to the field of view setting defined by the additional preset information (first field of view setting), and second to the field of view setting determined by the determination unit 405 according to the state of the subject (second field of view setting). For the image acquired when the first field of view setting is applied by the additional preset information, information on the distribution of the subject (main subject) image that was assumed at the time of registration can be obtained from the template associated with the additional preset information. Therefore, the determination unit 405 determines the second field of view setting for the second stage of change based on the difference between the distribution of the main subject image in the image acquired after the application of the first field of view setting and the distribution of the main subject image in the template.

[0092] The following describes the determination of the second field of view setting based on the template, and the details of field of view control related to changing to the second field of view setting. In the following description, it is assumed that the controller 120 has been configured to use additional preset information for the field of view setting of the target camera, and that the target camera has transitioned to the first field of view setting. Furthermore, to facilitate understanding of the invention, the template is described as information that defines the distribution of the image of the person, which is the main subject, within the field of view, for an area equal in size to the captured image.

[0093] First, the detection unit 404 performs human body detection processing on the captured image output by the imaging unit 403, thereby detecting the region in the captured image in which the image of a person appears. In this embodiment, when defining the template, the four types of icons shown in the object list 1102 of Figure 11 can be placed, so the detection unit 404 detects the full body front, upper body front, and full body side images related to the icons that represent the appearance of a human body. At this time, the imaging range related to the first field of view setting may include other people in addition to the bride and groom, who are the main subjects, but at the stage of human body detection processing in the detection unit 404, it does not specifically determine what kind of person the image is, and detects as many human bodies as possible in the captured image.

[0094] Furthermore, since the object list 1102 in Figure 11 is configured to allow the placement of icons corresponding to human faces, the detection unit 404 also detects human face regions in the captured image by performing face detection processing on the captured image.

[0095] Furthermore, the detection unit 404 associates information with each region that indicates which type of human body appearance (whole body front view, upper body front view, whole body side view, face) the region detected by the human body detection process and the face detection process represents.

[0096] When the detection unit 404 detects areas related to people appearing in these captured images, it further performs a matching process to identify areas corresponding to the main subject defined in the template associated with the additional preset information from among the detected areas (detection areas). In the matching process, first, one detection area to be judged is extracted for each of the main subject images defined in the template. The detection area to be judged for the main subject image defined in the template is the same type as the main subject image (whole body front, upper body front, whole body side, face, etc.) and is the area whose position in the captured image is closest to the reference position of the main subject image in the template. Next, in the matching process, the size of the detection area to be judged in the captured image is compared with the size of the main subject image in the field of view related to the template, and a determination is made as to whether the difference falls within a predetermined boundary value. The size difference may be evaluated, for example, by the area ratio of the size of the detection area to be judged to the size related to the template, or by the difference in the number of pixels. If the size difference falls within the boundary value as a result of the determination, the detection region to be determined is treated as the region corresponding to the image of the main subject. Conversely, if the size difference does not fall within the boundary value, the detection region to be determined is treated as the region not corresponding to the image of the main subject. In the imaging system of this embodiment, the matching process returns a matching success if a corresponding detection region has been identified for all the images of the main subject defined in the template, and returns a matching failure for any other result.

[0097] Upon successful matching, when the detection regions in the captured image corresponding to the images of each primary subject defined in the template (hereinafter referred to as corresponding subject regions) are identified, the determination unit 405 determines the second field of view setting based on the differences in these distributions. The determination of the second field of view setting is performed in a manner that reduces the difference between the distribution of primary subject images in the template and the distribution of corresponding subject regions in the captured image.

[0098] In one embodiment, for example as shown in Figure 13, a second field of view setting is determined so that, in the horizontal direction of the captured image, subjects captured within a range determined based on a corresponding subject area for a specific main subject are maintained within the field of view, and the corresponding subject area is moved to the position of the image of the main subject. Figure 13(a) shows the distribution of main subject images defined in template 1300 associated with the additional preset information of the target camera. Template 1300 defines the distribution of two types of main subject images (groom image 1301 and bride image 1302), and includes information on the centroid position of each. On the other hand, Figure 13(b) shows the captured image 1310 obtained when the first field of view setting related to the additional preset information is applied to the target camera. As shown, the captured image 1310 has a distribution of corresponding subject areas (groom area 1311) for the groom image 1301 and corresponding subject areas (bride area 1312) for the bride image 1302, which are identified by the matching process.

[0099] In this case, the overall impression of the framing related to the template is mainly formed by the information of the wider region (the captured subject) when the region is divided based on the main subject image. In the example of template 1300, the groom image 1301 and the bride image 1302 are distributed to the right of the center of the field of view in the horizontal direction. Therefore, when the region within the field of view is divided based on the centroid position 1303 of the rightmost bride image 1302, the impression of the framing is easily formed by the subject captured in the space 1305 from the centroid position 1303 to one end of the left side of the field of view (including the end 1304). Accordingly, the second field of view setting is determined so that the subject captured in the space 1315, which is determined by the centroid position 1313 of the bride region 1312 in the captured image and the end 1314, which is the left end of the captured image, is maintained in the captured image obtained after changing the field of view setting.

[0100] For example, in template 1300, if the centroid position 1303 of bride image 1302 is (x1, y1), then the coordinates of the leftmost end 1304 at the same vertical coordinates will be (0, y1). Now, suppose that in captured image 1310, the centroid position 1313 of bride region 1312 is (x1', y1'), and the coordinates of the leftmost end 1314 at the same vertical coordinates are (0, y1'). In this case, due to the difference between the coordinates (x1, y1) of the centroid position 1303 of bride image 1302 and the coordinates (x1', y1') of the centroid position 1313 of bride region 1312, the captured image may appear different from the framing that was assumed when the additional preset information was registered.

[0101] Therefore, as shown in Figure 13(c), the determination unit 405 determines the second field of view setting such that, within the modified field of view, the subject at the end 1314 is captured at one end of the field of view in the horizontal direction, and the x-coordinate of the centroid position 1313 of the bride region 1312 becomes x1. That is, the second field of view setting is determined so that the x-coordinate of the bride region 1312 is the same as the x-coordinate of the bride image 1302 defined in the template 1300. In the example of Figure 13(b), the second field of view setting is determined so that the center of the field of view is moved horizontally to the right (panned) and zoomed out to a position where the subject at the end 1314 is captured at the left end of the field of view. Based on the determined second field of view setting, the field of view control unit 406 controls the field of view, so that the captured image output by the imaging unit 403 is in the form shown in Figure 13(c). In other words, the second field of view setting is determined so that the position of the rightmost corresponding subject area in the horizontal direction in the captured image (center of gravity) and the position of the subject at the left edge of the field of view are aligned with the reference position of the rightmost main subject image in the horizontal direction in the template and the position of the left edge of the field of view.

[0102] In this embodiment, as one aspect, a method for determining the second field of view setting based on the distribution of the main subject image in the horizontal direction, as shown in Figure 13, has been described. However, the implementation of the present invention is not limited to this. The second field of view setting may also be determined by similarly aligning the distribution of the main subject image in the vertical direction to the template, instead of, or in addition to, the horizontal direction. That is, instead of, or in addition to, the method of aligning the rightmost corresponding subject area and the leftmost subject in the horizontal direction to the position defined in the template, the second field of view setting may be determined by aligning the leftmost corresponding subject area and the rightmost subject in the horizontal direction to the template. Furthermore, methods of aligning the uppermost corresponding subject area and the lowermost subject in the vertical direction to the template, or aligning the lowermost corresponding subject area and the uppermost subject in the vertical direction to the template, may be used to determine the second field of view setting.

[0103] Furthermore, when determining the second field of view setting based on the center of gravity of the corresponding subject area and the end of the field of view that is far from the corresponding subject area, the area of ​​the main subject's head may not be included in the field of view. For this reason, the determination unit 405 may further consider the distance between the position of the face area and the end of the field of view to determine a second field of view setting that results in suitable framing.

[0104] Furthermore, if the main subject is located close to or far from the target camera, the size of the corresponding subject area may be larger or smaller than the size of the main subject image defined in the template. For this reason, the determination unit 405 may further change the zoom to match the size of the corresponding subject area to the size of the main subject image defined in the template and determine the second field of view setting.

[0105] Once the determination unit 405 determines the second field of view setting, the field of view control unit 406 derives the drive amounts for pan, tilt, and zoom values ​​to be driven by the drive unit 407 based on the difference between the determined second field of view setting and the first field of view setting. The field of view control unit 406 then configures control information to cause the drive unit 407 to perform the drive operation corresponding to the derived drive amounts, outputs it to the drive unit 407, and drives the camera. This allows the target camera to be controlled to a state where the second field of view setting is applied.

[0106] <Field of View Control Processing> The following describes the angle-of-view control process performed by the remote camera 110 of this embodiment based on the operation input related to the change in the angle-of-view setting made in the controller 120, using the flowchart in Figure 14 to explain the specific process. The process corresponding to the flowchart can be realized by the CPU 301 reading the corresponding processing program stored in the ROM 302, for example, loading it into the RAM 303 and executing it. This angle-of-view control process will be described as starting, for example, when the communication unit 408 receives a control signal related to the change in the angle-of-view setting from the controller 120.

[0107] In S1401, the control unit 401 determines whether the received control signal (received control signal) contains information that uniquely identifies the preset information (preset registration number). If the control unit 401 determines that the received control signal contains information that uniquely identifies the preset information, it moves the process to S1402; otherwise, it moves the process to S1412.

[0108] In S1402, the field of view control unit 406, under the control of the control unit 401, starts driving the drive unit 407 to achieve the field of view setting specified in the preset information (specified preset information) identified by the received control signal. Specifically, the field of view control unit 406 derives the drive amount for the drive unit 407 based on the difference between the current field of view setting and the field of view setting specified in the specified preset information. The field of view control unit 406 then constructs control information corresponding to the derived drive amount and outputs it to the drive unit 407, thereby initiating the operation to change to the field of view setting related to the specified preset information.

[0109] In S1403, the field of view control unit 406, under the control of the control unit 401, determines whether the operation related to changing the field of view setting as defined in the specified preset information has been completed. If the field of view control unit 406 determines that the operation related to changing the field of view setting has been completed, it moves the process to S1404; otherwise, it repeats the process in this step.

[0110] In S1404, the control unit 401 determines whether the specified preset information is preset information associated with a template (additional preset information). Specifically, the control unit 401 refers to the specified preset information stored in the storage unit 402 and determines whether a template is associated with the specified preset information. If the control unit 401 determines that the specified preset information is additional preset information, it moves the process to S1405; if it determines that it is not additional preset information, it completes the field of view control process.

[0111] In S1405, the detection unit 404, under the control of the control unit 401, performs human body detection processing on the captured image output by the imaging unit 403 and stores the detection result.

[0112] In S1406, the detection unit 404 performs face detection processing on the captured image under the control of the control unit 401 and stores the detection result.

[0113] In S1407, the detection unit 404, under the control of the control unit 401, performs template-based matching processing on the detection area which is the detection result of the human body detection processing in S1405 and the face detection processing in S1406.

[0114] In S1408, the control unit 401 determines whether the matching process was successful or not. If the matching process was successful, the control unit 401 moves the process to S1409; otherwise, it completes the registration control process.

[0115] In S1409, the determination unit 405, under the control of the control unit 401, determines the second field of view setting based on the difference between the distribution of the corresponding subject area obtained as a result of the matching process and the distribution of the main subject image defined in the template.

[0116] In S1410, the field of view control unit 406, under the control of the control unit 401, starts driving the drive unit 407 to achieve the second field of view setting determined in S1409. Specifically, the field of view control unit 406 derives the drive amount for the drive unit 407 based on the difference between the second field of view setting and the current field of view setting (first field of view setting). The field of view control unit 406 then constructs control information corresponding to the derived drive amount and outputs it to the drive unit 407, thereby initiating the operation to change to the field of view setting related to the specified preset information.

[0117] In S1411, the field of view control unit 406, under the control of the control unit 401, determines whether the operation related to changing to the second field of view setting has been completed. If the field of view control unit 406 determines that the operation related to changing to the second field of view setting has been completed, it completes this field of view control process; otherwise, it repeats the process in this step.

[0118] On the other hand, if it is determined in S1401 that the received control signal does not contain information that uniquely identifies the preset information, the field of view control unit 406, under the control of the control unit 401, performs drive control of the drive unit 407 based on the received control signal in S1412. Once this drive control is completed, the field of view control unit 406 completes this field of view control process.

[0119] As described above, according to the imaging system of this embodiment, when registering a field of view setting to be applied to the imaging device as a preset, a template can be defined that specifies the distribution of predetermined objects within the field of view that serves as the basis for framing. Furthermore, when a request to change the field of view setting that specifies preset information associated with the template is received, the field of view setting is changed to one that approximates the template based on the captured image obtained after the change to the field of view setting related to the preset information. As a result, the imaging system of this embodiment can achieve suitable framing according to the subject.

[0120] This imaging system can achieve stable framing in video recording, particularly in cases where a suitable framing is predetermined for a specific imaging range, such as in the recording of a wedding ceremony, but the state of the subject within the field of view may change each time it is used. Furthermore, with this imaging system, the field of view setting of each imaging device is adjusted according to the state of the subject within the field of view, and the captured image is output accordingly. Therefore, the operator can easily select the video information to be recorded using the switcher 140 without having to make individual adjustments.

[0121] [Example 1] In the registration process of the embodiment described above, it was explained that the system accepts a template definition for the field of view setting applied to the remote camera 110 by the registrant making operational inputs to the pan / tilt button 902 and zoom slider 903 in the GUI 900. However, the implementation of the present invention is not limited to this, and the system may be configured to accept additional template definitions for field of view settings related to already registered preset information.

[0122] [Differentiation 2] In the embodiments described above, as shown in Figure 11, an image captured when no predetermined object is present in the imaging range is received, and a template is defined for the image captured. However, the present invention is not limited to this. The template definition does not need to be performed in real time on the image captured from the target camera; for example, it may be configured to be performed by referring to video information previously captured by the target camera and recorded in the video storage 150 or the like.

[0123] In particular, for applications where the basic ceremony progress and the angle of view settings and framing scenarios to be used when recording it are fixed, such as video recording of wedding ceremonies, it is efficient to be able to define templates based on video information previously captured by the target camera. That is, it is preferable for the registrant to be able to define templates from the distribution of the main subject image in the frames of already recorded video information, rather than manually changing the angle of view setting of the target camera and defining templates by placing icons for each angle of view setting. In other words, this modified form is convenient because the registrant can define templates based on the subject image in the frame, using video information recorded during transitions between changes in the optimal angle of view setting, without having to manually adjust the target camera or imagine the distribution of the main subject image.

[0124] Trace memory function In this modified example, the imaging system is provided with a trace memory function that allows recording and reading of field-of-view history information, which shows the history of operations related to changes in the field-of-view setting of each remote camera 110. With this function, each time the field-of-view setting of a remote camera 110 is changed, a history of the field-of-view setting and the time (or frame) of the change is added in chronological order and recorded as field-of-view history information. In other words, this function does not add new history during periods when there are no changes in the field-of-view setting, i.e., when the same field-of-view setting is maintained. Therefore, by referring to the continuous history of the field-of-view history information, it is possible to identify the period during which imaging was performed with the same field-of-view setting. This field-of-view history information may be recorded and managed by any device, but in this modified example, it is recorded in the video storage 150.

[0125] Furthermore, the video storage 150 also records video information captured by each remote camera 110. That is, in this modified example, video information related to each remote camera 110 and field of view history information, which is the trace memory at the time of recording of the video information, can be obtained from the video storage 150. According to this information, for video information related to one remote camera 110, the field of view setting information applied to the remote camera 110 in each frame of the video information can be referenced from the field of view history information. In other words, by referencing the information related to one history contained in the field of view history information related to one remote camera 110, it is possible to identify the frame interval in which the field of view setting of that history was set and captured by the remote camera 110.

[0126] GUI for information registration The following describes an example of a method for registering additional preset information related to the imaging system of this modified example, using a GUI different from the embodiment described above. The GUI for defining additional preset information used in the imaging system of this modified example may be, for example, the one shown in Figure 15(a). The GUI 1500 shown in Figure 15(a) is displayed on the display 707 when a request for registration of additional preset information based on video information is made in the PC 170, and the corresponding process is triggered by input on the GUI 1500, for example, by the mouse 709.

[0127] Prior to the display on GUI1500, the field of view history information for the target camera for which registration of additional preset information is desired is read. In the following example, for the sake of simplicity, it is assumed that the video storage 150 stores one video information file output by the target camera with different field of view settings applied, and that the field of view history information for the target camera is associated with this video information.

[0128] As shown in the diagram, GUI 1500 includes video frames 1501 contained in the video information related to the target camera, allowing the user to check the framing of the frame. Which frame of the video information is displayed in video frame 1501 is controlled by which of the field-of-view setting histories included in the field-of-view history information is selected. GUI 1500 is provided with a next history button 1503 and a previous history button 1504 for switching the selected history for the field-of-view history information related to the target camera, allowing the user to specify the history corresponding to the desired field-of-view setting by inputting an operation to these buttons. More specifically, when an operation is made to the next history button 1503, the history following the currently selected history from the field-of-view setting histories included in the field-of-view history information is selected. Conversely, when an operation is made to the previous history button 1504, the history preceding the currently selected history from the field-of-view setting histories included in the field-of-view history information is selected. This allows the user to sequentially refer to the contents of the frames when the field-of-view setting was changed in the pre-recorded video information.

[0129] When the GUI 1500 related to the target camera is displayed, the first history entry (the oldest field of view setting) from the field of view history information is selected, and the video frame 1501 first displays the frame captured at the same time as that history in the video information. In other words, the first history entry from the field of view history information is selected as the initial value. If the history indicates a field of view setting across multiple frames, that is, if the field of view setting was maintained across multiple frames, then multiple frames in the video information are assigned to that history, and any of these can be displayed in video frame 1501. More specifically, if one history entry is selected and there are different frames captured with the same field of view setting, the user can change the frame displayed in video frame 1501 by inputting an operation to the slider 1502. The slider 1502 is a GUI that presents a movable operation component when there are multiple frames corresponding to a single history entry, and its operation status identifies one frame within the frame interval captured with the same field of view setting.

[0130] For the captured image displayed in video frame 1501, a predetermined object is detected, similar to the detection performed during the field of view control process. In the example in Figure 15(a), for video recording of a wedding ceremony, as described above, the predetermined object types to be detected are the entire front of the human body, the upper front, the entire side, and the face. Once the frame displayed in video frame 1501 is determined and a human body or the like is detected in the captured image of that frame, a frame is added to the corresponding (detection target) subject image and it is displayed as a detection area, as shown in the figure. In the example in the figure, the detection area 1506a related to the groom, the detection area 1506b related to the bride, and the detection areas 1506c and d related to the attendees are detected and displayed with dashed frames.

[0131] In GUI1500, the user can register additional preset information by selecting a detection area from among the detection areas 1506a to d shown in the displayed video frame 1501 that corresponds to the subject they wish to define as a template. For example, if the user wishes to define the distribution of the bride and groom's subject images as a template for the field of view setting related to the displayed video frame 1501, the user should select detection areas 1506a and b as the main subject images. The selected detection area 1506 is clearly indicated by a thick border (selected area 1511), as shown in Figure 15(b), and is then registered in response to an operation input such as the next history button 1503, previous history button 1504, or end button 1505. More specifically, the registration unit 804 obtains the field of view setting related to the selected history from the field of view history information to construct preset information, and further constructs and associates a template based on the position and size of the selected area 1511 on the video frame 1501 and the type of image, and registers it. Here, the type of image that appears in the selected region 1511 is determined by the detection method used to detect the selected region 1511 (whole body front view, upper body front view, whole body side view, face). This provides a function that allows the registrant to select a desired subject image from the subject images included in the video information frame and easily register additional preset information based on the corresponding field of view history information.

[0132] Furthermore, the definition of a template based on video information is not limited to directly reflecting the detection results related to the frame. For example, as shown in the detection method list 1512 in Figure 15(b), the algorithm of the detection method used to detect the selected region 1511 may be displayed as an icon, and the template may be configured to allow changing to the desired detection method when defining it. In this case, the change in detection method may be performed by selecting another icon placed in the detection method list 1512. Also, for example, when a detection region is determined for a displayed video frame 1501, instead of using a frame to indicate the region, the corresponding icon may be superimposed and made editable, allowing adjustment of its position and size. In addition, once the selection of the selected region 1511 is completed for all of the time-series history shown in the field of view history information, it may be possible to register a set of additional time-series preset information.

[0133] [Difference 3] In the above-described embodiment, the field-of-view control process first performs an operation to change to the first field-of-view setting when the specified preset information is additional preset information, and then performs a process to determine the second field-of-view setting based on the captured image obtained after the completion of that operation. With this method, it is possible to compare the captured image obtained with the same field-of-view setting as when the preset information was registered with the distribution of the main subject image defined in the template, thereby improving the accuracy of the matching process.

[0134] On the other hand, for example, when the distance between the remote camera 110 and the main subject is large, even if the pan, tilt, and zoom states are not exactly the same as the first field of view setting, if they are close, the distribution of the main subject image appearing in the captured image will show a similar trend. That is, in the field of view control process, when the field of view setting of the remote camera 110 approaches the first field of view setting, the distribution of the main subject image in the captured image output from the imaging unit 403 is similar to the distribution of the main subject image in the captured image output when the first field of view setting is achieved. For this reason, the various processes performed by the detection unit 404 and the determination unit 405 in determining the second field of view setting can be performed with the same accuracy as in the above-described embodiment even if they are executed without going through the transition to the first field of view setting.

[0135] Therefore, the processing in S1405 of the field of view control processing may not be conditional on the completion of the operation related to changing to the first field of view setting in S1403, but rather may be started on the condition that, for example, the current field of view setting has come close to the first field of view setting. That is, the captured image used in the series of processes related to determining the second field of view setting may be the captured image output by the imaging unit 403 after the start of the operation related to changing to the first field of view setting and after it has been detected that the image has come close to the first field of view setting. In other words, the series of processes related to determining the second field of view setting is not executed on the captured image obtained after the start of the operation related to changing to the first field of view setting and before the image has come close to the first field of view setting, but is controlled to be executed on the captured image obtained after the image has come close to the first field of view setting. Here, whether or not the current field of view setting is close to the first field of view setting may be determined, for example, by whether or not the angle between the imaging direction defined by the current field of view setting and the imaging direction defined by the first field of view setting is smaller than a threshold.

[0136] By doing so, the series of processes related to determining the second field of view setting can be started without waiting for the completion of the operation related to changing to the first field of view setting, thus shortening the time required to execute the field of view control process compared to the embodiment described above.

[0137] Furthermore, the captured image obtained during the period in which the field of view setting is changed may exhibit image blur depending on the operating speed, potentially reducing the accuracy of deriving the second field of view setting, which provides a suitable framing. For this reason, the operation related to changing to the first field of view setting may be controlled to change at a normal operating speed until it approaches the first field of view setting, and then to decelerate thereafter. By doing so, the clarity of the captured image used in the series of processes related to determining the second field of view setting can be improved, thereby increasing the accuracy of deriving the second field of view setting.

[0138] [Differentiation Example 4] In the embodiments described above, preset information and templates are stored in each remote camera 110, and when a request to change the field of view setting that specifies the preset information is received, the determination unit 405 determines the field of view setting based on the template. That is, in the embodiments described above, an embodiment was described in which the remote camera 110, which is an imaging device according to the present invention, has a functional configuration of a field of view control device that performs change control to the field of view setting corresponding to the associated template when a change to a predetermined field of view setting is received.

[0139] However, the implementation of the present invention is not limited to this, and the functional configuration of the field of view control device (particularly the storage unit 402, detection unit 404, and determination unit 405) may be provided in other devices such as a controller 120 or a PC 170, rather than the remote camera 110. For example, in an embodiment in which the PC 170 is equipped with the functional configuration, the PC 170 also receives information indicating that an operation input related to changing the field of view setting, specifying additional preset information, has been made in the controller 120. Then, in the PC 170, after various detection and matching processes on the captured image after the application of the first field of view setting, the second field of view setting is determined. Then, control information for changing to the second field of view setting is transmitted from the PC 170 to the remote camera 110, and drive control is executed in the remote camera 110.

[0140] [Difference 5] In the embodiment described above, the predetermined object whose distribution is defined in the template is a person, and in the matching process, the region that is closest in position and whose size difference falls within the boundary value is identified as the corresponding subject region from among the detection regions where the type of human body appearance is the same. However, the implementation of the present invention is not limited to this, and for example, prior to the start of video recording using additional preset information, the face patterns of the main subjects (bride and groom) may be registered, and the corresponding subject region may be identified after recognizing the main subjects.

[0141] [Modification 6] In the embodiments described above, the predetermined object whose distribution is defined in the template was described as a person, but the predetermined object that is the target of the preset information registration and detection processing may be other types of subjects such as automobiles or animals.

[0142] [Difference 7] In the embodiments described above, the field of view control process is executed in response to a control signal related to a change in the field of view setting being received from the controller 120. However, the implementation of the present invention is not limited to this. In particular, some processing that determines the second field of view setting based on the difference from the distribution of the main subject image defined in the template may be performed, for example, by detecting the movement of the main subject during imaging.

[0143] Furthermore, since such dynamic changes to the field of view setting may alter the framing desired by the operator, it may be controlled so that these changes are not made during the period when the remote camera 110 is selected for video recording. Whether or not the remote camera 110 is selected for video recording may be determined by referring to the tally state of the remote camera 110, which changes according to a control signal from the switcher 140. For example, if the camera is in the tally state, the field of view setting may not be changed even if there is a difference from the distribution of the main subject image defined in the template, and the change may only be made when the camera is not in the tally state. Alternatively, the presence or absence of such field of view control according to the tally state may be configured to be selectable as an option.

[0144] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0145] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0146] 401: Control unit, 402: Memory unit, 403: Imaging unit, 404: Detection unit, 405: Determination unit, 406: Field of view control unit, 407: Drive unit

Claims

1. A determination means for determining the field of view setting to be applied to the imaging device, Based on the field of view setting determined by the determination means, an output means configures control information related to changing the field of view setting and outputs it to the imaging device, Acquisition means for acquiring the captured image of the aforementioned imaging device, A storage means for storing a template that defines the distribution of images of a predetermined object within a predetermined field of view, A means for identifying the position of the image of a predetermined object included in the captured image, It has, The determination of the field of view setting to be applied to the imaging device based on the template by the determination means includes a first step and a second step following the first step. The aforementioned determination means is In the first step, the angle of view setting to be applied to the imaging device is determined so as to reach the predetermined angle of view setting. In the second step, a field of view setting is determined to be applied to the imaging device such that, in the first image captured by the acquisition means in accordance with the predetermined field of view setting, the edge subject appearing at at least one end in either the horizontal or vertical direction in the first image captured image is captured at the end of the field of view in that direction, and the predetermined object is captured at the corresponding position defined in the template. An angle of view control device characterized by the following:

2. The end subject is a subject that appears at the end of the first image capture image in the direction that is further away from the centroid of the image of the predetermined object in the direction of the template. The angle of view control device according to claim 1.

3. The aforementioned specifying means is, In the second step, the image of an object of the same type as the predetermined object included in the captured image acquired by the acquisition means is detected. From among the images of the same type of object detected, identify the image of the predetermined object whose distribution is defined in the template. An angle of view control device according to claim 1 or 2, characterized in that it is a feature of the present invention.

4. The angle of view control device according to any one of claims 1 to 3, characterized in that the template specifies information for each of the predetermined objects, such as the type of object, its position in the angle of view, and its size in the angle of view.

5. The aforementioned predetermined object includes the human body, The aforementioned predetermined object types include the front of the human body, the side of the human body, and predetermined parts of the human body. The angle of view control device according to claim 4.

6. The angle of view control device according to claim 5, characterized in that the predetermined object type includes the face of a registered person.

7. The angle of view control device according to any one of claims 4 to 6, characterized in that the predetermined object includes a specific animal.

8. The angle of view control device according to any one of claims 4 to 7, characterized in that the template defines the distribution of multiple objects as the distribution of predetermined objects.

9. The system further includes an instruction means for receiving instructions to change to the predetermined angle of view setting, When the decision means receives an instruction from the instruction means to change to the predetermined angle of view setting, it makes a decision on the angle of view setting related to the first step. The first captured image is an image acquired after the field of view setting of the imaging device has come close to the predetermined field of view setting. The angle of view control device according to any one of claims 1 to 8.

10. The angle of view control device according to claim 9, characterized in that the determination means does not determine the angle of view setting in the second step until the angle of view setting of the imaging device approaches the vicinity of the predetermined angle of view setting.

11. The output means configures the control information to slow down the operation related to changing the field of view setting when the field of view setting of the imaging device approaches the predetermined field of view setting. The first captured image is an image acquired after the field of view setting of the imaging device has approached the predetermined field of view setting and the operation related to changing the field of view setting has slowed down. The angle of view control device according to claim 9 or 10, characterized in that it is a field of view control device.

12. The angle of view control device according to claim 11, characterized in that the determination means does not determine the angle of view setting in the second stage until the operation related to changing the angle of view setting of the imaging device is slowed down.

13. The angle of view control device according to any one of claims 1 to 12, further comprising a generation means for generating the template based on an operation input relating to specifying the distribution of images of a predetermined object for setting the angle of view of the imaging device, and storing it in the storage means.

14. The angle of view control device according to any one of claims 1 to 13, characterized in that the angle of view setting of the imaging device determines at least one of the pan, tilt, and zoom states of the imaging device.

15. The system further includes a determination means for determining the tally state of the imaging device, The output means outputs the control information to change the field of view setting when the imaging device is not in a tally state. The field of view control device according to any one of claims 1 to 14.

16. Imaging means, A field of view control device according to any one of claims 1 to 15, A driving means for driving the state of the imaging device to set the field of view corresponding to the control information output by the field of view control device, An imaging device characterized by having the following features.

17. An imaging system comprising an imaging device equipped with a drive means for changing the field of view setting, and a field of view control device, The aforementioned angle of view control device is A determination means for determining the field of view setting to be applied to the imaging device, Based on the angle of view setting determined by the determination means, an output means configures control information for the drive means related to changing the angle of view setting and outputs it to the imaging device, Acquisition means for acquiring the captured image of the aforementioned imaging device, A storage means for storing a template that defines the distribution of images of a predetermined object within a predetermined field of view, A means for identifying the position of the image of a predetermined object included in the captured image, It has, The determination of the field of view setting to be applied to the imaging device based on the template by the determination means includes a first step and a second step following the first step. The aforementioned determination means is In the first step, the angle of view setting to be applied to the imaging device is determined so as to reach the predetermined angle of view setting. In the second step, a field of view setting is determined to be applied to the imaging device such that, in the first image captured by the acquisition means in accordance with the predetermined field of view setting, the edge subject appearing at at least one end in either the horizontal or vertical direction in the first image captured image is captured at the end of the field of view in that direction, and the predetermined object is captured at the corresponding position defined in the template. An imaging system characterized by the following:

18. A determination process for determining the field of view setting to be applied to the imaging device, Based on the field of view setting determined in the aforementioned determination step, an output step is performed to configure control information related to changing the field of view setting and output it to the imaging device. The acquisition step involves acquiring the captured image from the aforementioned imaging device, A control method for a field of view control device having, The aforementioned field of view control device includes a storage means for storing a template that defines the distribution of images of a predetermined object within the field of view for a predetermined field of view setting. The control method further includes a selection step of identifying the position of the image of a predetermined object included in the captured image, The determination of the field of view setting to be applied to the imaging device based on the template in the determination step includes a first step and a second step following the first step. In the aforementioned decision-making process, In the first stage, the angle of view setting to be applied to the imaging device is determined so as to reach the predetermined angle of view setting. In the second stage, a field of view setting is determined to be applied to the imaging device such that, in accordance with the predetermined field of view setting, the edge subject appearing at at least one end in either the horizontal or vertical direction in the first image acquired in the acquisition step is captured at the end of the field of view in that direction, and the predetermined object is captured at the corresponding position defined in the template. A control method characterized by the following:

19. On the computer, A process to determine the field of view setting to be applied to the imaging device, Based on the field of view setting determined in the aforementioned determination process, a process is performed to configure control information related to changing the field of view setting and output it to the imaging device. The process of acquiring the captured image from the aforementioned imaging device, A program that executes, The computer includes a storage means for storing a template that defines the distribution of images of a predetermined object within a field of view for a predetermined field of view setting. The program further causes the computer to perform a process to identify the position of the image of the predetermined object included in the captured image, The determination of the field of view setting to be applied to the imaging device based on the template in the process of determining the above includes a first step and a second step following the first step, In the process of making the above determination, In the first stage, the angle of view setting to be applied to the imaging device is determined so as to reach the predetermined angle of view setting. In the second stage, a field of view setting is determined to be applied to the imaging device such that, in accordance with the predetermined field of view setting, the edge subject appearing at at least one end in either the horizontal or vertical direction in the first image acquired in the acquisition process is captured at the end of the field of view in that direction, and the predetermined object is captured at the corresponding position defined in the template. A program characterized by the following features.

Images