Image capture control device, image capture control method, and program

Abstract

This will reduce the workload for operators. [Solution] The shooting control device includes a detection means for detecting a subject from a captured image, a distance acquisition means for acquiring the distance between a plurality of detected subjects, and a control means for switching between a first control for the shooting device that takes an image and a second control for the shooting device that is different from the first control, according to the distance between subjects. The first control is a control that sets the shooting direction of the shooting device based on the position of the subject so that the shooting device tracks the subject, and the second control is a control that sets the shooting direction of the shooting device to a predetermined shooting direction so that the composition captured by the shooting device is a predetermined composition.

Description

Technical Field

[0001] The present invention relates to a technique for controlling a photographing apparatus.

Background Art

[0002] In recent years, a method of automatically performing photographing and production by an edge AI device controlling a photographing apparatus capable of changing the photographing direction (pan / tilt direction) and the angle of view (zoom value) has been spreading. Hereinafter, a photographing apparatus capable of changing pan / tilt / zoom is referred to as a PTZ camera. As a method of automatically controlling a PTZ camera, a technique of detecting a desired subject in a photographed video using AI (Artificial Intelligence) and controlling the PTZ camera so as to track the subject is known. Further, by applying the AI technology and determining the photographing direction of the PTZ camera based on the positional relationship of a plurality of detected subjects, it is possible to automatically control the PTZ camera so that not only a single subject but also a plurality of subjects are within the angle of view.

[0003] Further, Patent Document 1 discloses a technique of using an operating camera capable of controlling vertical and horizontal movement to include a moving object group composed of a plurality of moving objects included in a predetermined area within the photographing angle of view of the camera. By using this technique, for example, in a combat sport composed of a plurality of players and a referee such as judo or boxing, it is possible to photograph so that the plurality of players and the referee are within the photographing angle of view. That is, the photographing direction of the PTZ camera can be controlled so that the plurality of players and the referee are within the photographing angle of view of the PTZ camera, and photographing can be automated.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] On the other hand, in filming competitive sports, a sense of realism is achieved by appropriately switching between close-ups that capture the players up close and wide shots that capture the entire venue. However, even with the aforementioned PTZ camera, switching compositions and the resulting changes in camera work still require operation by the camera operator.

[0006] Therefore, the present invention aims to reduce the workload on the operator. [Means for solving the problem]

[0007] The present invention provides a photographic control device comprising: detection means for detecting a subject from a captured image; distance acquisition means for acquiring the distance between a plurality of detected subjects; and control means for switching between a first control for a photographic device that takes the image and a second control for the photographic device that is different from the first control, according to the distance between subjects, wherein the first control is a control that sets the shooting direction of the photographic device based on the position of the subject so that the photographic device tracks the subject, and the second control is a control that sets the shooting direction of the photographic device to a predetermined shooting direction so that the composition captured by the photographic device is a predetermined composition. [Effects of the Invention]

[0008] According to the present invention, the workload of the operator can be reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example configuration of the imaging system according to the first embodiment. [Figure 2] This figure shows an example of the internal configuration of each device in the first embodiment. [Figure 3] This is a flowchart of the automatic selection area setup operation in the first embodiment. [Figure 4] This figure shows an example of a UI for various settings related to shooting. [Figure 5]This is a flowchart of the overhead view setup operation in the first embodiment. [Figure 6] This is a flowchart of the tracking operation in the first embodiment. [Figure 7] This is an explanatory diagram illustrating an example of polar coordinate transformation. [Figure 8] This is an explanatory diagram of the longest distance between subjects. [Figure 9] This figure shows an example configuration of the imaging system according to the second embodiment. [Figure 10] This figure shows an example of the internal configuration of each device in the second embodiment. [Figure 11] This is a flowchart of the automatic selection area setup operation in the second embodiment. [Figure 12] This is a flowchart of the overhead view setup operation in the second embodiment. [Figure 13] This is a flowchart of the tracking operation in the second embodiment. [Figure 14] This is an explanatory diagram showing the longest and shortest distances between subjects. [Figure 15] This diagram shows examples of different body parts when the subject is a person. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not intended to limit the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). In addition, in the following embodiments, the same or similar configurations and processing steps are denoted by the same reference numerals, and redundant explanations are omitted.

[0011] <Photography system of the first embodiment> In the first embodiment, as an example, a shooting system configured to include a shooting device (PTZ camera) capable of changing the shooting direction (pan / tilt direction) and the angle of view (zoom value), an edge AI device, and a PC (Personal Computer) will be described. The first embodiment is an example in which the edge AI device serves as a shooting control device that controls the PTZ camera. In the first embodiment, the edge AI device detects a desired subject from the captured image of the PTZ camera, and controls the shooting direction and the angle of view of the PTZ camera so as to automatically track the subject. In the embodiments described below, as the subject to be detected, three people, namely two players and one referee who are engaged in a competitive game, are taken as an example, but the number of subjects to be detected is not limited to three people.

[0012] FIG. 1 is a diagram showing a schematic configuration example of the shooting system according to the first embodiment. As shown in FIG. 1, the shooting system of the present embodiment is a system in which a PTZ camera 100, an edge AI device 200, and a PC 300 are connected via a network 400. The network 400 is assumed to be, for example, a LAN (Local Area Network), but other networks may be used, and a video cable or the like may be included.

[0013] The PTZ camera 100 includes a shooting optical system, an image sensor, an image processing unit, and the like. The PTZ camera 100 transmits an image (referred to as a captured video) obtained by shooting with the image sensor and undergoing image processing by the image processing unit to the edge AI device 200 and the PC 300 via the network 400. The PTZ camera 100 also includes a drive unit for driving pan / tilt / zoom. The drive unit changes the shooting direction (pan / tilt direction) by rotating the PTZ camera in the pan / tilt direction, and changes the angle of view by changing the zoom value of the shooting optical system. Details of the configuration, functions, operations, etc. of the PTZ camera 100 in the present embodiment will be described later.

[0014] The PC 300 transmits information for various shooting settings to the edge AI device 200 and also displays the captured video received from the PTZ camera 100. The various shooting settings include, in addition to the general shooting settings in the PTZ camera, settings related to a predetermined target area and settings related to a predetermined composition, which will be described later in this embodiment. The PC 300 generates information on various shooting settings based on an input from a user (e.g., an operator) and transmits the information on various shooting settings for shooting to the edge AI device 200. The details of the configuration, functions, and operations of the PC 300 in this embodiment will be described later.

[0015] The edge AI device 200 performs inference by AI on the captured video received from the PTZ camera 100 to detect a subject. Further, the edge AI device 200 calculates the shooting direction and the angle of view of the PTZ camera 100 so as to track the subject based on the subject detected by the inference and the various shooting settings received from the PC 300. In the case of the first embodiment, the edge AI device 200 has a function as a shooting control device, generates a control signal for controlling the shooting direction and the angle of view of the PTZ camera 100, and transmits it to the PTZ camera 100 via the network 400. As a result, in the PTZ camera 100, a pan-tilt operation and a zoom operation are performed based on the control signal received from the edge AI device 200. Although details will be described later, the edge AI device 200 in this embodiment performs control of automatic tracking shooting of a subject by the PTZ camera 100, automatic switching of composition and camera work based on various shooting setting information, and the like. The details of the configuration, functions, and operations of the edge AI device 200 in this embodiment will be described later.

[0016] In the shooting system of this embodiment, the PC 300 accesses a web server inside the edge AI device 200 based on user input, and then transmits various shooting-related setting information to the edge AI device 200 based on further user input. The edge AI device 200 then controls the PTZ camera 100 to track the subject and perform actions such as switching to a predetermined composition, as described later. Note that various shooting-related settings can be performed in various ways, including accessing the web server inside the edge AI device 200, or by launching an application program on the PC 300, and are not limited to any one of these methods.

[0017] <Internal Configuration of Each Device in the Imaging System> Figure 2 shows an example of the internal configuration of each device included in the imaging system shown in Figure 1: the PTZ camera 100, the edge AI device 200, and the PC 300. First, let's explain the internal configuration of the PTZ camera 100. The PTZ camera 100 includes a CPU 101, RAM 102, ROM 103, video output I / F 104, network I / F 105, image processing unit 106, image sensor 107, drive I / F 108, drive unit 109, and an internal bus 110. The internal bus 110 is connected to the CPU 101, RAM 102, ROM 103, video output I / F 104, network I / F 105, image processing unit 106, and drive I / F 108. The image sensor 107 is connected to the image processing unit 106, and the drive unit 109 is connected to the drive I / F 108.

[0018] The CPU 101 is a central processing unit that controls the entire PTZ camera 100 and performs various calculations. ROM103 is a non-volatile storage device, such as flash memory, HDD, SSD (Solid State Drive), and SD card. ROM103 is used as a persistent storage area for the OS, various programs, and various data, as well as for short-term data storage. RAM102 is a storage device such as DRAM (Dynamic Random Access Memory). RAM102 is loaded with the OS (operating system), various programs, and various data from ROM103, and is also used as a workspace for the OS and various programs. The CPU 101 executes the program loaded from the ROM 103 into the RAM 102, thereby enabling the operation of the PTZ camera 100, as described later.

[0019] The image sensor 107 has an image sensor such as a CCD or CMOS, and acquires image data of an optical image formed by an imaging optical system (not shown), and outputs it to the image processing unit 106. The image processing unit 106 receives image data from the image sensor 107, converts it to a predetermined format, and performs image processing such as compression as necessary before transferring it to the RAM 102. The image processing performed by the image processing unit 106 also includes adjusting the image quality of the image data received from the image sensor 107, as well as cropping to extract only a predetermined area of ​​the image.

[0020] The video output I / F 104 is an interface (I / F) for outputting captured video, which has been acquired by the image sensor 107 and processed by the image processing unit 106, to an external source. The video output I / F 104 is composed of, for example, SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). In this embodiment, the video output I / F 104 is connected to the video input I / F 208 of the edge AI device 200, which will be described later.

[0021] The network interface 105 is an interface for connecting to the aforementioned network 400. The network interface 105 handles communication between the edge AI device 200 and external devices such as the PC 300 via a communication channel such as Ethernet®. In this embodiment, remote camera control of the PTZ camera 100 by the edge AI device 200 is assumed to be performed via the network interface 105, but it may also be performed via another interface such as a serial communication interface (not shown).

[0022] The drive I / F 108 is the connection point to the drive unit 109 and is responsible for communication to send control signals and other signals to the drive unit 109 and to receive information from the drive unit 109. The drive unit 109 has a mechanical drive system and a motor as a drive source as a rotation mechanism for changing the shooting direction (pan / tilt direction) of the PTZ camera 100. The drive unit 109 also has a lens drive system as a mechanism for changing the angle of view (zoom value) and focusing of the PTZ camera 100's shooting optical system. Based on control signals received from the CPU 101 via the drive I / F 108, the drive unit 109 drives the mechanical drive system and the motor as a drive source of the rotation mechanism to move the PTZ camera 100 in the horizontal direction (pan direction) or the vertical direction (tilt direction). In addition, based on control signals received from the CPU 101 via the drive I / F 108, the drive unit 109 operates the lens drive system within the shooting optical system to perform zoom and focusing operations to optically change the angle of view.

[0023] Next, we will explain the internal configuration of the edge AI device 200. The edge AI device 200 includes a CPU 201, RAM 202, ROM 203, network interface 204, video output interface 205, user input interface 206, inference unit 207, and video input interface 208, with each of these components connected by an internal bus 209.

[0024] CPU201 controls the entire edge AI device 200 and performs various calculations. ROM203 is a non-volatile storage device such as flash memory, HDD, SSD, or SD card. ROM203 is used as a persistent storage area for the OS, various programs, and various data, as well as for short-term data storage. RAM202 is a rewritable, high-speed memory device such as DRAM, from which the OS, various programs, and various data are loaded from ROM203. It is also used as a workspace for the OS and various programs. The CPU 201 executes the program loaded from the ROM 203 into the RAM 202, thereby enabling the operation of the edge AI device 200, as described later.

[0025] Network I / F204 is an interface for connecting to network 400, and is responsible for communication with external devices such as PTZ camera 100 and PC300 via network 400. The video output I / F205 is an interface for outputting setting information of the edge AI device 200, which is displayed within the UI (user interface) screen when setting a predetermined target area or composition on the PC300, as will be described later. The User Input I / F206 is an interface for connecting to a mouse, keyboard, and other input devices, and is configured using USB (Universal Serial Bus), etc. The video input interface 208 is an interface for receiving video footage from the aforementioned PTZ camera 100, and consists of SDI, HDMI, etc.

[0026] The inference unit 207 estimates the presence or absence of a predetermined detection target, such as a person, from the captured video received via the video input I / F 208, and, if a subject exists, its position. The inference unit 207 is composed of a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). While a GPU is generally effective when applied to learning processing, equivalent functionality may be achieved with a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Furthermore, the processing of the inference unit 207 may be handled by the CPU 201.

[0027] Next, I will explain the internal configuration of the PC300. The PC300 has a CPU 301, RAM 302, SSD 303, network interface 304, display unit 305, operation unit 306, and device interface 307, and each of these components is connected to an internal bus 308.

[0028] CPU301 controls the entire PC300 system and performs various calculations. The SSD303 is a non-volatile, high-capacity storage device that can be used as a persistent storage area for the OS, various programs, and various data, as well as for short-term storage of various data. RAM302 is a rewritable, high-speed storage device such as DRAM, which loads the OS, various programs, and various data from SSD303, and is also used as a workspace for the OS and various programs. The CPU 301 executes the program loaded from the SSD 303 to the RAM 302, thereby enabling the operation of the PC 300 as described later.

[0029] Network I / F 304 is an interface for connecting to network 400 and is responsible for communication with external communication devices such as the PTZ camera 100 and edge AI device 200 via network 400. Communication on PC300 involves sending various setting information related to shooting to the edge AI device 200 and receiving information such as captured video from the PTZ camera 100 and the current pan / tilt values ​​(shooting direction) and zoom values ​​(angle of view) of the PTZ camera 100.

[0030] The display unit 305 is a display device for displaying captured video from the PTZ camera 100, as well as UI screens used for setting predetermined target areas and compositions, as described later. While this example shows the PC 300 having a display device, the configuration is not limited to this; for example, a separate display monitor and controller may exist, each solely for displaying captured video and UI screens.

[0031] The operation unit 306 is an interface for receiving user operations to the PC 300, and examples include a mouse, keyboard, buttons, dial, joystick, touch panel, etc. The operation unit 306 receives user operations and inputs to the UI screen used for setting predetermined target areas and predetermined compositions, which will be described later. In this embodiment, mouse operation is assumed as the user operation to the UI screen, and the user's pressing operation to buttons etc. displayed on the UI screen, which will be described later, is assumed to be a mouse click operation. Of course, it is not limited to this, and the user operation to the UI screen may be various operations such as touch operation to the screen of a display device equipped with a touch panel. Based on the user operation to the UI screen, the PC 300 generates various setting information related to shooting for setting predetermined target areas and predetermined compositions, which will be described later, and transmits it to the edge AI device 200 via the network I / F 304. The device I / F307 is an interface for connecting to various input devices and consists of USB (Universal Serial Bus), among others.

[0032] <Explanation of the operation of each device in the shooting system> Next, the operation of each device in the imaging system of the first embodiment will be explained with reference to Figures 3 to 8. In this embodiment, the operation of the shooting system is broadly divided into setup operation and tracking operation. The setup operation is an operation to perform various settings related to shooting, such as a predetermined target area and a predetermined composition, which will be described later, before the tracking operation starts. The tracking operation is an operation to track the subject to be detected based on the various settings related to shooting set in the setup operation.

[0033] <Setup operation> First, let's explain the setup process. In this embodiment, the setup operation for making various settings related to shooting includes setup for setting a predetermined target area and setup for setting a predetermined composition. In this embodiment, the setup for defining a predetermined target area involves setting an automatic selection area. The automatic selection area is an area within the captured video where the subject to be tracked is automatically selected and detected.

[0034] In this embodiment, the setup for setting a predetermined composition involves setting the camera to capture a composition where the entire competition area is centered in the frame. Examples of compositions where the entire competition area is centered in the frame include wide shots that capture the entire competition area broadly, and in this embodiment, one example of such a composition is a bird's-eye view of the entire competition area (hereinafter referred to as a bird's-eye view composition). In the case of a competition involving two athletes and one referee, as illustrated in this embodiment, the bird's-eye view composition is used, for example, when capturing a scene at the start or end of the competition where the referee is in the center and the athletes are positioned on the left and right. It should be noted that the predetermined composition is not limited to a composition where the entire competition area is centered in the frame, a wide shot, or a bird's-eye view composition, but may also include, for example, a composition arbitrarily set by the user, or a specific composition depending on the type of competition or shooting purpose.

[0035] In the imaging system of this embodiment, when the PC300, edge AI device 200, and PTZ camera 100 are started up, the PC300 establishes a connection with the edge AI device 200 and the PTZ camera 100 and enters a standby state. When the standby PC300 receives a setting-up instruction for the automatically selected area from the user via the operation unit 306, it starts the operation of the flowchart shown in Figure 3(a), which will be described later. Furthermore, when the PC300 receives a setting-up instruction for the automatically selected area from the user, it sends a notification to the edge AI device 200. Upon receiving the notification from the PC300, the edge AI device 200 starts the operation of the flowchart shown in Figure 3(b), which will be described later.

[0036] Furthermore, when the standby PC300 receives a setting instruction for the overhead view from the user via the operation unit 306, it starts the operation of the flowchart shown in Figure 5(b), which will be described later. In addition, when the PC300 receives a setting instruction for the overhead view from the user, it sends a notification to that effect to the edge AI device 200 and the PTZ camera 100. When the PTZ camera 100 receives the notification from the PC300, it starts the operation of the flowchart shown in Figure 5(a), which will be described later. Also, when the edge AI device 200 receives the notification from the PC300, it starts the operation of the flowchart shown in Figure 5(c), which will be described later.

[0037] First, we will explain the operation of the flowchart shown in Figure 3(a) that is executed in the PC300 when it receives a setup instruction for the automatically selected area from the user. When the CPU 301 of the PC300 receives a setting instruction for the automatic selection area from the user, it reads and receives the initial value of the automatic selection area from the SSD 303 as the process in step S101. The initial value of the automatic selection area may be, for example, an area selected according to the type of competition from a fixed set of automatic selection areas predetermined for each type of competition, or the automatic selection area used last during the previous operation may be used. The CPU 301 may also, for example, query the edge AI device 200 to obtain information that will be used as the initial value of the automatic selection area.

[0038] Next, in step S102, the CPU 301 displays a UI screen on the display unit 305 for the user to configure settings such as the automatically selected area. Figure 4 shows an example of a UI screen used for setting up automatically selected areas, etc. Note that the UI screen exemplified in Figure 4 also includes a configuration for the user to adjust and finalize the overhead view, which will be described later.

[0039] As shown in Figure 4, the left-hand area of ​​the UI screen displays the captured video received from the PTZ camera 100, and the automatic selection area 500 is superimposed on the captured video. In the example in Figure 4, the captured video shows two players 600a and 600b competing in the competition area 501, one referee 601, and, for example, a person 602 such as a substitute player located outside the competition area 501. Although the person 602 outside the competition area 501 is assumed to be a substitute player, it could be another person such as a spectator. The automatic selection area 500 is an area set by the user to match the competition area 501 through operation of the operation unit 306. For example, after the initial value of the automatic selection area is set by the CPU 301, the user can set any automatic selection area 500 by operating the initial value of the automatic selection area through the operation unit 306, as described later.

[0040] The right-hand side of the UI screen contains the PTZ setting button 700, the automatic selection area confirmation button 701, the overhead composition adjustment start button 702, and the overhead composition confirmation button 703. The automatic selection area confirmation button 701 is pressed by the user to confirm the automatic selection area 500 after user operation has been performed on the automatic selection area 500 in the left-hand side of the UI screen. The PTZ setting button 700 includes a cross-shaped button 710 for the user to set the pan and tilt of the PTZ camera 100, and a tele-wide button 711 for the user to set the zoom (angle of view) of the PTZ camera 100. When the user operates the cross-shaped button 710 or the tele-wide button 711 of the PTZ setting button 700, the PC 300 sends pan, tilt, and zoom control commands to the PTZ camera 100 according to the user operation information. As a result, the shooting direction and angle of view of the PTZ camera 100 are changed, and the captured image displayed in the left-hand side of the UI screen is changed. The PTZ setting button 700 is also used when adjusting the overhead composition, which will be explained later. The roles of the overhead composition adjustment start button 702 and the overhead composition confirmation button 703, as well as the role of the PTZ setting button 700 when adjusting the overhead composition, will be explained later.

[0041] In this embodiment, an example is given in which the user sets an arbitrary automatic selection area 500 based on the initial automatic selection area, but it is not limited to this. For example, the CPU 301 may detect the competition area 501 from the captured video using AI technology, etc., and automatically set the automatic selection area 500 according to the detected competition area 501. Also, in the case of Figure 4, the automatic selection area 500 is represented as a rectangular area, but it is not limited to this, and any shape that is suitable for the competition area 501 is acceptable, such as a polygon or a circle. In this embodiment, as will be described later, the automatic selection area 500 is an area that automatically selects the subject to be tracked in the captured video, so it is possible to distinguish between the subjects to be tracked, such as players and referees, and other subjects such as substitute players. In other words, substitute players and spectators outside the automatic selection area are excluded from the tracking targets, so it is possible to track only the players and referees within the automatic selection area.

[0042] The UI screen described in Figure 4 may be displayed by an application program running on the PC300. Alternatively, the edge AI device 200 may be equipped with a web server, and the UI screen may be displayed as content downloaded by the PC300 from there.

[0043] Let's return to the explanation of the flowchart in Figure 3(a). After the processing in step S102, the CPU 301 loops through the processing in steps S103 to S104 until the user presses the automatic selection area confirmation button 701. In step S103, the CPU 301 obtains user operations on each of the four vertices of the automatically selected region 500 from the operation unit 306, and sets the automatically selected region 500 based on the position of each vertex on which the user operation was performed. In other words, the user can set any automatically selected region 500 by manipulating the position of each vertex of the automatically selected region 500 via the operation unit 306. The CPU 301 then writes the coordinate information of each vertex of the automatically selected region set according to the user operation to the RAM 302. Note that user operations on the positions of the four vertices of the automatically selected region 500 can be performed by various operations, such as drag-and-drop operations using a mouse, but this embodiment is not limited to any one of these operations.

[0044] Next, in step S104, the CPU 301 determines whether the automatic selection area determination button 701 has been pressed by the user via the operation unit 306. If the CPU 301 determines that the automatic selection area determination button 701 has been pressed, it exits the loop and proceeds to step S105. When the process proceeds to step S105, the CPU 301 reads the coordinate information of the automatically selected region stored in the RAM 302 and sends it to the edge AI device 200 via the network I / F 304.

[0045] Next, we will explain the flowchart process shown in Figure 3(b) that is executed on the edge AI device 200 during the setup of the automatically selected region. The CPU 201 of the edge AI device 200 is in a state of waiting to receive coordinate information of the automatically selected area. When it receives the coordinate information of the automatically selected area from the PC 300 via the network I / F 204 in step S201, it proceeds to the next step S202. When the process proceeds to step S202, the CPU 201 writes the coordinate information of the automatically selected region to the RAM 202.

[0046] Next, we will explain the flowchart process shown in Figure 5(b) that is executed in PC300 when it receives instructions from the user to set up an overhead view. As part of the overhead composition setup operation, PC300 sets the shooting direction (pan and tilt values) and field of view (zoom value) of the PTZ camera 100 to be set as an overhead composition for the edge AI device 200. In this embodiment, the overhead composition is a composition in which the entire competition area is shown in the center of the field of view, as described above, and is a composition for overhead shooting of a scene at the start or end of a competition in which the referee is in the center and the players are on the left and right. For example, the overhead composition is the composition of the captured video displayed in the left area of ​​the UI screen in Figure 4, that is, a composition that shows not only the players 600a, 600b and the referee 601 in the competition area 501, but also people 602 such as substitute players outside the competition area 501.

[0047] When the overhead composition setup begins, the CPU 301 of PC300 is waiting for user input for the overhead composition adjustment start button 702 located in the right-hand area of ​​the UI screen in Figure 4. In step S401, when the CPU 301 receives input from the user by pressing the overhead composition adjustment start button 702, it loops through the processes from step S402 to step S403 until the overhead composition confirmation button 703 is pressed.

[0048] In the UI screen shown in Figure 4, the overhead composition adjustment start button 702 located in the right-hand area is pressed when the user instructs to start adjusting the overhead composition, and the overhead composition confirmation button 703 is pressed when the user instructs to confirm the overhead composition. When the overhead composition adjustment start button 702 is pressed, the PC 300 determines that the user has instructed to start adjusting the overhead composition. Then, when the user operates the cross button 710 and the tele-wide button 711 of the PTZ camera 100, the PC 300 sends a control command to the PTZ camera 100 that includes the drive direction and drive amount of pan, tilt, and zoom according to the user's operation. As a result, the PTZ camera 100 performs overhead composition adjustment by adjusting the pan, tilt, and zoom. As a result of this overhead composition adjustment, if the user decides that the overhead composition is satisfactory and presses the overhead composition confirmation button 703, the PC300 determines the pan, tilt, and zoom values ​​of the PTZ camera 100 at that time as the pan, tilt, and zoom values ​​for the overhead composition. These pan, tilt, and zoom values ​​for the overhead composition are then saved to the edge AI device 200.

[0049] Let's return to the explanation of the flowchart in Figure 5(a). When the process proceeds to step S402, the CPU 301 waits for user input for the directional buttons 710 and tele-wide buttons 711 of the PTZ setting button 700 shown in Figure 4. When user input is received for the directional buttons 710 and tele-wide buttons 711 of the PTZ setting button 700, the CPU 301 sends pan-tilt-zoom control commands corresponding to that user input information to the PTZ camera 100. For example, if a pan-tilt operation is input using the directional buttons 710, the PC 300 sends a control command to the PTZ camera 100 via the network I / F 304 to drive the PTZ camera to pan-tilt with the corresponding pan-tilt value. Similarly, if a zoom operation is input using the tele-wide button 711, the CPU 301 sends a control command to the PTZ camera 100 via the network I / F 304 to drive the PTZ camera to zoom with the corresponding zoom value.

[0050] Next, in step S403, the CPU 301 determines whether or not the user has pressed the overhead composition selection button 703 via the operation unit 306. If the CPU 301 determines that the user has pressed the overhead composition selection button 703, it exits the loop and proceeds to step S404.

[0051] When the process proceeds to step S404, the CPU 301 sends a command to the PTZ camera 100 requesting the transmission of the current pan, tilt, and zoom values. Next, in step S405, the CPU 301 receives information transmitted from the PTZ camera 100 as a response to the request command sent in step S404 via the network I / F 304. Specifically, the information received at this time is the current pan, tilt, and zoom values ​​of the PTZ camera 100.

[0052] Next, in step S406, the CPU 301 transmits the pan, tilt, and zoom values ​​received in step S404 to the edge AI device 200 via the network I / F 304. These pan, tilt, and zoom values ​​are used by the edge AI device 200 to set the PTZ camera 100 to a shooting direction and field of view corresponding to an overhead composition.

[0053] Next, the operation of the PTZ camera 100 after the pan, tilt, and zoom values ​​for the overhead composition have been determined by the overhead composition setup operation described above will be explained with reference to the flowchart in Figure 5(a). The CPU 101 of the PTZ camera 100 is waiting to receive commands sent from the PC 300. In step S301, when the CPU 101 receives a command from the PC 300 via the network I / F 105 requesting the transmission of pan, tilt, and zoom values, it proceeds to step S302.

[0054] When the process proceeds to step S302, the CPU 101 reads the current pan, tilt, and zoom values ​​stored in RAM 102. The CPU 101 then reads the current pan, tilt, and zoom values ​​from the RAM 102 and sends them to the PC 300 via the network interface 105.

[0055] Next, the operation of the edge AI device 200 after the pan, tilt, and zoom values ​​for the overhead composition have been determined by the overhead composition setup operation described above will be explained with reference to the flowchart in Figure 5(c). The CPU 201 of the edge AI device 200 is waiting to receive information transmitted from the PC 300. In step S501, when the CPU 201 receives pan, tilt, and zoom values ​​for setting the overhead composition from the PC 300 via the network I / F 204, it proceeds to step S502. When the process proceeds to step S502, the CPU 201 writes the received pan, tilt, and zoom values ​​to the RAM 202 as pan, tilt, and zoom values ​​for the overhead shot.

[0056] <Tracking operation and switching to an overhead view> The shooting system of this embodiment allows switching between a first control and a second control, which is different from the first control, depending on the distance between subjects, and its operation will be described below. In this embodiment, the first control is given as a control that operates the PTZ camera 100 to automatically track the subject, and the second control is given as an example of controlling the PTZ camera 100 to an overhead composition.

[0057] In the shooting system of this embodiment, after the setup of the automatic selection area and overhead composition described above is completed, the subject tracking operation and switching to the overhead composition are performed using the various shooting setting information set by these setups. In the shooting system of the first embodiment, the edge AI device 200 detects the subject position from the image captured by the PTZ camera 100 and performs automatic tracking by controlling the pan, tilt, and zoom of the PTZ camera 100 according to the subject position. The edge AI device 200 also obtains the distance between subjects from the multiple subject positions that it has inferred, and switches between automatic tracking and the overhead composition based on that distance between subjects.

[0058] Figure 6(a) shows a flowchart of the tracking operation in the edge AI device 200. When controlling the tracking operation, the edge AI device 200 obtains the distance between subjects from the captured video and decides whether to switch to an overhead view composition based on that distance. Figure 6(b) shows a flowchart of the operation in the PTZ camera 100. First, referring to the flowchart in Figure 6(a), we will explain the control of tracking operations and the switching to an overhead view performed by the edge AI device 200.

[0059] In the shooting system of this embodiment, the PTZ camera 100 sequentially transmits captured video from the video output I / F 104 at a predetermined frame rate. The edge AI device 200 sequentially receives the captured video transmitted sequentially from the PTZ camera 100 at a predetermined frame rate via the video input I / F 208 and sequentially stores it in the internal RAM 202. Alternatively, the PTZ camera 100 may also sequentially transmit the captured video from the network I / F 105 at a predetermined frame rate. In this case, the edge AI device 200 sequentially receives the captured video via the network I / F 204 and stores it in the RAM 202. The loop processing from steps S601 to S611 in Figure 6(a) in the edge AI device 200 is performed for each frame of captured video.

[0060] In step S601, the CPU 201 of the edge AI device 200 sequentially reads the captured video stored in the RAM 202 and transfers it to the inference unit 207. As part of the processing in step S602, the inference unit 207 detects a subject from the captured video and writes the inference result information to the RAM 202. In this embodiment, the inference unit 207 has a trained model created using machine learning methods such as deep learning, and acquires the captured video as input data and outputs the inference result as output data. The inference result includes not only the position information of the person being tracked, such as an athlete or referee, but also the type of the subject being tracked (for example, whether it is an athlete or a referee), and a score indicating the likelihood of them being identified. The position information of the subject (person) also includes the coordinate information of the four vertices of the top-left, top-right, bottom-left, and bottom-right of the rectangular area surrounding the subject, as well as information such as the width and height of the rectangular area. The inference unit 207 may also be configured to output information indicating the part of the subject (person) on the image as output data. In this embodiment, if the subject is a person, examples of parts include the nose, eyes, ears, and head, as well as joint parts such as the shoulders, elbows, wrists, hips, knees, and ankles, and skeletal parts. The inference unit 207 may also be configured to output data that indicates at least one or more parts of the subject in the image. Examples of how this information about parts of a person can be used will be described later. The inference unit 207 acquires this inference result information as a set.

[0061] Next, in step S603, the CPU 201 reads from the RAM 202 the coordinate information indicating the automatically selected region that was stored in step S202 in Figure 3(b) above. Next, in step S604, the CPU 201 reads the position information of the rectangular area of ​​the subject from the inference results stored in the RAM 202 in step S602, and counts the number of subjects that exist within the automatically selected area based on the position information of that rectangular area. In other words, the CPU 201 counts the number of people that exist within the automatically selected area. In this embodiment, the CPU 201 counts subjects whose center point of the bottom edge of the rectangular area is included within the automatically selected area as subjects that exist within that automatically selected area.

[0062] Here, in order to determine whether the subject is included within the automatically selected area regardless of the pan / tilt direction or zoom value of the PTZ camera 100, the CPU 201 converts the coordinate system of the coordinate information indicating the center point of the bottom edge of the rectangular area of ​​the subject and the automatically selected area to a predetermined coordinate system. In this embodiment, the coordinate information of the center point of the bottom edge of the rectangular area of ​​the subject and each vertex of the automatically selected area is coordinate information of a Cartesian coordinate system expressed as (x,y) on the captured image. Therefore, the CPU 201 converts this Cartesian coordinate system coordinate information into polar coordinate system coordinate information where the pan / tilt angle when the PTZ camera 100 is facing the front of the competition area is set to 0 degrees, the angle in the pan direction is θq [rad], and the angle in the tilt direction is φq [rad]. As a result, the coordinate information of the subject and the automatically selected area can be expressed as coordinate information that does not depend on the pan / tilt / zoom values ​​of the PTZ camera 100. Therefore, the CPU 201 can determine whether the subject is included within the automatically selected area regardless of the pan / tilt / zoom values ​​of the PTZ camera 100.

[0063] Below, as an example of a method for converting from a Cartesian coordinate system represented by (x,y) to a polar coordinate system, we will explain how to convert the two-dimensional coordinate P(x,y) on the captured video to a three-dimensional coordinate Q(X,Y,Z) with the PTZ camera as the origin, using Figures 7(a) to 7(c). Figure 7(a) shows the captured image 1000 from the PTZ camera 100 in a Cartesian coordinate system (x,y), and represents the point (pixel) where the two-dimensional coordinate P(x,y) in the figure is converted to the three-dimensional coordinate Q(X,Y,Z). In Figure 7(a), x [pixels] to the right of the center of the captured image 1000 are positive values, and y [pixels] below the center are positive values. The image size of the captured image 1000 is assumed to be w × h [pixels].

[0064] Figure 7(b) shows a sphere 1001 in a three-dimensional space with the PTZ camera 100 at its origin O, with the radius being the distance from the PTZ camera 100 to the subject in the captured image. For simplicity of explanation, the radius of the sphere 1001 is normalized to 1 in Figure 7(b). As shown in Figure 7(b), when the position of the PTZ camera 100 is represented in a three-dimensional space with the origin O, the captured image 1000 shown in Figure 7(a) can be represented as a two-dimensional image tangent to the sphere 1001 at its center R.

[0065] Figure 7(c) shows the current pan angle θcam and tilt angle φcam of the PTZ camera 100, with the pan-tilt angle set to 0 degrees when the PTZ camera 100 is facing the front of the competition area. The front of the PTZ camera 100 is assumed to be the x-axis direction in Figure 7(c). The pan angle θcam, tilt angle φcam, horizontal zoom field of view ψwcam (not shown), and vertical zoom field of view ψhcam (not shown) can be obtained by the edge AI device 200 requesting the current pan-tilt-zoom values ​​from the PTZ camera 100.

[0066] As shown in Figure 7(b), if xpp is the distance in the x-axis direction from the center R of the captured video 1000 to the three-dimensional coordinate Q(X,Y,Z) and ypp is the distance in the y-axis direction, then these distances xpp and ypp can be calculated using the following equations (1) and (2). Furthermore, the three-dimensional coordinate Q(X,Y,Z) can be calculated using the following equation (3).

[0067]

number

[0068] Since the orientation of the PTZ camera 100 is in the direction of the pan angle θcam and the tilt angle φcam, the three-dimensional coordinate Q(X,Y,Z) can be calculated by rotating the coordinate axes around the Z axis by θcam and around the Y axis by φcam, as shown in equation (3). As a result, the CPU 201 can convert a point P(x,y) on the captured video 1000 into a three-dimensional coordinate Q(X,Y,Z) with the PTZ camera as the origin.

[0069] Next, CPU201 converts the three-dimensional coordinate Q(X,Y,Z) into the pan angle θq and tilt angle φq as seen from the PTZ camera 100, using equations (4) and (5) below.

[0070]

number

[0071] As described above, the CPU 201 converts the coordinate information of the center point of the bottom edge of the rectangular region of the subject and the four vertices indicating the automatically selected region into the pan angle θq and tilt angle φq as seen from the PTZ camera 100, using equations (1) to (5), and calculates. As a result, even if the pan, tilt, and zoom values ​​of the PTZ camera 100 change, the CPU 201 can perform the processing in step S604.

[0072] The aforementioned method for converting to polar coordinates is merely one example; any existing calculation method that converts to polar coordinates may be used. In this embodiment, the pan, tilt, and zoom values ​​of the PTZ camera 100 are used to convert to polar coordinates. However, in the case of a camera that can only control pan, for example, the polar coordinates can be converted based on the pan value. The same applies to cameras that can only control tilt; the polar coordinates can be converted based on the tilt value.

[0073] In this embodiment, the reason for using the midpoint of the bottom edge of the rectangular area of ​​the subject as described above is to determine whether the subject's position is included within the automatically selected area. Furthermore, using the midpoint of the bottom edge of the rectangular area of ​​the subject is just one example of how to determine whether the subject is included within the automatically selected area; another method involves using information indicating parts of the subject. That is, information indicating parts of the subject can be used to determine whether the subject is included within the automatically selected area by using information on at least one part of each part acquired by the inference unit 207 described above.

[0074] When the subject is a person, information indicating each part can include, as an example, the coordinate information of the positions of various parts such as the head, nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles, as shown in Figure 15. Furthermore, the information indicating each part may also include the coordinate information of a rectangular area surrounding the part. In Figure 15, for example, position 1105 indicates the position of the head, positions 1106 and 1107 indicate the positions of the shoulders, positions 1103 and 1104 indicate the positions of the wrists, and positions 1101 and 1102 indicate the positions of the ankles. Also in Figure 15, rectangular area 1108 shows an example of a rectangular area surrounding the feet. The aforementioned inference unit 207 is capable of outputting coordinate information for at least one of these parts. The CPU 201 then uses the information of at least one part output by the inference unit 207 to determine whether the subject is included within the automatically selected area.

[0075] Specifically, the CPU 201 may determine that a subject (person) is included within the automatic selection area if, for example, at least one of the coordinate information points for the right ankle and the left ankle is included within the automatic selection area. This allows the CPU 201 to more accurately determine whether a subject (person) is within the automatic selection area, even in cases where, for example, one of the person's feet is outside the automatic selection area near its boundary.

[0076] Furthermore, for example, if the coordinate information of at least one of the following—wrist, head, or shoulder—in addition to the ankle, is included within the automatic selection area, the CPU 201 may determine that the subject is included within the automatic selection area. This allows the CPU 201 to determine that the subject is included within the automatic selection area even if the ankle is not, as long as the wrist or head is included. In such cases, determining whether a part of the subject's body, not just the ankle, is included within the automatic selection area allows for a more accurate determination of whether the subject is included within the automatic selection area.

[0077] For example, if the coordinate information of a rectangular area surrounding the right or left foot of a person in the subject is included within the automatically selected area, the CPU 201 may determine that the subject is included within the automatically selected area. Specifically, the CPU 201 may determine that the subject is included within the automatically selected area if at least one of the midpoints of the bottom edges of the rectangular areas surrounding the right and left feet is included within the automatically selected area.

[0078] Furthermore, in addition to methods that determine whether a part of the subject is included in the automatically selected area, it is also possible to use determination methods based on positions that can be obtained from the location of the subject's body parts, such as determining whether the midpoint of both feet or the surrounding area of ​​the subject's body parts is included.

[0079] Let's return to the explanation of the flowchart in Figure 6(a). Next, in step S605, the CPU 201 determines whether the number of subjects counted in step S604 satisfies the condition that it is a predetermined number. In this embodiment, as mentioned above, two players competing and one referee are used as an example, so the predetermined number to be determined in step S605 is three. If the CPU 201 determines that the number of subjects counted is three, it proceeds to the automatic tracking process from step S606 onwards. On the other hand, if it determines that the number is not three, it skips the processes from steps S606 to S611 and proceeds to the next loop process.

[0080] Furthermore, after the tracking operation has started when it has been determined that there are three subjects, if it is determined in step S605 during the loop processing that there are not three subjects, the CPU 201 may control the pan, tilt, and zoom values ​​of the PTZ camera 100 to be fixed. In other words, after automatic tracking has started, if the number of people in the automatic selection area falls below a predetermined number (less than three), the CPU 201 stops controlling the automatic tracking. One example of when the number of subjects falls below three is when two of the three athletes leave the automatic selection area. In this case, since the control of automatic tracking is stopped, it is possible to prevent the two athletes from going out of frame by focusing the tracking on the one person remaining in the automatic selection area (competition area) (for example, the referee). After that, when the two athletes return to the automatic selection area and the CPU 201 determines in step S605 that there are three subjects in the automatic selection area, the process proceeds to step S606, and the control of the PTZ camera 100 (automatic tracking) is resumed.

[0081] When the process proceeds to step S606, the CPU 201 performs a distance acquisition process to obtain the distance between each subject included in the automatically selected area, and a distance determination process to determine whether the longest distance between subjects is greater than or equal to a predetermined distance. The predetermined distance is a distance threshold set as an appropriate distance according to the type of competition. For example, in the case of a competition where the positions of the players at the start of the match are generally fixed, such as in judo or sumo, it is preferable to set the distance between the players at the start of the match as the predetermined distance. Of course, it is not limited to this, and the predetermined distance may be various distances according to the type of competition, or it may be a distance arbitrarily set by the user.

[0082] Here, we will explain the distance between subjects and the longest distance between subjects using Figure 8. Figure 8(a) shows an example of the relative positions of each player and the referee at the start or end of the competition, and Figure 8(b) shows an example of the relative positions of each player and the referee during the competition. In the positional relationship between each athlete and the referee at the start or end of the competition, as shown in Figure 8(a), the longest subject distance 800a among the subject distances between the two athletes 600a and 600b and the one referee 601 is the distance between athlete 600a and athlete 600b. In contrast, in the positional relationship between each athlete and the referee during the competition, as shown in Figure 8(b), the subject distances between athletes 600a and 600b and the referee 601 are often closer. In the example in Figure 8(b), the longest subject distance 800b is, for example, the distance between athlete 600b and referee 601. Thus, the longest subject distance often differs between the start or end of the competition and during the competition. Therefore, by obtaining the longest subject distance, it is possible to determine whether it is the start or end of the competition, or during the competition.

[0083] In this embodiment, as described above, the distance between players at the start of the match is set as a predetermined distance. This allows, for example, to determine that the match is in progress if the longest distance between subjects is less than the predetermined distance, while it can be determined that the match has started or ended if the longest distance between subjects is equal to or greater than the predetermined distance.

[0084] If the longest distance between subjects in step S606 is less than a predetermined distance, CPU201 proceeds to step S607. Step S607 is reached when the CPU 201 determines that the three subjects detected within the automatically selected area will be tracked, and then calculates the centroid position of these three subjects. For example, the CPU 201 calculates the centroid position of two or more subjects (e.g., three people) from the average of the center point positions of each rectangular area of ​​those subjects. Note that the method for calculating the centroid position of a subject is not limited to this example, and other calculation methods may be used, such as using the center point of the circumscribing rectangular area that encloses all three subjects, or distinguishing between players and referees and using the average of the center point positions of only the players as the centroid position.

[0085] Next, in step S608, the CPU 201 determines whether the centroid position calculated in step S607 matches the center position of the field of view in the captured video. If the CPU 201 determines that the centroid position matches the center position of the field of view, it skips the subsequent steps up to S611 and proceeds to the next loop. On the other hand, if the CPU 201 determines that the centroid position does not match the center position of the field of view, it proceeds to step S609.

[0086] In step S609, the CPU 201 calculates the difference between the center of gravity calculated in step S607 and the center of the field of view in the captured video, and calculates the pan-tilt angular velocity corresponding to this difference as the pan-tilt adjustment amount. In this embodiment, the difference between the calculated center of gravity and the center of the field of view in the captured video is calculated, but the difference may also be calculated in polar coordinate space by performing a conversion to polar coordinates as described above. For example, for calculating the angular velocity, one method is to multiply the distance, which is the difference between the coordinate values ​​in the pan direction and the tilt direction, by a predetermined coefficient, and then determine the rotation direction of the pan-tilt depending on whether the calculated value is positive or negative. Since these techniques are known, a detailed explanation is omitted.

[0087] In step 609, the CPU 201 calculates the zoom adjustment amount so as to keep the size of the subject's rectangular area roughly constant. The size of the subject's rectangular area may be calculated not only from the size of the subject's bounding rectangle, but also by detecting the size of individual body parts, such as the face, and keeping those sizes constant. Alternatively, the size of the subject's rectangular area may be calculated by randomly selecting one subject from the automatically selected area, or by calculating the average size of the rectangular areas of three subjects. Alternatively, the zoom adjustment amount may be calculated so as to keep the size of the bounding rectangle encompassing all three subjects constant.

[0088] The subject tracking method using the technique described above, which calculates and controls the direction and speed of pan-tilt rotation, is just one example. Any method that allows for subject tracking may be used, such as calculating the target position during pan-tilt rotation to track the subject.

[0089] Next, in step S610, the CPU 201 converts the result calculated in step S609 into a control command according to a predetermined protocol for controlling the PTZ camera 100, and writes it to RAM 202. Next, in step S611, the CPU 201 reads the control command that was converted in step S610 and written to RAM 202, sends it to the PTZ camera 100 via network I / F 204, and then returns to the beginning of the loop.

[0090] In this embodiment, an example was given in which the center of gravity position and the center of the field of view coincide in step S608. However, a so-called dead zone may be provided, for example, in which the PTZ camera 100 is not controlled if the difference between the center of gravity position and the center of the field of view is within a predetermined range. This can reduce, for example, over-sensitive control of the PTZ camera 100.

[0091] Next, if step S606 determines that the longest distance between subjects is greater than or equal to a predetermined distance and the process proceeds to step S612, the CPU 201 reads the pan, tilt, and zoom values ​​indicating the overhead composition that were written to the RAM 202 in step S502. The CPU 201 then determines these pan, tilt, and zoom values ​​as the tracking target positions. In other words, the pan, tilt, and zoom values ​​written to the RAM 202 in step S502 are the pan, tilt, and zoom values ​​for the overhead composition. Therefore, by determining these values ​​as the tracking target positions, the composition of the PTZ camera 100 is switched to an overhead composition.

[0092] Next, in step S613, the CPU 201 generates control commands from the pan, tilt, and zoom values ​​of the overhead view read in step S612, in accordance with a predetermined protocol for controlling the PTZ camera 100, and writes them to the RAM 202. Next, in step S614, the CPU 201 reads the control command written to RAM 202 in step S613, sends it to the PTZ camera 100 via network I / F 204, and then returns to the beginning of the loop process.

[0093] Next, we will explain the flowchart in Figure 6(b) that is executed in the PTZ camera 100 during tracking operation. In step S701, the CPU 101 of the PTZ camera 100 receives control commands from the edge AI device 200, which is operating as shown in the flowchart in Figure 6(a), via the network I / F 105. Next, in step S702, the CPU 101 writes the control command sent from the edge AI device 200 to the RAM 102, and then proceeds to step S702. When the process proceeds to step S702, the CPU 101 reads the values ​​of the drive direction and drive amount corresponding to the adjustment amount in the pan and tilt directions from the control commands stored in RAM 102. The CPU 101 also reads the values ​​of the lens drive direction and drive amount corresponding to the adjustment amount in the zoom from the control commands.

[0094] Next, in step S703, the CPU 101 calculates drive parameters for pan, tilt, and zoom operation based on the values ​​read from RAM 102 in step S702. For example, the CPU 101 calculates drive parameters for controlling the motors for pan and tilt operation in the drive unit 109, and drive parameters for zoom operation, based on the values ​​read from RAM 102. Alternatively, the CPU 101 may convert the values ​​of the drive direction and drive amount included in the received control command into drive parameters by referring to a conversion table previously stored in ROM 103.

[0095] Next, in step S704, the CPU 101 controls the drive unit 109 via the drive I / F 108 based on the drive parameters calculated in step S703. The drive unit 109 performs pan, tilt, and zoom drives based on these parameters, thereby controlling the shooting direction (pan and tilt direction) and angle of view (zoom) of the PTZ camera 100.

[0096] As described above, the shooting system of the first embodiment switches between tracking and overhead composition depending on the distance between subjects. As a result, with the shooting system of this embodiment, the composition and camera work of the PTZ camera can be switched according to the situation at the start, end, or during a competitive event, reducing the workload for the user (operator).

[0097] <Second Embodiment> In the first embodiment, an example was described in which the edge AI device 200 detects a subject from the video captured by the PTZ camera 100 and switches between tracking and overhead view depending on the distance between the detected subjects. In the second embodiment, an example is described in which the processing related to the edge AI device 200 of the first embodiment is performed inside the PTZ camera 100. In other words, the second embodiment is an example in which the functions of the shooting control device are included within the PTZ camera 100. The following description will focus on the differences from the first embodiment.

[0098] Figure 9 shows an example of the configuration of the imaging system according to the second embodiment. As shown in Figure 9, in the second embodiment, the shooting system has a PTZ camera 900 and a PC 300 connected via a network 400. In the second embodiment, the PTZ camera 900 detects a subject from the captured video image and automatically tracks the subject by performing pan, tilt, and zoom operations according to the detection result. The PTZ camera 900 in the second embodiment also acquires the distance between subjects and switches between tracking operation and overhead composition based on the distance between subjects. The PC 300 in the second embodiment performs various settings related to shooting, as in the first embodiment described above, and this setting information is sent to the PTZ camera 900.

[0099] Figure 10 is a diagram showing the internal configuration of the PTZ camera 100 and PC300 in the imaging system according to the second embodiment. The internal configuration and operation of the PC300 in the second embodiment are generally the same as those of the PC300 in the first embodiment, so a detailed explanation is omitted. However, in the second embodiment, the device with which the PC300 communicates via the network I / F304 is the PTZ camera 900. Also, the CPU 901 to the internal bus 910 of the PTZ camera 900 are generally equivalent to the CPU 101 to the internal bus 110 of the PTZ camera 100 in the first embodiment, so a detailed explanation of them is omitted.

[0100] In the second embodiment, the PTZ camera 100 has an inference unit 911. The inference unit 911 estimates the presence or absence of a subject and, if a subject exists, its position, etc., from the image data transferred from the image processing unit 906 to the RAM 902. The configuration and inference processing of the inference unit 911 are generally the same as those of the inference unit 207 in the edge AI device 200 of the first embodiment, so a detailed explanation of them is omitted. Note that the processing of the inference unit 911 may also be handled by the CPU 901.

[0101] Next, the operation of each device in the imaging system of the second embodiment will be explained with reference to Figures 11 to 13. Note that the flowcharts in Figures 11, 12, and 13 correspond to Figures 3, 5, and 6 of the first embodiment, and the processing of each step is generally the same, but the following explanation will focus on the processing that differs from the first embodiment.

[0102] Figures 11(a) and 11(b) are flowcharts showing the setup process for various settings related to shooting for the automatically selected area in the shooting system of the second embodiment. Figure 11(a) shows the operation flowchart of the PTZ camera 900, and Figure 11(b) shows the operation flowchart of the PC 300. In the second embodiment, the PC 300 generates various setting information related to shooting for the automatically selected area based on user operation and transmits it to the PTZ camera 900. The PTZ camera 900 then stores the various setting information related to shooting that it received from the PC 300.

[0103] In the flowchart of Figure 11(b) showing the setup operation of the automatic selection area on the PC300 side, the processes of steps S901 to S904 are generally equivalent to the processes of steps S101 to S104 in Figure 4(a) of the first embodiment, so their explanation is omitted. When the CPU 301 determines that the user has pressed the automatic selection area confirmation button 701 via the operation unit 306, it exits the loop and proceeds to step S905. When the process proceeds to step S905, the CPU 301 of the PC 300 reads coordinate information indicating the automatically selected region from the RAM 302 and transmits it to the PTZ camera 900 via the network I / F 304.

[0104] In step S801 of Figure 11(a), the CPU 901 of the PTZ camera 900 receives coordinate information indicating the automatically selected area sent from the PC 300 via the network I / F 905. Then, as the next step, S802, the CPU 901 writes the coordinate information indicating the received automatically selected region to the RAM 902.

[0105] Figures 12(a) and 12(b) are flowcharts showing the setup process for various settings related to shooting for an overhead composition in the shooting system of the second embodiment. Figure 12(a) shows the operation flowchart of the PTZ camera 900, and Figure 12(b) shows the operation flowchart of the PC 300. In the second embodiment, the PC 300 generates various setting information related to shooting for an overhead composition based on user operation and transmits it to the PTZ camera 900. The PTZ camera 900 then stores the various setting information related to shooting that it received from the PC 300.

[0106] First, we will explain the operation of the PC300 while referring to Figure 12(b). The process in step S1101 is generally equivalent to the process in step S401 described in Figure 5(b) of the first embodiment, so its explanation will be omitted. The loop processing from the next step S1102 to step S1103 is generally the same as the loop processing from step S402 to step S403 in Figure 5(b) of the first embodiment, so its explanation will be omitted. Then, if the CPU 301 determines that the user has pressed the overhead composition determination button 703 via the operation unit 306, it exits the loop processing and proceeds to step S1104.

[0107] When the process proceeds to step S1104, the CPU 301 sends a command (called a memory command) via the network I / F 304 to the PTZ camera 900 instructing it to store the pan, tilt, and zoom values.

[0108] Next, we will explain the operation of the PTZ camera 900 while referring to Figure 12(a). In step S1001, the CPU 901 of the PTZ camera 900 receives a memory command sent from the PC 300 via the network I / F 905. Next, in step S1002, the CPU 901 writes the pan, tilt, and zoom values ​​of its own device at the time it receives the memory instruction command from the PC 300 to the RAM 902 as values ​​for the overhead view.

[0109] Figure 13 is a flowchart showing the tracking operation performed by the PTZ camera 900 after the setup of the automatic selection area and overhead composition described above is completed in the shooting system of the second embodiment. In the shooting system of the second embodiment, the PTZ camera 900 detects the subject position from the captured image and performs automatic tracking by panning, tilting, and zooming according to the subject position. The PTZ camera 900 of the second embodiment also calculates the distance between subjects from the subject position inferred by the inference unit 911 and switches between automatic tracking and overhead composition based on that distance between subjects.

[0110] In the PTZ camera 900 of the second embodiment, as in the first embodiment described above, the captured video footage, which is sequentially captured at a predetermined frame rate, is sequentially stored in the internal RAM 902. The PTZ camera 900 then detects a subject from the captured video footage stored in the RAM 902 and performs loop processing to track that subject. The loop processing from steps S1201 to S1212 in Figure 13 is performed on the captured video footage for each frame.

[0111] First, in step S1201, the CPU 901 of the PTZ camera 900 sequentially reads the captured video stored in the RAM 902 and transfers it to the inference unit 911. Next, in step S1202, the inference unit 911 detects a subject from the captured video read from the RAM 902 and writes the inference result information as the detection result back to the RAM 902. The inference unit 911 of the second embodiment also has a trained model created using machine learning methods such as deep learning, similar to the inference unit 207 of the first embodiment, and acquires captured video as input data and outputs the inference result as output data. The inference result, as described above, includes information such as the location information, type, and probability score of a person such as a player or referee. The location information includes the coordinate information of the four vertices of the rectangular area, as well as information such as the width and height of the rectangular area.

[0112] Next, in step S1203, the CPU 901 reads the coordinate information indicating the automatically selected region that was stored in the RAM 902 in step S802 of Figure 11. Next, in step S1204, the CPU 901 reads the position information of the rectangular area of ​​the subject from the inference results stored in the RAM 902 in step S1202, and counts the number of subjects present in the automatically selected area based on the position information of that rectangular area. The process of counting the number of people present in the automatically selected area is the same as the process described in the first embodiment above.

[0113] Next, in step S1205, the CPU 901 determines whether the number of subjects counted in step S1204 is a predetermined number (3 people in this embodiment). If the CPU 901 determines that the number of subjects counted is 3 people, it proceeds to step S1206. On the other hand, if it determines that the number is not 3 people, it skips the processes from step S1206 to step S1212 and proceeds to the next loop process.

[0114] In the second embodiment, as described in the first embodiment, if the number of subjects becomes less than three in step S1205 after tracking is started due to the determination that there are three subjects, the CPU 901 may fix the pan, tilt, and zoom values. Subsequently, when two players return to the automatic selection area and the CPU 901 determines in step S1205 that there are three subjects in the automatic selection area, the process proceeds to step S1206, and control of the PTZ camera 100 is resumed.

[0115] When the process proceeds to step S1206, the CPU 901 obtains the longest distance between subjects included in the automatically selected area and determines whether this longest distance is greater than or equal to a predetermined distance. The predetermined distance is the same distance threshold as described in the first embodiment. Then, in step S1206, if the longest distance between subjects is less than the predetermined distance, the CPU 901 proceeds to step S1207.

[0116] In step S1207, the CPU 901 determines the three subjects detected within the automatically selected area to be tracked and calculates the centroid positions of those three subjects, as described in the first embodiment. Next, in step S1208, the CPU 901 determines whether the centroid position calculated in step S1207 matches the center position of the field of view in the captured video. If the CPU 901 determines that the centroid position matches the center position of the field of view, it skips the subsequent processing and proceeds to the next loop. On the other hand, if the CPU 901 determines that the centroid position does not match the center position of the field of view, it proceeds to step S1209.

[0117] In step S1209, the CPU 901 calculates the difference between the center of gravity calculated in step S1207 and the center of the field of view on the captured image, and calculates the amount of pan and tilt adjustment corresponding to that difference. Also in step 1209, the CPU 901 calculates the amount of zoom adjustment so as to keep the size of the rectangular area of ​​the subject constant. As described in the first embodiment, zoom adjustment may be performed based on the size of a person's organs, such as the size of their face. The size of the rectangular area of ​​the subject may be the size of the rectangular area of ​​a subject randomly selected from the automatically selected area, or it may be the average size of each rectangular area. Alternatively, the amount of zoom adjustment may be calculated so as to keep the size of the bounding rectangular area encompassing three subjects constant.

[0118] In the next step, S1210, the CPU 901 calculates the drive value corresponding to the adjustment amount in the pan and tilt directions, and also calculates the lens drive direction and drive amount values ​​corresponding to the adjustment amount in the zoom direction. Next, in step S1211, the CPU 901 derives (calculates) drive parameters for pan, tilt, and zoom operation based on the values ​​calculated in step S1210.

[0119] Then, in step S1212, the CPU 901 controls the drive unit 909 via the drive I / F 908 based on the drive parameters derived in step S1211. As a result, the drive unit 909 drives based on these drive parameters, causing the PTZ camera 900 to change its shooting direction (pan and tilt operation) and also change its field of view. After step S1212, the CPU 901 returns processing to step S1201, which is the beginning of the loop processing.

[0120] Furthermore, if it is determined in step S1206 that the longest distance between subjects is greater than or equal to a predetermined distance and the process proceeds to step S1213, the CPU 901 reads the pan, tilt, and zoom values ​​corresponding to the overhead composition written in step S1202 from the RAM 902. The CPU 901 then determines these pan, tilt, and zoom values ​​as the tracking target position. In other words, by determining the pan, tilt, and zoom values ​​written to the RAM 902 in step S1202 as the tracking target position, the composition of the PTZ camera 100 is switched to an overhead composition.

[0121] Next, in step S1214, the CPU 901 derives drive parameters for panning and tilting in a desired direction and at a desired speed, and drive parameters for adjusting the field of view, from the pan, tilt, and zoom values ​​indicating the overhead view read in step S1213.

[0122] Then, in step S1215, the CPU 901 controls the drive unit 909 via the drive I / F 908 based on the drive parameters derived in step S1214. As a result, the drive unit 909 drives based on these parameters, causing the PTZ camera 900 to change its shooting direction and perform a zoom operation. After step S1215, the CPU 901 returns processing to step S1201, which is the beginning of the loop processing.

[0123] As described above, in the second embodiment of the imaging system, by providing an inference unit 911 inside the PTZ camera 900, it becomes possible to achieve PTZ camera control similar to that of the first embodiment without using an edge AI device 200 as described in the first embodiment.

[0124] <Third Embodiment> In the first embodiment, an example was described in which tracking operation and overhead composition switching are performed based on the longest distance between subjects to determine whether it is the start or end of the competition or during the competition. In the third embodiment, an example is described in which not only the longest distance between subjects but also the shortest distance between subjects is considered as the distance between subjects, thereby enabling more accurate determination of whether it is the start or end of the competition or during the competition.

[0125] For example, in some competitive sports, the referee may be separated from the competition even during the match. In such competitive sports, the longest distance between subjects may exceed a predetermined distance, and in this case, the first embodiment described above would switch from automatic tracking to an overhead view. Therefore, in the third embodiment, by considering not only the longest distance between subjects but also the shortest distance between subjects, it becomes possible to more accurately determine whether the competition is in progress, or at the start or end of the competition. The configuration of the shooting system in the third embodiment is the same as in Figure 1, and the internal configuration of each device in the shooting system is the same as in Figure 2, so a detailed explanation of them is omitted. Also, the setup operation of each device is the same as in the first embodiment, so a description of that is also omitted. The following description will focus on the differences from the first embodiment.

[0126] In the third embodiment of the shooting system, the processing in the flowchart of Figure 6 is generally the same as described above, but the processing in step S606 differs from that of the first embodiment. In the third embodiment, in step S606, the CPU 201 of the edge AI device 200 calculates not only the longest distance between subjects but also the shortest distance between subjects. Then, based on these longest and shortest distances between subjects, the CPU 201 determines whether to perform automatic tracking or switch to an overhead composition.

[0127] The following describes the process performed in step S606 of Figure 6 in the third embodiment. In the third embodiment, in step S606, the CPU 201 determines whether the shortest distance between subjects is less than the predetermined distance if the longest distance between subjects is greater than or equal to a predetermined distance.

[0128] Figure 14 is a diagram used to explain the longest and shortest distances between subjects. Similar to the example in Figure 8 described above, Figure 14 shows an example of the positional relationship between two athletes 600a and 600b and one referee 601 during a competition. In the example in Figure 14, distance 800c is shown as the longest distance between subjects and distance 801 is shown as the shortest distance between subjects, based on the distances between the two athletes 600a and 600b and the one referee 601. In the first embodiment described above, the determination of whether it is the start, end, or in progress of the competition is made based only on the longest distance between subjects 800b illustrated in Figure 8(b). In contrast, in the third embodiment, as shown in Figure 14, in addition to the longest distance between subjects 800c, the shortest distance between subjects 801 is also used as an indicator for determination.

[0129] In the third embodiment, the CPU 201 calculates the longest subject-to-subject distance 800c and the shortest subject-to-subject distance 801. The CPU 201 then determines whether the longest subject-to-subject distance is greater than or equal to a predetermined distance, and if so, it further determines whether the shortest subject-to-subject distance is greater than or equal to the predetermined distance. Here, the predetermined distance used as a comparison target with the subject-to-subject distance can be the same distance threshold as described in the first embodiment described above.

[0130] In step S606, CPU201 determines whether the longest distance between subjects 800c is greater than or equal to a predetermined distance. If the longest distance between subjects 800c is not greater than or equal to the predetermined distance (i.e., less than the predetermined distance), it proceeds to step S607. If the longest distance between subjects 800c is greater than or equal to the predetermined distance, CPU201 further determines whether the shortest distance between subjects 801 is also greater than or equal to the predetermined distance. If the longest distance between subjects 800c is greater than or equal to the predetermined distance, but the shortest distance between subjects 801 is less than the predetermined distance, CPU201 proceeds to step S607. In other words, if the shortest distance between subjects 801 is less than the predetermined distance, it is considered that the athletes are grappling with each other during a competition, etc., so CPU201 proceeds to the automatic tracking process from step S607 onwards. On the other hand, if the longest distance between subjects 800c is greater than or equal to a predetermined distance, and the shortest distance between subjects 801 is also greater than or equal to a predetermined distance, the CPU 201 proceeds to step S612 or later to switch to an overhead composition.

[0131] As explained above, according to the third embodiment, it is possible to more accurately determine whether the competition is in progress, or whether it is the start or end of the competition. In the third embodiment, an example was given in which the edge AI device 200 determines the distance between subjects, similar to the example in the first embodiment. However, the third embodiment is also applicable when the PTZ camera 900 determines the distance between subjects, as in the second embodiment. Furthermore, while the first to third embodiments described above provided examples of switching between tracking operation and overhead composition control depending on the distance between subjects, the method can also be applied to switching between control other than tracking operation and overhead composition, or switching between control other than tracking operation and composition other than overhead composition. In addition, while the first to third embodiments provided examples of switching between two controls, tracking operation and overhead composition, depending on the distance between subjects, the method can also be applied to cases where three or more controls are switched depending on the distance between subjects.

[0132] The present invention can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the invention should not be limited by them. That is, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0133] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Composition 1) A detection means for detecting a subject from a captured image, Distance acquisition means for acquiring the distance between the detected plurality of subjects, A control means for switching between a first control for the imaging device that captures the aforementioned image and a second control for the imaging device that is different from the first control, according to the distance between the acquired subjects, It has, The first control is a control that tracks the subject using the imaging device, The second control is a control that sets the composition to be photographed by the photographing device to a predetermined composition, and is characterized in that the photographing control device is characterized in that the second control is a control that sets the composition to be photographed by the photographing device to a predetermined composition. (Configuration 2) The shooting control device according to configuration 1, characterized in that the control means switches to the first control when the longest distance among the distances between the subjects is less than a predetermined distance. (Composition 3) The shooting control device according to configuration 1 or 2, characterized in that the control means switches to the second control when the longest distance among the distances between the subjects is greater than or equal to a predetermined distance. (Composition 4) The shooting control device according to configuration 1 or 2, characterized in that the control means switches to the first control when the longest distance among the distances between the subjects is greater than or equal to a predetermined distance, or when the shortest distance among the distances between the subjects is less than the predetermined distance. (Composition 5) The shooting control device according to configuration 1 or 2, characterized in that the control means switches to the second control when the longest distance among the distances between the subjects is greater than or equal to the predetermined distance, and the shortest distance among the distances between the subjects is also greater than or equal to the predetermined distance. (Composition 6) The system includes a determination means for determining whether the subject detected by the detection means is included in a predetermined target area of ​​the captured image, The shooting control device according to any one of configurations 1 to 5, characterized in that the distance acquisition means acquires the distance between the subjects included in the predetermined target area. (Composition 7) The shooting control device according to configuration 6, characterized in that the distance acquisition means acquires the distance between the predetermined number of subjects included in the predetermined target area when the number of subjects included in the predetermined target area is a predetermined number. (Composition 8) The photographing control device according to configuration 6, characterized in that when the control means sets the control of the photographing device to the first control, the number of subjects included in the predetermined target area is a predetermined number. (Composition 9) The shooting control device according to any one of configurations 6 to 8, characterized in that the control means stops the first control when the number of subjects included in the predetermined target area falls below a predetermined number after the first control has been started. (Composition 10) The shooting control device according to any one of configurations 6 to 9, characterized in that, after starting the first control, if the number of subjects included in the predetermined target area falls below a predetermined number, the shooting direction and field of view of the shooting device are fixed to the shooting direction and field of view at the time when the number of subjects included in the predetermined target area falls below a predetermined number. (Composition 11) The imaging control device according to any one of the configurations 7 to 10, characterized in that the predetermined number is 3. (Composition 12) The detection means detects the subject as a rectangular area, The shooting control device according to any one of configurations 6 to 11, characterized in that the determination means determines that a subject whose lower edge of the detected rectangular area is located within the predetermined target area is a subject included in the predetermined target area. (Composition 13) A shooting control device according to any one of the configurations 6 to 12, characterized by having setting means for setting a predetermined target area for the captured image. (Composition 14) The shooting control device according to configuration 13, characterized in that the setting means sets the predetermined target area based on instructions from the user. (Composition 15) The shooting control device according to configuration 13, characterized in that the setting means detects a specific area from the captured image and sets the detected specific area as the predetermined target area. (Composition 16) The camera has a conversion means that converts at least one of the pan value, tilt value, and zoom value of the camera into polar coordinates with the position of the camera as the origin. The shooting control device according to any one of configurations 6 to 15, characterized in that the determination means determines whether or not the subject detected by the detection means is included in the predetermined target area expressed in polar coordinates. (Composition 17) The photographic control device according to any one of configurations 1 to 16, characterized in that the control means performs the first control on the photographic device based on the centers of gravity of two or more subjects. (Composition 18) The photographing control device according to any one of configurations 1 to 17, characterized in that the subject is a person. (Composition 19) The detection means detects one or more parts of the subject, The shooting control device according to any one of configurations 6 to 11, characterized in that the determination means determines that a subject in which at least one of the detected parts is located within the predetermined target area is a subject included in the predetermined target area. (Method 1) A detection process for detecting a subject from a captured image, A distance acquisition step of acquiring the distance between the detected plurality of subjects, A control step of switching between a first control for the imaging device that captures the aforementioned image and a second control for the imaging device that is different from the first control, according to the distance between the acquired subjects, It has, The first control is a control that tracks the subject using the imaging device, The second control is a control that sets the composition to be photographed by the photographing device to a predetermined composition, and is characterized by this photographing control method. (Program 1) A program that causes a computer to function as a shooting control device described in any one of configurations 1 to 19. [Explanation of symbols]

[0134] 100: PTZ camera, 200: Edge AI device, 300: PC, 400: Network

Claims

1. A detection means for detecting a subject from a captured image, Distance acquisition means for acquiring the distance between multiple detected subjects, A control means for switching between a first control for the imaging device that captures the aforementioned image and a second control for the imaging device that is different from the first control, according to the distance between the subjects, It has, The first control is a control that sets the shooting direction of the shooting device based on the position of the subject so that the shooting device tracks the subject, The second control is a control that sets the shooting direction of the shooting device to a predetermined shooting direction so that the composition captured by the shooting device is a predetermined composition.

2. The shooting control device according to claim 1, characterized in that the control means switches to the first control when the longest distance among the distances between subjects is less than a predetermined distance.

3. The shooting control device according to claim 1 or 2, characterized in that the control means switches to the second control when the longest distance among the distances between subjects is greater than or equal to a predetermined distance.

4. The shooting control device according to claim 1, wherein the control means switches to the first control when the longest distance among the distances between subjects is greater than or equal to a predetermined distance, or when the shortest distance among the distances between subjects is less than the predetermined distance.

5. The shooting control device according to claim 1, characterized in that the control means switches to the second control when the longest distance among the distances between subjects is greater than or equal to a predetermined distance, and the shortest distance among the distances between subjects is also greater than or equal to the predetermined distance.

6. The system includes a determination means for determining whether the subject detected by the detection means is included in a predetermined target area of ​​the captured image, The shooting control device according to claim 1, characterized in that the distance acquisition means acquires the distance between subjects included in the predetermined target area.

7. The shooting control device according to claim 6, characterized in that the distance acquisition means acquires the distance between the predetermined number of subjects included in the predetermined target area when the number of subjects included in the predetermined target area is a predetermined number.

8. The photographing control device according to claim 6, characterized in that when the control means sets the control of the photographing device to the first control, the number of subjects included in the predetermined target area is a predetermined number.

9. The shooting control device according to claim 6, characterized in that the control means stops the first control if, after starting the first control, the number of subjects included in the predetermined target area falls below a predetermined number.

10. The shooting control device according to claim 6, characterized in that, after the first control is started, if the number of subjects included in the predetermined target area falls below a predetermined number, the shooting direction and field of view of the shooting device are fixed to the shooting direction and field of view at the time when the number of subjects included in the predetermined target area falls below a predetermined number.

11. The imaging control device according to any one of claims 7 to 10, characterized in that the predetermined number is 3.

12. The detection means detects the subject as a rectangular area, The shooting control device according to claim 6, characterized in that the determination means determines that a subject whose lower edge of the detected rectangular area is located within the predetermined target area is a subject included in the predetermined target area.

13. The shooting control device according to claim 6, further comprising setting means for setting a predetermined target area for the captured image.

14. The shooting control device according to claim 13, characterized in that the setting means sets the predetermined target area based on instructions from the user.

15. The shooting control device according to claim 13, characterized in that the setting means detects a specific area from the captured image and sets the detected specific area as the predetermined target area.

16. The camera has a conversion means that converts at least one of the pan value, tilt value, and zoom value of the camera into polar coordinates with the position of the camera as the origin. The shooting control device according to claim 6, characterized in that the determination means determines whether or not the subject detected by the detection means is included in the predetermined target area expressed in polar coordinates.

17. The photographic control device according to claim 1, characterized in that the control means performs the first control on the photographic device based on the centers of gravity of two or more subjects.

18. The photographic control device according to claim 1, characterized in that the subject is a person.

19. The detection means detects one or more parts of the subject, The shooting control device according to claim 6, characterized in that the determination means determines that a subject in which at least one of the detected parts is located within the predetermined target area is a subject included in the predetermined target area.

20. A detection process for detecting a subject from a captured image, A distance acquisition step to acquire the inter-subject distance between a plurality of detected subjects, A control step of switching between a first control for the imaging device that captures the aforementioned image and a second control for the imaging device that is different from the first control, according to the distance between the subjects, It has, The first control is a control that sets the shooting direction of the shooting device based on the position of the subject so that the shooting device tracks the subject, The second control is a method of controlling a photograph, characterized in that it sets the shooting direction of the photographing device to a predetermined shooting direction so that the composition photographed by the photographing device is a predetermined composition.

21. Computers, A detection means for detecting a subject from a captured image, Distance acquisition means for acquiring the distance between multiple detected subjects, A control means for switching between a first control for the imaging device that captures the aforementioned image and a second control for the imaging device that is different from the first control, according to the distance between the subjects, It has, The first control is a control that sets the shooting direction of the shooting device based on the position of the subject so that the shooting device tracks the subject, The second control is a program that functions as a shooting control device, which controls the shooting direction of the shooting device to a predetermined shooting direction so that the composition captured by the shooting device is a predetermined composition.

Images