Imaging system, imaging device, information processing device, control method for imaging system, and program

Abstract

This imaging system provides an imaging mechanism that automatically sets a region to hold the tracking target in an area that avoids the overlapping position of the superimposed images. [Solution] An imaging system comprising at least one imaging device having a subject tracking function and an information processing device that is communicatively connected to the imaging device, comprising: a first setting means for setting the position and size of a second image different from the first image which is displayed together with a first image acquired from the imaging device; and a second setting means for setting a tracking hold area, which is an area in which the imaging device is controlled to keep the subject to be tracked in the first image within the tracking hold area, according to the display position of the second image set by the first setting means.

Description

Technical Field

[0001] The present invention relates to an imaging system, an imaging device, an information processing device, a control method for an imaging system, and a program.

Background Art

[0002] Conventionally, by using image analysis technology in network cameras, pan-tilt cameras, video production cameras, etc., automatic tracking drive that automatically drives pan, tilt, and zoom in accordance with the movement of the detected subject so as to keep the subject within the angle of view has been used.

[0003] In Patent Document 1, a technique is disclosed in which, when performing automatic tracking drive, the position (area) in the angle of view of the camera where the subject is to be held is set in advance, and driving is performed so that the subject is held in the set area (hereinafter, tracking target holding area).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0007] Therefore, the object of the present invention is to provide an imaging system that can automatically set a region for holding the tracking target in an area that avoids the superimposed position of the images when images are superimposed. [Means for solving the problem]

[0008] To achieve the above objective, an imaging system as one aspect of the present invention is an imaging system comprising at least one imaging device having a subject tracking function and an information processing device that is communicably connected to the imaging device, characterized in that it includes a first setting means for setting the position and size of a second image different from the first image which is displayed together with a first image acquired from the imaging device, and a second setting means for setting the tracking holding region, which is a region in which the imaging device is controlled to keep the subject to be tracked in the first image within the tracking holding region, according to the display position of the second image set by the first setting means. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an imaging system that can automatically set a region for holding the tracking target in an area that avoids the overlapping position of the images when images are superimposed. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating the configuration of the imaging system of Embodiment 1. [Figure 2] This is a flowchart illustrating the process for setting the tracking target holding area in Embodiment 1. [Figure 3] This is a diagram illustrating the process for setting the tracking target holding area in Embodiment 1. [Figure 4]This figure illustrates variations in the position of the tracking target holding area in Embodiment 1. [Figure 5] This is a diagram illustrating the types of tracking target holding regions in Embodiment 1. [Figure 6] This is a diagram illustrating the configuration of the imaging system in Embodiment 2. [Figure 7] This is a flowchart illustrating the display process of the UI for setting the tracking target holding area in Embodiment 2. [Figure 8] This diagram illustrates the display process of the UI for setting the tracking target holding area in Embodiment 2. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0012] <Embodiment 1> The configuration of the imaging system 10 of Embodiment 1 will be described below with reference to Figure 1. Figure 1 is a diagram illustrating the configuration of the imaging system 10 of Embodiment 1. The imaging system 10 of this embodiment consists of an imaging device 100 having a tracking function for tracking a subject to be tracked and an information processing device 110. In the imaging system 10 of this embodiment, the imaging device 100 and the information processing device 110 are connected to communicate via an IP network 120. First, the configuration of the imaging device 100 will be described below.

[0013] The imaging device 100 of this embodiment has a functional configuration consisting of an imaging unit 101, an image processing unit 102, a control unit 103, a communication unit 104, a storage unit 105, a drive unit 106, and a tracking control unit 107.

[0014] The imaging unit 101 consists of an imaging optical system comprising a zoom lens, a focus lens, an aperture mechanism, and an image sensor such as a CMOS sensor or CCD sensor that converts the subject image (optical image) formed through the imaging optical system into an electrical signal. The imaging optical system may be configured to allow adjustment of the zoom magnification and focus position by a lens drive motor, and may also be configured to allow insertion and removal of filters that transmit or attenuate specific wavelengths. The image sensor is sensitive to light in the visible light region, but is not limited to this, and may be sensitive to non-visible light regions such as infrared light.

[0015] The image processing unit 102 is composed of a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., and performs various signal processing on electrical signals input from the imaging unit 101. For the sake of explanation, the image processing unit 102 and the control unit 103, which will be described later, are separated here, but they may be composed of the same device. In other words, the control unit 103 may perform the processing that the image processing unit 102 would perform. In that case, the image processing unit 102 can be a functional unit of the control unit 103.

[0016] The control unit 103 is composed of a CPU, ASIC, FPGA, etc., and converts the data input from the image processing unit 102 into data that can be output to the IP network 120 via the communication unit 104 and outputs it. The control unit 103 also changes the imaging conditions by controlling the image processing unit 102 and the imaging unit 101 in response to operation instructions from the information processing unit 110. The control unit 103 also detects the position of the subject from the image acquired by the imaging unit 101 or from various sensors (not shown). Furthermore, based on the image superposition position information received from the information processing unit 110, the control unit 103 sets a tracking target holding area, which is an area that holds the tracking target, at a position that avoids the superimposed image (a position that does not overlap with the superimposed image).

[0017] The communication unit 104 includes a network distribution engine and peripheral devices such as a RAM and an Ethernet PHY module. Then, the communication unit 104 converts the data obtained from the control unit 103 into a format that can be distributed to the IP network 120, and outputs the converted data to the IP network 120. Also, the communication unit 104 communicates data with the information processing device 110 via the IP network 120.

[0018] The storage unit 105 is composed of memory devices such as a Flash ROM and an EEPROM, and stores various setting information of the imaging device 100. The control unit 103 reads and rewrites the information stored in the storage unit 105 as necessary.

[0019] The drive unit 106 is composed of a motor or the like, and performs a pan operation or a tilt operation according to a control signal from the control unit 103 or the following tracking control unit 107, and changes the imaging area of the imaging device 100. In other words, by driving the drive unit 106, the imaging direction of the imaging device 100 can be changed.

[0020] The tracking control unit 107 controls the drive unit 106 so as to hold the subject detected by the control unit 103 within the angle of view or within the tracking target holding area set by the control unit 103, and changes the imaging area of the imaging device 100. Note that the detected subject is the subject to be tracked. Here, for the sake of explanation, the control unit 103 and the tracking control unit 107 are described separately, but they may be configured by the same device. In other words, the control unit 103 may execute the processing performed by the tracking control unit 107. In that case, the tracking control unit 107 can be a functional unit of the control unit 103.

[0021] Each functional unit such as the above-described imaging unit 101, image processing unit 102, and tracking control unit 107 is realized by the control unit 103 expanding the program stored in the ROM into the RAM and executing it. Then, the control unit 103 holds the execution results of each process described below in the storage unit 105, the RAM, a storage medium not shown, and the like.

[0022] Next, the configuration of the information processing device 110 will be described below. The information processing device 110 consists of hardware such as a general-purpose PC or computer including a processor (CPU or MPU), memory (RAM, ROM), HDD, and communication interface (not shown), as well as input devices such as a mouse and keyboard, and output devices such as a display. The computer may also be a switching device such as a switcher. The processor of the information processing device 110 executes programs stored in the memory and HDD, thereby realizing each process in the information processing device 110.

[0023] The processor (CPU or MPU) is a central processing unit that provides overall control of the information processing unit 110. RAM temporarily stores computer programs executed by the processor. RAM provides a work area used by the processor when executing processing. RAM can also function as frame memory or buffer memory. ROM stores programs used by the processor to control the information processing unit 110. HDD is a storage device (secondary storage device) that records image data, video data, etc. The communication interface communicates with external devices such as the imaging device 100 via the IP network 120. Interface 440 can perform communication according to TCP / IP and HTTP, etc.

[0024] The information processing device 110 of this embodiment consists of a system control unit 111, an operation display unit 112, a storage unit 113, a communication unit 114, an overlay position setting unit 115, and an image overlay unit 116.

[0025] The system control unit 111 processes the display of images output from the imaging device 100 (hereinafter referred to as camera images) on the operation display unit 112 and stores them in the storage unit 113. The system control unit 111 also controls the superimposition position setting unit 115 and the image superimposition unit 116 (described later) to generate a distribution image by superimposing an image or video different from the camera image stored in the storage unit 113 onto the camera image. The system control unit 111 also controls the display of a UI (user interface) on the operation display unit 112 that accepts control information from the user regarding imaging control and image superimposition processing for the imaging device 100. When controlling the display of such a UI (making it operable or modifiable on the operation display unit 112), the system control unit 111 also functions as a display control means. The control information for the imaging device 100 input to the UI is transmitted to the imaging device 100 via the communication unit 114 by the system control unit 111. The user can read and rewrite data such as images stored in the storage unit 113 by specifying it via the operation display unit 112.

[0026] The operation display unit 112 displays camera images, video, or UI output from the system control unit 111. The operation display unit 112 may be composed of, for example, a display. The storage unit 113 stores (stores) camera image and video data output from the system control unit 111. The communication unit 114 includes a network distribution engine and peripheral devices such as RAM and an Ethernet PHY module. The communication unit 114 converts the data acquired from the system control unit 111 into a format that can be distributed to the IP network 120 and outputs the converted data to the IP network 120. The communication unit 114 also communicates data with the imaging device 100 via the IP network 120.

[0027] The superimposition position setting unit 115 converts the position of the image to be superimposed on the camera image set from the operation display unit 112 into coordinate information (hereinafter referred to as image superimposition position information), and sets the position of the superimposed image in the image superimposition unit 116. The superimposition position setting unit 115 outputs the converted image superimposition position information to the image superimposition unit 116. The superimposition position setting unit 115 also sets (determines) the tracking target holding area in the imaging device 100 from the image superimposition position information and transmits (outputs) it to the imaging device 100 via the system control unit 111 and the communication unit 114. The superimposition position setting unit 115 also functions as an output means when transmitting (outputting) the image superimposition position information to the imaging device 100. It is desirable that the tracking target holding area is an area within the field of view of the imaging device 100. In other words, it is desirable that the tracking target holding area is an area in the captured image captured by the imaging device 100. Here, for explanatory purposes, the superimposed position setting unit 115 and the system control unit 111 are described separately, but they may be configured as the same device. In other words, the system control unit 111 may perform the processing that the superimposed position setting unit 115 would normally perform. In that case, the superimposed position setting unit 115 can be a functional unit of the system control unit 111.

[0028] The image superposition unit 116 superimposes a specified image or video onto the camera image according to the image superposition position information input from the superposition position setting unit 115. When superimposing a specified image or video onto the camera image, the image superposition unit 116 also functions as an image superposition means (image display control means). Here, for explanatory purposes, the image superposition unit 116 and the system control unit 111 are described separately, but they may be configured as the same device. In other words, the system control unit 111 may execute the processing performed by the image superposition unit 116. In that case, the image superposition unit 116 can be a functional unit of the system control unit 111.

[0029] Each of the above-mentioned functional units, such as the superimposition position setting unit 115 and the image superimposition unit 116, is realized by the system control unit 111 loading the program stored in ROM into RAM and executing it. The system control unit 111 then stores the execution results of each process, which will be described later, in the storage unit 113, RAM, or a storage medium not shown.

[0030] Next, using Figures 2 and 3, we will explain the setting process flow for the tracking target holding area, which is the area that holds the tracking target when an image different from the camera image is superimposed on the camera image, as performed by the imaging system 10 of this embodiment.

[0031] Figure 2 is a flowchart showing the process of setting the tracking target holding area of ​​the imaging system 10. Figure 3 is a diagram for explaining the process of setting the tracking target holding area. Figure 3(A) is a diagram illustrating the camera image 301 displayed on the operation display unit 112 by the user. Figure 3(B) is a diagram illustrating the case where an image to be superimposed on the camera image 301 (superimposed image 302) and the position in which the superimposed image 302 is placed are set. Figure 3(C) is a diagram illustrating the process of setting the tracking target holding area 303 in a position that avoids the superimposed image 302. As shown in Figure 3, the superimposed image 302 is superimposed on the camera image 301 and displayed together with the camera image 301 on the operation display unit 112.

[0032] In the following, for illustrative purposes, an example is shown in which a superimposed image, which is a different image from the camera image, is superimposed on a full HD (1920x1080) camera image, but this is not the only example. Also, before the start of processing in Figure 2, the system control unit 111 displays a UI on the operation display unit 112 that accepts control information from the user regarding imaging control to the imaging device 100 and image superimposition processing. Note that the notation of each process (step) is omitted by adding an S at the beginning of each step.

[0033] In S201, the system control unit 111 determines whether the camera image (first image) 301 has been selected. If it is determined that the camera image has been selected, the process proceeds to S202. On the other hand, if it is determined that the camera image has not been selected, the system waits until the camera image 301 is selected. The camera image 301 is selected by the user operating the UI displayed on the operation display unit 112. That is, the system control unit 111 determines that the camera image 301 has been selected by detecting that the user has operated the UI displayed on the operation display unit 112 to select the camera image 301. For example, in S201, the user operates the UI to select the camera image 301 as shown in Figure 3(A). If the user wishes to change the field of view of the imaging device 100, they can change the field of view of the imaging device 100 (the imaging direction of the imaging device can be changed) by operating the drive unit 106 using a UI that is not shown.

[0034] In S202, the system control unit 111 determines whether the position for placing the superimposed image (second image) 302 superimposed on the camera image 301 has been set. If it is determined that the position for placing the superimposed image 302 on the camera image 301 has been set, the process proceeds to S203. On the other hand, if it is determined that the position for placing the superimposed image 302 on the camera image 301 has not been set, the process waits until the position for placing the superimposed image 302 on the camera image 301 has been set. The position of the superimposed image 302 is set by the user operating the UI displayed on the operation display unit 112 to set the position for placing the superimposed image 302 superimposed on the camera image 301. For example, in S202, the placement position of the superimposed image 302 relative to the camera image 301 is set as shown in Figure 3(B). Then, the superimposed position setting unit 115 converts the position of the superimposed image 302 superimposed on the camera image (superimposed position) set from the operation display unit 112 into image superimposed position information. The converted information (image superposition position information) is output to the image superposition unit 116 by, for example, the system control unit 111. The image superposition unit 116 starts the superposition process of the superimposed image 302 onto the camera image 301 based on the outputted image superposition position information. The timing of when the image superposition unit 116 starts the superposition process of the superimposed image 302 onto the camera image 301 may be any timing before the start of S206.

[0035] In this embodiment, for illustrative purposes, the size of the superimposed image 302 is set to 1400x800, and its placement positions are set to the start point (51, 141) and end point (1450, 940). However, the user is not limited to this and can freely select the size of the superimposed image 302 by operating a setting UI (not shown). The UI displayed on the operation display unit 112, as described above, also functions as a first setting means that allows the user to set the position and size of the superimposed image (second image). The superimposed image (second image) is an image superimposed on the camera image (first image) acquired from the imaging device 100, and is described as an image different from the camera image.

[0036] In S203, the system control unit 111 transmits the image superposition position information of the superimposed image 302, which was set in S202, to the imaging device 100 via the communication unit 114 and the IP network 120. During processing, the communication unit 114 of the information processing device 110 also functions as a transmission means.

[0037] In S204, the control unit 103 receives the image superimposition position information (display position information) transmitted from the information processing device 110 in S203 via the IP network 120 and the communication unit 104. During this process, the communication unit 104 of the imaging device 100 also functions as a receiving means.

[0038] In S205, the control unit 103 sets a tracking target holding area (tracking holding area) 303, which is the area that holds the subject to be tracked, at a position that avoids the superimposed image 302, based on the image superimposed position information received in S204. In other words, the control unit 103 sets the tracking target holding area 303 in an area of ​​the camera image 301 that does not overlap with the superimposed image 302, based on the image superimposed position information received from the information processing device 110. For example, in S205, the control unit 103 sets the tracking target holding area 303 at the position of the camera image 301 (within the camera image) as shown in Figure 3(C). When setting the tracking target holding area, the control unit 103 also functions as a second setting means.

[0039] Here, the starting point (1471, 141) and ending point (1900, 940) are set to avoid the superimposed image 302. Subsequently, the control unit 103 transmits the set position information of the tracking target holding area 303 to the tracking control unit 107.

[0040] In S206, the tracking control unit 107 controls the drive unit 106 of the imaging device 100 to start tracking the subject based on the position information of the tracking target holding area 303 transmitted in S205. During this process, the tracking control unit 107 also functions as a tracking control means. Here, the tracking target holding area 303 is set to start at a position 20 pixels away in the x direction from the right edge of the superimposed image 302, and the end point is similarly set to a position 20 pixels away from a position outside the field of view. However, it is not limited to this, and the end point may be set to a position further away from a position outside the field of view in order to reduce the possibility of the subject going outside the field of view.

[0041] As described above, the imaging device 100 of this embodiment receives superposition position information, which is information about the position where a superimposed image 302 is superimposed on the camera image 301, as set by the processing on the information processing device 110, via the communication unit 104. Subsequently, based on the received superposition position information, the control unit 103 sets a tracking target holding area 303 for holding the subject to be tracked within the camera image 301. Then, based on the position information of the tracking target holding area, the control unit 106 is controlled to start tracking the subject to be tracked.

[0042] Furthermore, in the process shown in Figure 2, the information processing device 110 transmits image superposition position information to the imaging device 100, and the imaging device 100 sets the tracking target holding area 303, but this is not limited to this. For example, the information processing device 110 may also set the tracking target holding area 303. In this case, the system control unit 111 does not transmit the image superposition position information to the imaging device 100 in S203. Then, based on the image superposition position information set in S202, the system control unit 111 sets the tracking target holding area 303 to a position that avoids the superimposed image 302 superimposed on the camera image 301 (a position where they do not overlap). After that, the position information of the tracking target holding area 303 set on the information processing device 110 side is transmitted to the imaging device 100. At this time, the system control unit 111 may also transmit the image superposition position information converted in S202 to the image superposition unit 116 and start the superposition process of the superimposed image 302 on the camera image 301. Then, the imaging device 100 controls the drive unit 106 based on the position information of the tracking target holding area 303 received, similar to S206, to start tracking the subject to be tracked. With this processing, S203 and S204 become unnecessary.

[0043] Next, referring to Figure 4, we will explain the variations in the arrangement of the tracking target holding area, which is automatically set by the control unit 103 (or system control unit 111) according to the position of the superimposed image superimposed on the camera image. Figure 4 is a diagram illustrating the variations in the position of the tracking target holding area in Embodiment 1. Figures 4(A) to 4(D) illustrate how the tracking target protection area is arranged according to the position and size of the superimposed image superimposed on the camera image. Hereinafter, the arrangement setting process of the tracking target holding area shown in Figures 4(A) to (D) is performed by the control unit 103 (or system control unit 111) in the same manner as described above.

[0044] As shown in Figure 4(A), if the superimposed image 402 is positioned in the center of the camera image 401 and there is space on both the left and right sides, the tracking target holding area 403 is set to be positioned on the left side (left edge) of the camera image 401 so as not to overlap with the superimposed image 402. Furthermore, the tracking target holding area 404 is set to be positioned on the right side (right edge) of the camera image 401 so as not to overlap with the superimposed image 402. In this embodiment, when the superimposed image is positioned as shown in Figure 4(A), the tracking target holding area can be divided into two areas (multiple areas) and positioned on the camera image so as not to overlap with the superimposed image. In other words, the control unit 103 (or system control unit 111) of this embodiment can set multiple tracking target holding areas depending on the position in which the superimposed image is superimposed on the camera image.

[0045] As shown in Figure 4(B), when superimposed image 405 is positioned at the left edge of the camera image and superimposed image 406 is positioned at two locations at the right edge of camera image 401, the tracking target holding area 407 is set to be positioned near the center of camera image 401. In this case, the tracking target holding area 407 is set to be positioned so as not to overlap with superimposed images 405 and 406. Although the superimposed images are positioned at the left and right edges in this example, even if they are positioned elsewhere or if the number of superimposed images increases, the tracking target holding area is set to be positioned in a location that avoids all superimposed images (a position that does not overlap).

[0046] As shown in Figure 4(C), when the superimposed image 408 is placed above the camera image 401, the tracking target holding area 409 is set to avoid the superimposed image 408 and its lower part, taking into consideration the possibility of it overlapping with the subject's face. In this example, a rectangular tracking target holding area 409 is set to avoid the superimposed image 408 and its lower part. However, this is not the only option; if the size of the tracking target fits below the superimposed image 408, the tracking target holding area may be set as a polygon that avoids only the superimposed image 408. This allows the tracking target holding area 409 to be set larger when the size of the tracking target is small. In other words, the control unit 103 (or system control unit 111) of this embodiment can change and set the shape of the tracking target holding area according to the position where the superimposed image is superimposed on the camera image.

[0047] The examples in Figures 4(A) to 4(C) above show an example where the tracking target holding area is set as large as possible while avoiding the superimposed image on the camera image 401. However, it is not limited to this, and as shown in Figure 4(D), the tracking target holding area 411 may be positioned to avoid the superimposed image 410, and as close as possible to the center of the camera image 401.

[0048] In this way, regardless of where the superimposed image is positioned on the camera image, the control unit 103 (or system control unit 111) can automatically set the shape, position, or size of the tracking target holding area according to the superimposed position of the superimposed image.

[0049] For illustrative purposes, this section describes a process that changes the shape and number of tracking target holding areas according to the position and size of the superimposed image. However, the control unit 103 is not limited to this; it may also change the size and position of the tracking target holding areas based on information such as the size of the tracking target and the position of the subject's face, body, etc.

[0050] Furthermore, regarding the tracking target holding area, as shown in Figure 5(A), if the tracking target holding area 502 is set to be small and only a part of the camera image 501, the tracking target subject 503 is always tracked so as to be located near the center of the area (hereinafter referred to as area-center tracking). On the other hand, as shown in Figure 5(B), if the tracking target holding area 502 occupies most of the camera image 401, the control unit 103 may change the type of tracking so that tracking is performed only when the subject 503 moves out of the tracking target holding area (hereinafter referred to as in-area tracking). In this way, the control unit 103 (or system control unit 111) can set the type of tracking according to the size of the tracking target holding area. That is, it can decide whether to track the subject to be tracked so that it is located at the center of the tracking target holding area, or to track the subject to be tracked only when it moves out of the tracking target holding area, according to the size of the tracking target holding area. The tracking control unit 107 then tracks the subject to be tracked using the tracking method set above. Furthermore, the tracking control unit 107 may set the type of tracking according to the size of the tracking target holding area.

[0051] This reduces the possibility of losing the subject or having it overlap with the superimposed image when the size of the tracking target holding area is small, and suppresses unnecessary driving when a large tracking target holding area can be used. As a result, it is possible to reduce image shaking caused by the driving of the imaging device 100.

[0052] As described above, the imaging system 10 of this embodiment can automatically set the position and size of the tracking target holding area that holds the subject to be tracked, according to the image superimposition position information of the superimposed image superimposed on the camera image. This reduces the effort required to manually set the tracking target holding area while visually confirming the position of the superimposed image superimposed on the camera image.

[0053] <Embodiment 2> The configuration of the imaging system 20 according to Embodiment 2 will be described below with reference to Figure 6. Figure 6 is a diagram illustrating the configuration of the imaging system 20 of Embodiment 2. In Embodiment 2, the differences from Embodiment 1 will be explained, and similar points will be omitted from explanation as appropriate.

[0054] The imaging system 20 is configured by connecting imaging devices 100a, 100b, 100c, and 100d with an information processing device 110 via an IP network 120. In other words, in Embodiment 2, the imaging system 20 is configured with multiple imaging devices (imaging devices 100a to 100d) and an information processing device 110. Note that the configuration and functions of each imaging device (100a to 100d) are the same as those of the imaging device 100 in Embodiment 1, so their description is omitted. Similarly, the configuration and functions of the information processing device 110 are the same as those of the information processing device 110 in Embodiment 1, so their description is omitted.

[0055] The display process for the tracking target holding area setting UI will be explained below with reference to Figures 7 and 8. Figure 7 is a flowchart illustrating the display process for the tracking target holding area setting UI of Embodiment 2. Figure 8 is a diagram illustrating the display process for the tracking target holding area setting UI of Embodiment 2. Figure 8(A) is an example of the camera selection UI displayed on the operation display unit 112. Figure 8(B) is an example of the superimposed image selection UI displayed on the operation display unit 112. Figure 8(C) is an example of the superimposed position and size selection UI displayed on the operation display unit 112. Figure 8(D) is an example of the set tracking target holding area displayed on the operation display unit 112. Figure 8(E) is an example of the tracking target holding area modification UI displayed on the operation display unit 112.

[0056] In S701, the system control unit 111 displays a UI on the operation display unit 112 in which images (camera images A to D) acquired by each imaging device (100a to 100d) connected to the information processing device 110 are arranged in a camera selection frame 801. An example of this display state is shown in Figure 8(A). The user selects an image by operating the cursor 802 with a pointing device such as a mouse, or by keyboard input. Here, as shown in Figure 8(A), it is assumed that camera image B acquired by imaging device 100b is selected by operating the cursor 802 or by keyboard input.

[0057] Furthermore, after selecting a camera image, the system can proceed to S702 by selecting the setup completion UI shown in Figure 8(A) using the cursor 802 or keyboard input. In other words, if the setup completion UI is not selected, the display screen in Figure 8(A) will continue, and it will not be possible to proceed to S702. Moreover, if the setup completion UI is selected without selecting a camera image, the system is configured not to proceed to S702. That is, processing will wait until the setup completion UI is selected. In this case, for example, the system control unit 111 may display a message indicating the selection of a camera image or an error notification on the current display screen.

[0058] In S702, the system control unit 111 displays a UI on the operation display unit 112 in which the superimposition images (images A to D) stored in the memory unit 113 are arranged in the superimposition image selection frame 803. An example of this display state is shown in Figure 8(B). The selection of the superimposition images is performed by user operation in the same way as the selection of the camera image in Figure 8(A). Here, it is assumed that image C is selected as shown in Figure 8(B). After selecting the superimposition images, the user can proceed to S703 by selecting the setting completion UI shown in Figure 8(B) using the cursor 802 or keyboard input. That is, if the setting completion UI is not selected, the display screen in Figure 8(B) continues, and it is not possible to proceed to S703. If the setting completion UI is selected without selecting any superimposition images, the system is configured not to proceed to S703. In other words, the process waits until the setting completion UI is selected. In this case, for example, the system control unit 111 may display a message or error notification on the current display screen indicating that an superimposition image should be selected.

[0059] In S703, the system control unit 111 displays a UI on the operation display unit 112 that specifies the superposition position 805 of image C selected in S703 on the area 804 of camera image B selected in S701. An example of this display state is shown in Figure 8(C). Here, the user can perform operations such as moving image C by directly dragging it using the cursor 802, scaling the image by dragging the image scaling UI 806, and rotating the image by dragging the image rotation UI 807. This allows the superposition position 805 of image C to be set. In addition, as setting control for image C, it may be configured to enable editing operations other than scaling, scaling, and rotation, such as cropping and highlighting, using a UI not shown.

[0060] In S704, the system control unit 111 determines whether the image addition UI 808 has been selected. If it is determined that the image addition UI 808 has been selected, the system returns to S702 and displays the same superimposed image selection frame 803 again to allow the user to select the superimposed images to be added (for example, images A, B, and D). On the other hand, if the image addition UI 808 has not been selected, the system proceeds to S705. In S704, the case where the image addition UI 808 has not been selected refers to the case where the setting completion UI has been selected. The selection between the image addition UI 808 and the setting completion UI is made by operating the cursor 802 or by keyboard input. If neither the image addition UI 808 nor the setting completion UI has been selected, the display screen shown in Figure 8(C) continues, and it is not possible to return to S702 or proceed to S705. In other words, the system waits until either the image addition UI 808 or the setting completion UI is selected.

[0061] In S705, the superposition position setting unit 115 converts the superposition position 805 of the image C selected by the user into image superposition position information. Subsequently, the system control unit 111 sets the tracking target holding area 809 according to the image superposition position information converted by the superposition position setting unit 115 and displays it on the operation display unit 112. An example of the display state at this time is shown in Figure 8(D).

[0062] In S706, the system control unit 111 determines whether the area modification UI 810 has been selected. If it is determined that the area modification UI 810 has been selected, the process proceeds to S707. On the other hand, if it has not been selected, the process proceeds to S708. In S706, the case where the area modification UI 810 has not been selected refers to the case where the setting completion UI has been selected. The selection between the area modification UI 810 and the setting completion UI is made by operating the cursor 802 or by keyboard input. If neither the area modification UI 810 nor the setting completion UI has been selected, the display screen shown in Figure 8(D) continues, and the process cannot proceed to S707 and S708. In other words, the process waits until either the area modification UI 810 or the setting completion UI is selected.

[0063] In S707, the system control unit 111 displays a UI for modifying (changing) the tracking target holding area 809. The user can, for example, move the tracking target holding area 809 by directly dragging it using the cursor 802, or modify (change) the position of the tracking target holding area 809 by dragging the area resizing UI 811. An example of the display state in this case is shown in Figure 8(E).

[0064] Furthermore, the tracking type for the tracking target holding area 809 can be selected using the tracking type selection UI 812 shown in Figure 8(E). The system control unit 111 sets the tracking method to track the subject to be tracked using the tracking type selected in the tracking type selection UI 812. Specifically, the tracking type selection UI 812 allows selection between tracking within the area or tracking centered on the area. Note that in the example in Figure 8(E), the target to be modified is the tracking target holding area, but this is not limited to this. For example, the superposition position 805 of image C, which is a superimposed image, may also be modified in the same way.

[0065] In S708, the system control unit 111 transmits the position information of the tracking target holding area 809 on the camera image, which was set in S705, to the imaging device 100b that captured the camera image B via the communication unit 114 and the IP network 120. At the same time, the system control unit 111 transmits the image superposition position information converted in S705 to the image superposition unit 116 and starts the superposition process of image C onto camera image B. After the imaging device 100b receives the position information of the tracking target holding area 809 on the camera image transmitted from the system control unit 111 in S708, it starts tracking control of the subject based on this position information.

[0066] As described above, the imaging system 20 of this embodiment offers the same advantages as in Embodiment 1, plus the ability to automatically set the tracking target holding area according to the image superposition position information selected by the user, and to modify and edit the image to reflect the user's intentions.

[0067] In Embodiment 2, four imaging devices are used for illustrative purposes, but the imaging system 20 may be configured with a single imaging device or more than four imaging devices.

[0068] Although preferred embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims.

[0069] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0070] This embodiment includes the following configurations, methods, and programs.

[0071] (Composition 1) An imaging system comprising at least one imaging device having a subject tracking function, and an information processing device that is communicatively connected to the imaging device, A first setting means for setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired from the imaging device, The system includes a second setting means for setting the tracking and holding region, which is a region in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region in the first image, according to the display position of the second image set by the first setting means. An imaging system characterized by the following:

[0072] (Configuration 2) The imaging system according to configuration 1, characterized in that the second setting means sets the tracking and holding region in a region within the first image that does not overlap with the second image.

[0073] (Composition 3) The imaging system according to configuration 1 or 2, characterized in that it has a tracking control means that controls the imaging direction of the imaging device to track the subject based on the information of the tracking holding region set by the second setting means.

[0074] (Composition 4) The imaging system according to any one of configurations 1 to 3, characterized in that the second setting means sets the type of tracking according to the size of the tracking holding area.

[0075] (Composition 5) The imaging system according to any one of configurations 1 to 4, characterized in that the second setting means determines, according to the size of the tracking and holding area, whether to track the subject to be tracked so that it is positioned at the center of the tracking and holding area, or to track the subject when it moves out of the tracking and holding area.

[0076] (Composition 6) The imaging system according to any one of configurations 1 to 5, characterized in that the tracking and holding region is a region in the image captured by the imaging device.

[0077] (Composition 7) The first setting means is a user interface that accepts operations from the user, The imaging system according to any one of configurations 1 to 6, characterized by having a display control means for displaying the user interface on a display means in an operable manner.

[0078] (Composition 8) The imaging system according to any one of configurations 1 to 7, characterized in that the second setting means sets a plurality of tracking holding regions according to the position of the second image displayed together with the first image.

[0079] (Composition 9) The imaging system according to any one of configurations 1 to 8, characterized in that the second setting means sets the shape of the tracking holding area according to the position of the second image displayed together with the first image.

[0080] (Composition 10) The imaging system according to any one of configurations 1 to 6, characterized in that it has a user interface that can modify at least one of the position or size of the tracking and holding area, or the position where the second image is displayed.

[0081] (Composition 11) The imaging system according to configuration 10, characterized by having a display control means for displaying the user interface on a display means in an operable manner.

[0082] (Composition 12) The imaging system according to any one of configurations 1 to 11, characterized by having a third setting means for setting a tracking method when tracking the subject to be tracked.

[0083] (Composition 13) Image display control means that controls the imaging device to display the second image together with the first image acquired by the imaging device, The system includes an output means that acquires display position information, which is information about the position where the second image set by at least the first setting means is to be displayed, and outputs it to the second setting means, The imaging system according to any one of configurations 1 to 12, characterized in that the second setting means sets the display position of the second image to an area within the first image that does not overlap with the area of ​​the first image, based on the display position information output from the output means.

[0084] (Composition 14) An imaging device having a subject tracking function, Receiving means for receiving display position information, which is information about the position where a second image different from the first image is displayed together with a first image set by an information processing device that is communicably connected to the imaging device, The system includes a second setting means for setting the tracking and holding region within the first image, which is a region in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, based on the display position information received from the receiving means. An imaging device characterized by the following features.

[0085] (Composition 15) An information processing device that is communicatively connected to at least one imaging device having a subject tracking function, A first setting means for setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired by the imaging device, A second setting means sets the tracking and holding region within the first image, which is a region in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, according to the display position of the second image set by the first setting means, The system includes a transmission means for transmitting information about the tracking and holding region to the imaging device. An information processing device characterized by the following:

[0086] (Composition 16) A control method for an imaging system comprising at least one imaging device having a subject tracking function and an information processing device that is communicatively connected to the imaging device, A first setting step involves setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired by the imaging device. The system includes a second setting step of setting within the first image a region in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, according to the display position of the second image set in the first setting step, A method for controlling an imaging system, characterized by the features described above.

[0087] (Composition 17) A program for causing a computer to execute a control method for an imaging system comprising at least one imaging device having a subject tracking function and an information processing device that is communicatively connected to the imaging device, wherein the program causes the computer to: A first setting step involves setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired by the imaging device. A second setting step is performed to set a region within the first image in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, according to the display position of the second image set in the first setting step. A program characterized by the following features. [Explanation of Symbols]

[0088] 10 Imaging System 100 Imaging device 101 Imaging Unit 102 Image Processing Unit 103 Control Unit 104 Communications Department 105 Storage section 106 Drive unit 107 Tracking Control Unit 110 Information Processing Device 111 System Control Unit 112 Operation display section 113 Storage section 114 Communications Department 115 Superposition position setting section 116 Image Overlay Area 120 IP Networks

Claims

1. An imaging system comprising at least one imaging device having a subject tracking function, and an information processing device that is communicatively connected to the imaging device, A first setting means for setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired from the imaging device, The system includes a second setting means for setting the tracking and holding region, which is a region in which the imaging device is controlled to keep the subject to be tracked in the first image within the tracking and holding region, according to the display position of the second image set by the first setting means. An imaging system characterized by the following features.

2. The imaging system according to claim 1, characterized in that the second setting means sets the tracking and holding region in a region within the first image that does not overlap with the second image.

3. The imaging system according to claim 1, further comprising a tracking control means that controls the imaging direction of the imaging device to track the subject based on the information of the tracking holding region set by the second setting means.

4. The imaging system according to claim 1, characterized in that the second setting means sets the type of tracking according to the size of the tracking holding area.

5. The imaging system according to claim 1, characterized in that the second setting means determines, according to the size of the tracking and holding area, whether to track the subject to be tracked so that it is positioned at the center of the tracking and holding area, or to track the subject when it moves out of the tracking and holding area.

6. The imaging system according to claim 1, characterized in that the tracking and holding region is a region in the image captured by the imaging device.

7. The first setting means is a user interface that accepts operations from the user, The imaging system according to claim 1, further comprising a display control means for displaying the user interface on a display means in an operable manner.

8. The imaging system according to claim 1, characterized in that the second setting means sets a plurality of tracking and holding regions according to the position of the second image displayed together with the first image.

9. The imaging system according to claim 1, characterized in that the second setting means sets the shape of the tracking and holding area according to the position of the second image displayed together with the first image.

10. The imaging system according to claim 1, characterized in that it has a user interface that allows modification of at least one of the position or size of the tracking and holding area, or the position where the second image is displayed.

11. The imaging system according to claim 10, further characterized by having a display control means for displaying the user interface on a display means in an operable manner.

12. The imaging system according to claim 1, further comprising a third setting means for setting a tracking method when tracking the subject to be tracked.

13. Image display control means that controls the imaging device to display the second image together with the first image acquired by the imaging device, The system includes an output means that acquires display position information, which is information about the position where the second image set by at least the first setting means is to be displayed, and outputs it to the second setting means, The imaging system according to claim 1, characterized in that the second setting means sets the display position of the second image to an area within the first image that does not overlap with the area of ​​the first image, based on the display position information output from the output means.

14. An imaging device having a subject tracking function, Receiving means for receiving display position information, which is information about the position where a second image different from the first image is displayed together with a first image set by an information processing device that is communicably connected to the imaging device, The system includes a second setting means for setting the tracking and holding region, which is an area in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, based on the display position information received from the receiving means, within the first image. An imaging device characterized by the following features.

15. An information processing device that is communicatively connected to at least one imaging device having a subject tracking function, A first setting means for setting the position and size of a second image, which is different from the first image and is displayed together with the first image acquired by the imaging device, A second setting means sets the tracking and holding region within the first image, which is a region in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, according to the display position of the second image set by the first setting means, The system includes a transmission means for transmitting information about the tracking and holding region to the imaging device. An information processing device characterized by the following:

16. A control method for an imaging system comprising at least one imaging device having a subject tracking function and an information processing device that is communicatively connected to the imaging device, A first setting step involves setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired by the imaging device. The system includes a second setting step of setting within the first image a region in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, according to the display position of the second image set in the first setting step, A method for controlling an imaging system, characterized by the features described above.

17. A program for causing a computer to execute a control method for an imaging system comprising at least one imaging device having a subject tracking function and an information processing device that is communicatively connected to the imaging device, wherein the program causes the computer to: A first setting step involves setting the position and size of a second image, which is different from the first image, to be displayed together with the first image acquired by the imaging device. A second setting step is performed to set a region within the first image in which the imaging device is controlled to keep the subject to be tracked within the tracking and holding region, according to the display position of the second image set in the first setting step. A program characterized by the following features.

Images