Controller, method for control, and program

Abstract

To properly determine an arrangement of images in a display unit in a case where images of a plurality of imaging devices are acquired.SOLUTION: A controller for controlling to display images taken by a plurality of imaging devices in a display unit acquires first information according to a position of a first imaging device from a first image taken by the first imaging device and acquires second information according to a position of a second imaging device from a second image taken by the second imaging device. The controller determines the positions of the first image and the second image to be displayed on the display unit on the basis of the first information and the second information.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a control method for displaying a plurality of captured images. [Background technology]

[0002] In the field of image distribution, a distribution system is known in which a user views images taken by multiple imaging devices and distributes one of the images. In such a distribution system, images input from the multiple imaging devices are displayed on one screen, and the user selects one of them to distribute. In this case, the position on the display unit where each of the images input from the multiple imaging devices is to be arranged is often set manually.

[0003] When displaying multiple captured images on one screen in this way, multiple imaging devices must be connected to a control device, which takes more preparation time than connecting a single imaging device to a control device. Therefore, there is a need to automate the settings that are currently done manually in order to shorten the preparation time.

[0004] Patent Document 1 describes an imaging device having a plurality of imaging units, in which the arrangement of images on a display unit is selected from several fixed patterns. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication 2022-15264 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when it is assumed that a plurality of image capturing devices are freely arranged in a location where images are captured, it is difficult to appropriately determine the image layout on the display unit.

[0007] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a control device that can appropriately determine the image layout on a display unit when acquiring images from a plurality of imaging devices. [Means for solving the problem]

[0008] One aspect of the present invention is a control device that controls the display of multiple captured images captured by multiple imaging devices on a display unit, the control device having a first acquisition means that acquires first information corresponding to a position of a first imaging device from a first captured image captured by a first imaging device, a second acquisition means that acquires second information corresponding to a position of a second imaging device from a second captured image captured by a second imaging device, and a determination means that determines positions of the first captured image and the second captured image to be displayed on the display unit based on the first information and the second information. Effect of the Invention

[0009] According to the present invention, when images are acquired from a plurality of image capturing devices, the image arrangement on the display unit can be appropriately determined. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a distribution system according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the device configuration of an imaging device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of the configuration of a server device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the functional configuration of a server device according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of the configuration of a client device according to the first embodiment. [Figure 6] 5 is a flowchart showing an example of an operation of the server device according to the first embodiment. [Figure 7] FIG. 4 is a diagram illustrating the operation of the server device according to the first embodiment. [Figure 8]10 is a flowchart showing an example of an operation of a distribution system according to a second embodiment. [Figure 9] 13A to 13C are explanatory diagrams of image arrangements in various modes according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. The embodiment described below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. In addition, a configuration may be made by appropriately combining parts of each embodiment described later.

[0012] <First embodiment> (Distribution system configuration) 1 is a diagram showing an example of the configuration of a distribution system including an imaging device according to this embodiment. The distribution system 100 is composed of a server system 110 that distributes images, and client systems 120 (120a, 120b, 120c) that receive the images. The server system 110 and client systems 120 will be described as being connected via a network 105, but they may also be connected by an image cable.

[0013] (Server system configuration) The server system 110 in this embodiment is composed of five imaging devices 101, imaging device 101a to imaging device 101e, a server device 102, an input device 103, a display device 104, and a LAN that enables each device to communicate with each other. In this embodiment, the description will be given assuming that five imaging devices 101 (imaging device 101a, imaging device 101b, imaging device 101c, imaging device 101d, imaging device 101e) are connected to the server device 102, but the number of imaging devices 101 is not limited to this.

[0014] The imaging device 101 transmits captured images to the server device 102 via a LAN (Local Area Network) connection. A LAN cable is assumed as the connection method, but other methods may be used. For example, SDI (Serial Digital Interface) or HDMI (registered trademark) (High-Definition Multimedia Interface) may be used. Alternatively, wireless LAN or UVC (Universal Serial Bus Video Class) may be used. In the case of a LAN cable, up to about 100 devices can be connected.

[0015] An imaging device 101, an input device 103, and a display device 104 are connected to the server device 102 via a LAN. Furthermore, the server device 102 is capable of communicating with a client system 120 via a network 105.

[0016] The input device 103 is a user interface capable of operating the imaging device 101, such as a keyboard or a multi-directional input stick controller, and in this embodiment, the imaging device 101 can be operated via the server device 102, but is not limited to this. For example, the input device 103 may be capable of transmitting an operation instruction to the imaging device 101 without going through the server device 102.

[0017] The display device 104 is capable of displaying an image captured by the imaging device 101 on a display unit 1041. Furthermore, the display device 104 receives a control instruction from the server device 102 and changes the display screen of the display unit 1041.

[0018] In this embodiment, the input device 103 and the display device 104 are described as separate devices, but the input device 103 and the display device 104 may be integrated together, such as a controller with an integrated panel.

[0019] The client device 106 is connected to the network 105, and images distributed from the server device 102 can be viewed on the client device 106.

[0020] (Configuration of imaging device) 2 is a diagram showing an example of the configuration of an imaging device according to this embodiment. The imaging device 101 includes an imaging unit 201, an encoder unit 202, and a network interface (I / F) 203. It also includes a CPU (Central Processing Unit) 204, a RAM (Random Access Memory) 205, and a ROM (Read Only Memory) 206. The imaging optical system 200 is detachably provided in the imaging device 101.

[0021] The imaging optical system 200 is a lens that focuses light from a subject onto an imaging surface of an imaging element 201a (described later), and is composed of, for example, a zoom lens, a focus lens, and a blur correction lens.

[0022] In this embodiment, the imaging device 101 and the imaging optical system 200 are provided as separate bodies that can be detached from each other. However, the imaging device 101 may have the imaging optical system 200, such as an imaging device with an integrated lens.

[0023] The imaging unit 201 captures an image of a subject using the imaging optical system 200 to generate an image. The imaging unit 201 has an image sensor 201a, an amplifier 201b, and an image processing unit 201c. Exposure parameters can be set and changed for each pixel group consisting of multiple pixels (e.g., 128×128 pixels) on the imaging surface. The exposure parameters are parameters related to exposure, and include exposure time, analog gain, and exposure value.

[0024] The imaging element 201a converts light from a subject, which is collected on an imaging surface by the imaging optical system 200, into an electrical signal for each pixel and outputs the signal. The imaging element 201a captures an image of the subject with an exposure time for each pixel group according to the exposure parameters that have been set. The imaging element 201a is, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. Furthermore, the imaging element 201a is an IC chip in which pixels made of photoelectric conversion elements are arranged in a matrix. The imaging element 201a is mainly highly sensitive to visible light, and each pixel is highly sensitive to either red (R), green (G), or blue (B), but also has a certain degree of sensitivity to infrared light. Therefore, it is possible to clearly capture an object that is bright with infrared light, such as during a time period when sunlight is present and in a place illuminated by infrared light illumination.

[0025] The amplifier 201b amplifies and outputs the electrical signal output from the image sensor 201a. The amplifier 201b is provided for each pixel or pixel group, and the signal amplification factor (analog gain) is set according to the exposure parameters for each pixel group.

[0026] The image processing unit 201c performs A / D conversion on the analog electrical signal output from the amplifier 201b to a digital signal, and generates a digital image by performing image processing including demosaicing, white balance processing, gamma processing, etc. The image processing unit 201c amplifies or attenuates the digital value of the image signal output from each pixel or pixel group for each pixel or pixel group, thereby correcting the brightness of the image.

[0027] The encoder unit 202 encodes the image data output from the imaging unit 201 (image processing unit 201c) into a predetermined file format. Examples of the predetermined file format include Motion JPEG, High Efficiency Image File Format (HEIF), H.264, and H.265.

[0028] The network I / F 203 acquires the encoded image data from the encoder unit 202 and transmits it to the client device 106 via the network 105. The encoded image data may be acquired not only from the encoder unit 202, but also from an internal storage device such as a RAM 205 or a ROM 206, which will be described later, or from a removable storage medium (not shown), such as an SD card. The data may be stored in this manner after being output from the image processing unit 201c. In this case, the image data is stored as RAW data before encoding.

[0029] The CPU 204 is a central processing unit that controls the image pickup apparatus 101 .

[0030] The RAM 205 temporarily stores computer programs executed by the CPU 204. The RAM 205 also provides a work area used when the CPU 204 executes processes. The RAM 205 also functions as a frame memory and a buffer memory.

[0031] The ROM 206 stores a program for the CPU 204 to control the image capturing apparatus 101 and the like.

[0032] (Server device configuration) 3 is a diagram showing an example of the configuration of a server device 102 connected to an imaging device according to this embodiment. The server device 102 is an information processing device having a CPU 301, a RAM 302, a ROM 303, an image input I / F 304, an input I / F 305, an output I / F 306, and a network I / F 307.

[0033] The CPU 301 is a central processing unit that controls the server device 102 .

[0034] The RAM 302 has a work area for temporarily storing programs for the CPU 301 to control the server device 102, images input from the imaging device 101, and the like.

[0035] The ROM 303 stores a program for the CPU 301 to control the server device 102, images input from the imaging device 101, and the like.

[0036] The image input I / F 304 is connected to the imaging device 101 via a LAN cable, an HDMI cable, an SDI cable, a wireless LAN, or a USB cable, and an image captured by the imaging device 101 is input to the server device 102.

[0037] The input I / F 305 is an interface that is connected to the input device 103 and receives operations for the server device 102 inputted via the input device 103 from a user (server administrator).

[0038] The output I / F 306 is an interface that is connected to the display device 104 to display the image output from the imaging device 101 on the display device 104 .

[0039] The network I / F 307 is an interface that connects a LAN cable or the like to the network 105. An image captured by the imaging device 101 and input to the server device 102 is distributed to the client device 106 via the network 105. It is also an interface that receives an image distribution request from the client device 106 and distributes the image.

[0040] (Server device functional configuration) FIG. 4 is a diagram showing an example of the functional configuration of the server device 102 according to the present embodiment. Of the functional blocks shown in FIG. 4, the functions realized by software are realized by storing a program in a memory such as the ROM 303. Then, the program is read into the RAM 302 and executed by the CPU 301. The functions realized by hardware are realized by reading a memory such as the ROM 303 by the FPGA and generating a circuit. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as the FPGA and realized as hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit). Note that the configuration of the functional blocks shown in FIG. 4 is an example, and a plurality of functional blocks may constitute one functional block, or any functional block may be divided into blocks performing a plurality of functions. Also, all the configurations may be realized by software, or all the configurations may be realized by hardware.

[0041] The subject determination unit 401 determines, from the image acquired from the imaging device 101, a subject (reference subject) that is to be used for calculating the subject's orientation by the orientation calculation unit 403 (described later). Specifically, people are detected from images a to e acquired from the imaging devices 101 (a to e), and a person who appears in common is set as a reference subject and is used for orientation calculation by the orientation calculation unit 403 (described later). The method of determining the reference subject is determined based on similarity, but is not limited to this. For example, the reference subject may be determined by AI processing. In addition, when multiple people are detected from an image, the reference subject may be determined by a user, or the reference subject may be determined using AI.

[0042] The feature extraction unit 402 extracts features of the reference subject determined by the subject determination unit 401 from the image acquired from the imaging device 101. The target from which features are extracted is assumed to be the face of the reference subject, but other parts may be extracted, and positioning points may be extracted from man-made objects other than people, such as panels and markers. The extraction method is assumed to extract facial feature points such as the eyes, nose, and mouth, but other parts, such as the facial outline and bone structure, may be extracted as feature points. The feature extraction unit 402 stores the extracted feature points in the RAM 302.

[0043] The orientation calculation unit 403 calculates the orientation of the subject's face from the feature extracted by the feature extraction unit 401 in order to estimate the position of the device in the device position estimation unit 404 described later. In this embodiment, the orientation of the subject's face is calculated, but is not limited to this. For example, even if the subject is an artificial object other than a person, such as a panel or a marker, it is sufficient that the relative position of the imaging device 101 and the subject can be calculated in the device position estimation unit 404 described later as long as it is an associated value. In this embodiment, the orientation of the subject is calculated as an average value per arbitrary time, and may be a fixed value that does not change at any time, or a variable value that can be set by the user. Alternatively, it may be a variable value that is appropriately adjusted so that the deviation of the angle of the orientation falls within a threshold value. The allowable deviation range of the angle of the orientation may be set by the user, or may be a fixed value that is determined. Alternatively, the calculation result of the device position estimation unit 404 is fed back, and the allowable deviation range is appropriately adjusted so that the blank space on the display unit 1041 is reduced. For example, the allowable range of blurring in the vertical direction may be set large only when the angle is significantly different, so that the positions of the images in the vertical direction match to a certain extent.

[0044] The device position estimation unit 404 estimates the positions of the imaging devices 101a to 101e based on the calculation results from the orientation calculation unit 403.

[0045] The display position determination unit 405 determines the display positions of the images acquired from the imaging devices 101a to 101e on the display unit 1041 of the display device 104, based on the positions of the imaging devices 101a to 101e estimated by the device position estimation unit 404. Furthermore, it transmits a control instruction to the display device 104 so as to display the images on the display unit 1041 at the determined display positions.

[0046] (Configuration of client system 120) The client system 120 in this embodiment is composed of, for example, a client device 106 and a display device 107, receives an image from the server system 110 via the network 105, and displays the received image on the display device 107. Also, as in the case of client system 120b, an input device 108 may be further provided, and an image distribution request may be made from the input device 108 to the server system 110 via the client device 106b. In a typical distribution system, images distributed from the server system 110 are received by a plurality of client devices 110, and displayed on the display unit 108 in the client system 120. In this embodiment, the images distributed from the server system 110 are received by client systems 120a to 120c, but the number of client systems that receive the images is not limited to this.

[0047] The input device 108 is a keyboard, a mouse, or the like, and is used to operate the client device 106 .

[0048] In this embodiment, the client device 106, the display device 107, and the input device 108 are separate entities, but the client device 106, the display device 107, and the input device 108 may be integrated into one device, such as a notebook PC having a touch panel display.

[0049] (Client Device Configuration) 5 is a diagram showing an example of the configuration of a client device connected to the imaging device according to this embodiment via the Internet. The client device 106 is an information processing device having a CPU 501, a RAM 502, a ROM 503, an input I / F 504, an output I / F 505, and a network I / F 506.

[0050] The CPU 501 is a central processing unit that controls the client device 106 .

[0051] The RAM 502 has a work area for temporarily storing programs and the like for the CPU 501 to control the client device 106 .

[0052] The ROM 503 stores a program for the CPU 501 to control the client device 106 and the like.

[0053] The input I / F 504 is an interface that is connected to the input device 108 and receives operations for the client device 106 inputted from the user of the client system 120 via the input device 108 .

[0054] The output I / F 505 is an interface that is connected to the display device 107 to display the image output from the imaging device 101 on the display device 107 .

[0055] The network I / F 506 is an interface that is connected to the image capturing apparatus 101 via the network 105 , and is used to input operation information for the image capturing apparatus 101 and to accept images output from the image capturing apparatus 101 .

[0056] (Description of the operation of the server device according to the present invention) 6 is a flowchart showing an example of the operation of the server device 102 according to this embodiment. This operation is started when the server device 102 acquires a captured image from the imaging device 101. This operation is a pre-setting that is performed in parallel with the timing of lighting adjustment, which is performed in the preparation stage for shooting in a recording environment such as a studio. In the lighting adjustment, a person is placed on the stage to check the degree of whiteout of the face, and the height and angle of the lighting are changed.

[0057] Fig. 7 is an explanatory diagram supplementing the explanation of the flowchart in Fig. 6. As shown in Fig. 7, five imaging devices 101 are installed in a studio 701, and the angle of view is adjusted so that the subjects in the studio are included in the angle of view as shown in S601. Furthermore, imaging device 101a outputs captured image 711a, imaging device 101b outputs captured image 711b, imaging device 101c outputs captured image 711c, imaging device 101d outputs captured image 711d, and imaging device 101e outputs captured image 711e as captured images.

[0058] In this embodiment, a person is used as the subject, and the display position on the display unit is determined based on the direction of the subject's face, but the subject is not limited to this. A marker or a moving object may be used as the subject as long as the positional relationship between the imaging device and the subject is known. The details of the operation will be described below with reference to Figures 6 and 7.

[0059] In step S601, the captured image 711 is acquired from the imaging device 101, and the subject determination unit 401 determines the reference subject. In this embodiment, if a face cannot be detected in the acquired image, the imaging device 101 is operated so that the face can be detected. In this case, the imaging device is operated via the input device 103, but the input device 1041 for operating the imaging device, which is directly connected to each imaging device, may be used. In addition, if the imaging device 101 has a person detection function and a pan / tilt mechanism, the angle of view may be automatically adjusted. In this case, the angle of view may be adjusted using person detection that searches for a registered person, or a search may be performed at a zoom angle of view so as to improve the identification score, or a combination of a rough search at a wide angle of view and a detailed search at a zoom angle of view may be performed to improve the identification speed. Alternatively, the user may select a reference subject from an image captured by one of the imaging devices 101 using the input device 103 and the display device 104, and the other imaging device 101 may automatically pan and tilt to search for the selected reference subject. In addition, in this embodiment, when there is a single person in each image, the person is determined as the reference subject, and when there are multiple people in each image, the subject is selected by clicking the mouse as described above, but this is not limited to this. For example, the device may automatically select a subject stored in advance in a storage medium such as the ROM 303 of the server device 102 and adjust the angle of view, or the range and speed of the subject's movement may be examined, and the angle of view may be adjusted for a person with little movement and determined as the subject. After the subject determination unit 401 determines the subject, the process proceeds to step S602.

[0060] In step S602, the feature amount extraction unit 402 extracts the feature amount for the reference subject in each captured image 711 in Fig. 7B. Then, the process proceeds to step S603.

[0061] In step S603, the orientation calculation unit 403 calculates the orientation of the subject's face in the image acquired in step S601 based on the feature amount extracted in step S602. This operation is performed for each image of the multiple image capture devices 101 connected to the server device 102. The calculated value is, for example, but is not limited to, a value indicating the degree of deviation up and down and left and right, assuming that the angle when facing the image capture device 101 is 0 degrees. After the orientation of the subject's face is calculated, the process proceeds to step S604.

[0062] In this embodiment, the direction calculation unit 403 calculates the direction of the face by comparing the feature amount acquired in step S602 with the face recognition database 703. For example, in step S603, in the captured image 711a captured by the imaging device 101a, the feature amount extracted in step S602 is compared with the face recognition database of FIG. 7(c). The direction of the subject is calculated by assuming that the angle when the subject faces the camera head-on is 0 degrees in both the horizontal (X-axis) direction and the vertical (Y-axis) direction, and assuming that the subject faces left as +, right as -, upward as +, and downward as -. At this time, the direction of the face of the subject in the captured image 711a is calculated to be (X, Y) = (-20, -20) by the direction calculation unit 403, and the process proceeds to step S604. Note that the direction of the face may be detected only in the horizontal direction, or may be detected only in the vertical direction. The face direction is calculated using a face recognition database (face recognition DB) shown in Fig. 7(c), which is a database related to face directions set in advance, but is not limited to this. For example, the relative positional relationship between the subject and the imaging device may be calculated using AI or a database on the cloud.

[0063] In step S604, the position of the imaging device 101 in real space is estimated by the device position estimation unit 404. The position of the imaging device 101 is estimated based on the face direction of the person calculated in step S603. After the estimation, the process proceeds to step S605. For example, the imaging device 101a captures an image from the right side of the subject using a tripod. At this time, the face direction of the subject in the captured image 711a captured by the imaging device 101a is calculated to be (X, Y) = (-20, -20) from step S603. In step S604, it can be estimated from the calculation result of the face direction in step S603 that the image is captured from the upper right of the subject in real space. The same process is performed in the imaging devices 101b to 101e, and the process proceeds to step S605. In this way, the information indicating the face direction is information according to the position of the imaging device 101. More specifically, the information indicating the face direction is information indicating the relative positional relationship (relative position) between the subject and the imaging device.

[0064] In step S605, based on the positions of the imaging devices 101 estimated in step S604, the display position determination unit 405 determines how to display the images acquired from each imaging device 101 on the display unit 1041 of the display device 104. For example, based on the information on the positions of the imaging devices 101 estimated in step S604, the arrangement of the display unit 1041 of the display device 104 is determined as shown in 705. In the example of Fig. 7, the imaging devices 101 in the studio 701 are arranged in a concave shape, and are similarly arranged on the display unit 1041. After determining the image arrangement, the display position determination unit 405 transmits a control instruction regarding the image arrangement to the display device 104, and this flow ends.

[0065] The display device 104 receives a control instruction from the server device 102 and controls the display on the display unit 1041 .

[0066] By performing such control, in server system 110, which is designed to allow multiple imaging devices to be freely positioned in locations where images will be taken, the image layout on the display unit can be appropriately determined based on the actual positions of the imaging devices.

[0067] <Second embodiment> In the first embodiment, an example was described in which the arrangement of captured images on a display unit is automatically determined based on the arrangement position of an imaging device. Next, an operation will be described in which a method of arranging captured images on a display unit (arrangement combination) suitable for a use case to be used is selected as shown in Fig. 8. Note that the same reference numerals are given to processes that perform the same processes as in the first embodiment, and descriptions thereof will be omitted.

[0068] The distribution system including the imaging device 101 according to this embodiment and the device configuration of the imaging device 101 according to this embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0069] (Operation description) Fig. 8 is a flowchart showing an example of the operation of the distribution system according to this embodiment. This operation is started by reading a program stored in a storage medium such as ROM 303 in server device 102, and having CPU 301 read and execute the program using a memory such as RAM 302 as a work area. Descriptions of processes similar to those in Fig. 6 will be omitted.

[0070] In step S801, the user selects, via the input device 103, the mode in which the images from the imaging device 101 are to be arranged on the display unit 1041 of the display device 104. This selection may be made from the input device 108 via the network 105. Alternatively, the selection may be made automatically depending on the shooting conditions. After the selection, the process proceeds to S601.

[0071] In step S802, the subsequent processing content is changed depending on the mode selected in step S801. In this embodiment, a case where there are three modes is described, but there may be more or less than three modes. If the mode prioritizes studio layout, the process proceeds to S803. If the mode prioritizes director visibility, the process proceeds to S804. If the mode prioritizes performer visibility, the process proceeds to S805.

[0072] The operations in steps S803, S804 and S805 will be described with reference to Fig. 9. After the operations in each step are completed, the process proceeds to step S605.

[0073] (Mode Description) FIG. 9(a) is an explanatory diagram of image arrangement in the studio arrangement mode executed in step S803. This mode is assumed to be used when a person 901 who decides the image to be distributed actually checks the studio with his / her eyes and selects an image-distributing imaging device by looking at the atmosphere. For example, when a subject enters from a blind spot of an imaging device, if you look with your eyes, you can predict in advance which imaging device will be captured next, and it has the advantage that it is possible to switch the image to be distributed at the right timing. What is important at this time is that the arrangement positions of the imaging devices arranged in the actual studio and the images of each imaging device displayed on the display device 104 have the same positional relationship. Specifically, the imaging device 101a uses a tripod, and the imaging device 101b is arranged on the imaging device 101a. Since the imaging devices 101a and 101b are in a vertical relationship, the captured image 711a(1) is displayed on top of the captured image 711b(2) on the display unit 1041 in FIG. 9(a).

[0074] FIG. 9B is an explanatory diagram of image arrangement in the director mode executed in step S804. This mode is assumed to be used when a person 902 who decides the image to be distributed is in a room separated by a wall from the studio and selects an imaging device to distribute the image by looking only at the display unit 1041. What is important at this time is to arrange the images of the imaging devices on the display unit 1041 so that they are actually visible as if the person 902 were in the studio. For example, the director mode is effective when there are multiple subjects, person 903 and person 904, in the studio, and the imaging device 101a photographs person 904 and the imaging device 101b photographs person 903. In the situation of FIG. 9B, if the person 903 is photographed in the mode that prioritizes studio arrangement, person 903 will be displayed in the upper right corner of the display device and person 904 will be displayed in the upper left corner of the display device. This is the opposite arrangement of the people, and it is difficult for person 902 to imagine the arrangement of the people in the studio. Therefore, in accordance with the positioning of the people (person 903 on the left side as viewed from the side of the imaging device 101, and person 904 on the right side), the positions of the imaging devices 101a, 101b and 101d, 101e are swapped as shown in the display unit 1041 in Fig. 2(b). As a result, person 903 is placed at the top left and person 904 at the top right, making it possible to view the images in accordance with an actual studio.

[0075] FIG. 9(c) is an explanatory diagram of image arrangement in the performer visible mode executed in step S805. This mode is assumed to be used when the person 905 who decides the image to be distributed is also a performer and shoots himself and distributes, for example, in a live broadcast of a game. In this case, the user who selects the image is the photographer, so the arrangement of the imaging devices is determined so that they face the photographer. FIG. 9(c) is a diagram showing a person 905 being photographed by two imaging devices. In this case, in FIG. 9(a) or (b), the position of the imaging device when the person 905 is seen from the imaging device 101 side is estimated, and the imaging device 101a is placed to the right of the display unit 1041 and the imaging device 101b is placed to the left of the display unit 1041 in order to determine the arrangement of the images. However, in the case of FIG. 9(c) where the user looking at the display unit 1041 is the subject, the arrangement of the imaging devices when the imaging device 101 is seen from the person 905 side is reversed in the modes of FIG. 9(a) and FIG. 9(b). Therefore, the image of the imaging device 101a placed on the left is placed directly to the left of the display device 104, and the image of the imaging device 101b placed on the right is placed directly to the right of the display unit 1041, thereby aligning the placement positions of the imaging devices with the display positions on the display unit 1041. Also, by mirror-inverting each image, for example, the left hand is displayed on the left of the display device 104. This makes it possible to view the images in accordance with the actual placement positions of the imaging devices.

[0076] By performing such control, in server system 110, which is designed to allow multiple imaging devices to be freely arranged in locations where images will be taken, it is possible to select not only the relative positions of the actual imaging devices, but also the method of arranging the captured images on the display unit that is appropriate for the use case to be used.

[0077] <Other embodiments> The present invention can be realized by a process of reading and executing a program that realizes one or more functions of the above-mentioned first and second embodiments. This program is supplied to a system or device via a network or a storage medium, and is read and executed by one or more processors in a computer of the system or device. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0078] 101 Imaging device 102 Server equipment 1041 Display section 405 Display position determination section

Claims

1. A control device that controls a plurality of captured images captured by a plurality of imaging devices to be displayed on a display unit, a first acquisition means for acquiring first information corresponding to a position of a first imaging device from a first captured image captured by the first imaging device; a second acquisition means for acquiring second information corresponding to a position of a second imaging device from a second captured image captured by the second imaging device; a determination means for determining positions of the first captured image and the second captured image to be displayed on the display unit based on the first information and the second information; A control device comprising:

2. 2. The control device according to claim 1, wherein the first information and the second information of the subject are information related to a relative position between the imaging device and the subject.

3. The control device according to claim 2 , further comprising a direction calculation unit that calculates a direction of the face of the subject from the first captured image.

4. the determining means determines display positions of the first captured image and the second captured image on the display unit based on a relative position between the imaging device and the subject.

3. The control device according to claim 2, wherein:

5. the determining means determines display positions of the first captured image and the second captured image on the display unit based on a position of the imaging device in real space; 3. The control device according to claim 2, wherein:

6. A combination of display positions on the display unit is selectable.

3. The control device according to claim 2.

7. A control method for controlling a display unit to display a plurality of captured images captured by a plurality of imaging devices, comprising: a first acquisition step of acquiring first information corresponding to a position of a first imaging device from a first captured image captured by the first imaging device; a second acquisition step of acquiring second information corresponding to a position of a second imaging device from a second captured image captured by the second imaging device; a determining step of determining positions of the first captured image and the second captured image to be displayed on the display unit based on the first information and the second information; A control method comprising the steps of:

8. A program for causing a computer to execute the control method according to claim 7.

Images