Display control device, display control method, and program

Abstract

To facilitate management by a user of a multi-camera system.SOLUTION: A display control device generates a user interface (UI) image indicating a multi-camera system having a plurality of imaging devices, in which information indicating the two or more imaging devices performing a prescribed cooperative operation using only a part of the plurality of imaging devices among the plurality of imaging devices is displayed in a form different from the imaging device not performing the prescribed cooperative operation, and performs control of displaying the UI image.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a management technique for a system in which a plurality of image capturing devices are coordinated to capture images. [Background technology]

[0002] There exists a multi-camera system that captures images of a subject from multiple directions synchronously using multiple imaging devices (cameras) installed at different positions, and performs image processing such as generating a virtual viewpoint image or a stereoscopic image using the multiple images (multiple viewpoint images) obtained by the imaging. Patent Document 1 describes a technology that uses multiple cameras to generate a highly accurate three-dimensional model of a subject using distance images obtained by imaging.

[0003] In a multi-camera system, since there are many pieces of equipment such as imaging devices, it is important to efficiently manage them. Patent Document 2 describes a technology for generating and displaying a user interface screen that enables a user to easily recognize the status of multiple imaging devices in a multi-camera system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-015863 A [Patent Document 2] JP 2019-088025 A Summary of the Invention [Problem to be solved by the invention]

[0005] It is assumed that two or more imaging devices, which are part of a large number of imaging devices included in a multi-camera system, are used in cooperation with each other. In such a case, a user interface screen such as that described in Patent Document 2 does not allow a user to recognize which imaging devices are operating in cooperation with each other. This can make it difficult for a user to manage the multi-camera system. [Means for solving the problem]

[0006] The present invention provides a technique for facilitating user management of a multi-camera system.

[0007] A display control device according to one embodiment of the present invention has a user interface (UI) image showing a multi-camera system having a plurality of imaging devices, in which information indicating two or more imaging devices among the plurality of imaging devices that perform a predetermined coordinated operation in which only a portion of the plurality of imaging devices are used is displayed in a format different from that of imaging devices that do not perform the predetermined coordinated operation, and has a display control means for controlling the display of the UI image on a display means. Effect of the Invention

[0008] According to the present invention, it becomes possible for a user to easily manage a multi-camera system. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of an image generation system. [Diagram 2] FIG. 1 is a diagram showing an example of camera arrangement. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information display device. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an information display device. [Diagram 5] FIG. 4 is a diagram illustrating an example of data stored in a data storage unit. [Figure 6] 10 is a diagram illustrating an example of data stored in a stereo camera information storage unit. FIG. [Figure 7] FIG. 13 is a diagram showing an example of a screen display showing the state of the system. [Figure 8] FIG. 11 is a diagram showing an example of a flow of processes executed by the information display device. [Figure 9] FIG. 13 is a diagram showing an example of a screen display when the system is in a normal state. [Figure 10] FIG. 13 is a diagram showing an example of a screen display when an abnormality occurs in the system. [Figure 11] FIG. 13 is a diagram showing a second example of a screen display when the system is in a normal state. [Figure 12] FIG. 13 is a diagram showing a second example of a screen display when an abnormality occurs in the system. [Figure 13] FIG. 13 is a diagram showing a third example of a screen display when the system is in a normal state. [Figure 14] FIG. 13 is a diagram showing a third example of a screen display when an abnormality occurs in the system. [Figure 15] FIG. 11 is a diagram showing a fourth example of a screen display when the system is in a normal state. [Figure 16] FIG. 13 is a diagram showing a fourth example of a screen display when an abnormality occurs in the system. [Figure 17] FIG. 13 is a diagram showing another example of a screen display showing the system status. [Figure 18] 11 is a diagram illustrating another example of data stored in the stereo camera information storage unit. FIG. [Figure 19] FIG. 13 is a diagram showing an example of a screen display when an alternative camera is selected. [Figure 20] FIG. 13 is a diagram showing an example of a screen display when an alternative camera is selected. [Figure 21] FIG. 13 is a diagram showing an example of a screen display for instructing switching to an alternative camera. [Figure 22] FIG. 11 is a diagram showing a fifth example of a screen display when an abnormality occurs in the system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (System Configuration) An example of the configuration of an image generation system according to this embodiment is shown in Fig. 1. This image generation system includes, for example, a camera group 100 including a plurality of cameras (imaging devices) arranged at different positions, an image generation device 160, and an information display device 170.

[0012] The camera group 100 includes a number of cameras arranged at different positions, for example, as shown in FIG. 2. In the example of FIG. 2, the cameras are arranged to surround a target area 200 to be imaged, and are configured to capture the target area 200 from different directions to obtain captured images. The cameras constituting the camera group 100 are, for example, digital cameras. The cameras may be cameras that capture still images or cameras that capture moving images. The cameras may be cameras that capture both still images and moving images. In this embodiment, the term "image" includes both still images and moving images, unless otherwise specified. In this embodiment, the imaging unit having an image sensor and the lens that collects light rays on the image sensor are collectively called a camera.

[0013] Returning to FIG. 1, in the camera group 100 arranged as shown in FIG. 2, seven cameras are daisy-chained to form one group. FIG. 1 shows a state in which there are four such groups. That is, the first group includes cameras 111 to 117, the second group includes cameras 121 to 127, the third group includes cameras 131 to 137, and the fourth group includes cameras 141 to 147. The cameras included in the first to fourth groups are connected to the HUB 150 via the cameras 111, 121, 131, and 141, respectively. Note that the number of cameras included in one group is merely an example, and the number may be six or less, or eight or more. The number of groups is not limited to four.

[0014] The HUB 150 is a hub for connecting the camera group 100, the image generating device 160, and the image display device 170 to each other.

[0015] The image generating device 160 acquires image data captured by each camera included in the camera group 100 via the HUB 150. The image generating device 160 has a function of generating a virtual viewpoint image using, for example, an image captured by the camera group 100. That is, the image generating device 160 acquires multiple viewpoint images obtained by a large number of cameras included in the camera group 100 performing image capturing synchronously (at the same timing) from different positions and directions, and generates a virtual viewpoint image based on the multiple viewpoint images. The image generating device 160 extracts a moving subject from the multiple viewpoint images as a foreground, estimates a three-dimensional shape of the subject based on data indicating the extracted foreground, and generates a three-dimensional model of the subject. The three-dimensional model of the subject is generated using, for example, a visual volume intersection method. Then, the image generating device 160 draws the subject as viewed from the position and direction of a virtual viewpoint in a virtual space set by a user using a user interface (UI) (not shown) based on the generated three-dimensional model of the subject. The virtual viewpoint image in this embodiment includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily specified by the user. The virtual viewpoint image also includes an image corresponding to a viewpoint specified by the user from a plurality of candidates and an image corresponding to a viewpoint automatically specified by the device. The UI is a mechanism for accepting operations by the user, such as a mouse, a keyboard, an operation button, a touch panel, etc. The UI may be built into the image generating device 160 or may be provided outside the image generating device 160.

[0016] In this embodiment, the image generating device 160 generates a virtual viewpoint image, but the present invention is not limited to this. For example, the image generating device 160 may have a function of generating an image other than a virtual viewpoint image, such as a stereoscopic image or a panoramic image, using image data captured at the same time by a plurality of cameras.

[0017] The information display device 170 receives status information from each camera included in the camera group 100 via the HUB 150. The information display device 170 then executes control to display a screen that enables a user to recognize the status information on a display device such as a display that the information display device 170 has or that is connected to the information display device 170. The status information includes, for example, at least one of the communication connection status of the camera, the shooting status of the camera, the output status of the camera, the presence or absence of an error in the camera, or the shooting parameters of the camera. Here, the communication connection status of the camera is, for example, information indicating a status indicating whether communication is possible, such as a connected status or a non-connected status. The connected status here refers to, for example, a status in which communication with the camera can be performed via the information display device 170. Note that the communication connection status indicates whether the camera is in a state in which communication with the image generating device 160 is possible, and it is not necessary to consider whether communication with the information display device 170 is possible. That is, since it is sufficient to monitor whether each device included in the image generation system can operate properly, even if the camera and the information display device 170 are not connected, if the camera can execute communication within the system, the camera may be treated as being in a connected state. The shooting state of the camera is, for example, information indicating a state such as shooting, preparation, power OFF, and power ON. The output state of the camera is information indicating whether image data is being output. The presence or absence of an error in the camera is, for example, information indicating normal, temperature error, lens error, and the like. The shooting parameters of the camera are, for example, information including an aperture value (iris), shutter speed, white balance, ISO sensitivity, zoom, focus, and ND filter. In addition, the information display device 170 may execute display control to acquire the installation location of each camera and the connection relationship between the cameras from a database and display the acquired information on a screen. This allows the system administrator (user) to constantly monitor the state of the image generation system 10 by viewing the information displayed by the information display device 170.

[0018] The image generation system 10 may have a configuration different from that shown in FIG. 1. For example, a plurality of cameras included in the camera group 100 may be directly connected to the image generation device 160 and the information display device 170. The cameras included in the camera group 100 may not be daisy-chained, and some or all of the cameras may be directly connected to the HUB 150 or the image generation device 160 and the information display device 170. The devices may be connected by wire or wirelessly. The image generation device 160 and the information display device 170 may each be composed of a plurality of devices, or these devices may be implemented as a single device.

[0019] In this embodiment, two or more cameras that are part of the cameras included in the camera group 100 are operated in cooperation with each other. For example, in generating a virtual viewpoint image, a group of two or more cameras among the multiple cameras can be used as a stereo camera to identify the distance to the surface of the subject and to improve the accuracy of the representation of the unevenness of the three-dimensional model. In generating a virtual viewpoint image, it can be said that all of the many cameras belonging to the camera group 100 work together, but the cooperation operation in this embodiment refers to a predetermined cooperation operation in which only a part of the many cameras belonging to the camera group 100 work together. The predetermined cooperation operation can be used for purposes other than, for example, improving the accuracy of the three-dimensional model in the virtual viewpoint image as described above. For example, it is assumed that some of the cameras included in the camera group 100 are used in cooperation as a stereo camera to generate one stereoscopic image. In this embodiment, an information presentation technology by the information display device 170 is provided to facilitate management of two or more cameras used as such a stereo camera.

[0020] (Device configuration) Next, a configuration example of the information display device 170 will be described. FIG. 3 is a diagram showing a hardware configuration example of the information display device 170. The information display device 170 includes a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 304, a communication interface (communication I / F 305), a display device 306, an operation device 307, and a bus 306. The CPU 301 controls the entire information display device 170 using computer programs and data stored in the ROM 302 and the RAM 303. The information display device 170 may have one or more dedicated hardware pieces different from the CPU 301, and at least a part of the processing by the CPU 301 may be executed by the dedicated hardware pieces. The dedicated hardware pieces include, for example, at least one of an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and a digital signal processor (DSP). The ROM 302 stores programs and parameters that do not require modification. The RAM 303 temporarily stores programs and data supplied from the auxiliary storage device 304, and data supplied from the outside via the communication I / F 305. The auxiliary storage device 304 includes, for example, a hard disk drive and stores various content data such as images and audio. The communication I / F 305 includes a communication circuit for wired or wireless communication, and is used for communication with an external device such as the camera group 100. For example, when the communication I / F 305 provides a wired communication function, it includes a connector to which a communication cable is connected and a modem for wired communication. When the communication I / F 305 provides a wireless communication function, it includes an antenna, a radio frequency (RF) chip, a baseband chip, and the like. The display device 306 includes a mechanism for displaying a screen such as a liquid crystal display or an organic EL display. The display device 306 may include a display mechanism other than a display, such as a lamp or a light emitting diode. The information display device 170 may perform display control for outputting information to an external display device and displaying a screen. That is, the information display device 170 can be a display control device that executes display control processing for a display device built into the information display device 170 itself or an external display device.In this case, the information display device 170 may be configured to output information to an external display device via the communication I / F 305. When such a configuration is used, the display device 306 may be omitted. The operation device 307 includes an operation mechanism such as at least one of a keyboard, a mouse, a touch panel, a joystick, or a button. The operation device 307 may be an interface for accepting operation input information from an external operation acceptance device. The display device 306 and the operation device 307 may be configured as a single device using a touch panel display or the like. The bus 308 connects the hardware elements of the information display device 170 to each other.

[0021] 4 shows an example of the functional configuration of information display device 170. Information display device 170 has, as its functions, for example, a data storage unit 401, a stereo camera information storage unit 402, a data reading unit 403, a status acquisition unit 404, a UI generation unit 405, a display unit 406, a user input unit 407, a control unit 408, and a control signal output unit 409.

[0022] The data storage unit 401 stores data necessary for the UI generation unit 405 to generate a UI to be displayed. This data includes at least one of, for example, layout data of the target area 200, identification information of the camera, connection group information to which the camera belongs, connection destination of the camera, or installation coordinate information of the camera. The layout data of the target area 200 may be indicated by, for example, coordinates with a predetermined coordinate (for example, the center of the target area 200) as the origin. The identification information of the camera is information for identifying the cameras arranged around the target area 200 by identification information uniquely assigned to each camera. The connection group information to which the camera belongs is, for example, information indicating to which group formed by seven cameras daisy-chained in the camera group 100 in FIG. 1 the camera belongs. The connection destination of the camera is information indicating the device to which the information display device 170 is connected as viewed from the camera. In the following, the information display device 170 side as viewed from the camera may be called the "upstream side". The camera installation coordinate information is information indicating the position where the camera is installed, using, for example, a coordinate system used when indicating the layout data of the target area 200.

[0023] FIG. 5 shows an example of data held by the data holding unit 401. In the example of FIG. 5, it is shown that the cameras 111 to 117 connected in a daisy chain belong to a connection group called "L1". On the other hand, it is shown that the cameras 121 to 122 connected in a daisy chain in a different system belong to a connection group called "L2". It is also shown that the cameras 111 to 117 are connected in a daisy chain such that the upstream connection destination of the camera 117 is the camera 116, the upstream connection destination of the camera 116 is the camera 115, the upstream connection destination of the camera 112 is the camera 111, and so on. It is also shown that the camera 111 located at the most upstream side in the daisy chain is connected to a HUB. Note that when the cameras 111 to 117 are directly connected to a HUB, the connection destination of these cameras is the "HUB". The installation position is represented by a position on an XY plane consisting of X and Y coordinates when, for example, the center of the target area 200 is defined as the origin, as described above. In FIG. 5, for example, the installation position of camera 111 is indicated by coordinates (x11, y11). Here, the installation position of the camera on the XY plane is shown, but the height direction position of the camera may be indicated by introducing a Z coordinate. Data storage unit 401 may acquire such information by an operation input by a user. Also, data storage unit 401 may acquire such information from each camera as status information of each camera. Note that, although an example in which data storage unit 401 is included in information display device 170 is shown in FIG. 4, this is not limiting. For example, data storage unit 401 may be implemented in a recording device external to information display device 170. Also, data storage unit 401 may be implemented in each camera as a function for storing information related to the camera.

[0024] The stereo camera information holding unit 402 holds data necessary for the UI generating unit 405 to display a UI related to the stereo camera. This data includes, for example, a stereo camera ID and identification information for each of a plurality of cameras used as a stereo camera. The stereo camera ID is identification information given to a group that associates a plurality of cameras when the plurality of cameras are used as a stereo camera in cooperation with each other. The camera identification information is identification information for a plurality of cameras used as a stereo camera corresponding to one stereo camera ID. FIG. 6 shows an example of data held by the stereo camera information holding unit 402. In the example of FIG. 6, the camera 112 and the camera 131 are used as a stereo camera, and "StrCam1" is assigned as the stereo camera ID. In addition, two cameras, the camera 111, the camera 121, and the camera 141, are used as a stereo camera, and "StrCam2" is assigned as the stereo camera ID. Note that this information is an example, and information related to the stereo camera may be managed in a format other than this. For example, the stereo camera ID may be managed for each camera.

[0025] The data reading unit 403 reads various data from the data storage unit 401 and the stereo camera information storage unit 402 , and outputs the data to the UI generation unit 405 .

[0026] The status acquisition unit 404 acquires status information of each camera. As described above, the status information includes at least one of the communication connection status of the camera, the shooting status of the camera, the output status of the camera, the status of the camera with or without an error, or the shooting parameters of the camera. The status acquisition unit 404 may acquire status information of the image generating device 160. The status information of the image generating device 160 includes at least one of the communication connection status, the virtual viewpoint image generation status, the output status, or the status of the error. The communication connection status indicates whether the image generating device 160 is connected to the information display device 170 via, for example, a HUB. The communication connection status indicates whether the image generating device 160 is in a state where it can communicate with the camera, and does not necessarily indicate whether it can communicate with the information display device 170. The virtual viewpoint image generation status indicates a status such as running, preparing, or completed for image generation in the image generating device 160. The virtual viewpoint image generation status may also indicate information such as the CPU load of the image generating device 160. The output state indicates whether the image generating device 160 is outputting a generated image or not. The error presence / absence state indicates whether the image generating device 160 is operating normally or not, and if not, indicates information indicating the cause of the error, such as a temperature error. Note that the error presence / absence state may indicate, for example, that the image generating device 160 is operating normally but is in a state in which an error may occur, such as the CPU being in a high temperature state. The state acquisition unit 404 outputs each piece of acquired information to the UI generation unit 405.

[0027] The UI generating unit 405 generates a UI image including information indicating the state of the system and the camera, and the stereo camera, based on the data output from the data reading unit 403 and the state information output from the state acquiring unit 404. If any state has changed, the UI generating unit 405 may generate a UI image in which the state has changed. The UI generating unit 405 may monitor whether each state has changed and generate a UI image after the state has changed. Here, the data reading unit 403 and the state acquiring unit 404 may each determine whether or not there has been a state change, and if there has been a state change, transmit a predetermined notification to the UI generating unit 405. The UI generating unit 405 may periodically acquire data from the data reading unit 403 and the state acquiring unit 404, and generate a UI image based on the latest data. That is, the UI generating unit 405 may generate a UI image based on the latest information without recognizing whether or not there has been a state change. The generated UI image will be described later. The UI generating unit 405 outputs data of the generated UI image to the display unit 406.

[0028] The display unit 406 uses a frame buffer and a display panel to store data of the UI image output from the UI generation unit 405 in the frame buffer. When data is stored in the frame buffer, the display unit 406 overwrites the data in the frame buffer with the newly acquired data in response to acquiring new data from the UI generation unit 405. The display unit 406 then reads out the data stored in the frame buffer at a predetermined refresh rate and displays the UI image on a display panel such as a liquid crystal display. When the information display device 170 displays a UI image on an external display device, the display unit 406 performs display control to display the UI image on the display device.

[0029] The user input unit 407 receives operation input from the user and generates operation information. The user input unit 407 receives user operations such as switching the display mode described below and specifying a camera when changing the combination of stereo cameras. Then, the user input unit 407 generates, as operation information, information indicating that the specific user operation has been performed, and, when a predetermined operation such as a click operation has been performed on the screen, information such as coordinates indicating the position on the screen where the operation has been performed. The user input unit 407 outputs the generated operation information to the control unit 408.

[0030] The control unit 408 issues a control instruction to each functional unit of the information display device 170 based on the operation information input from the user input unit 407. For example, when the control unit 408 receives operation information indicating an instruction to switch the display mode from the user input unit 407, the control unit 408 generates display instruction information between the stereo camera configuration display and the normal display according to the operation information. Then, the control unit 408 can transmit the display instruction information to the UI generation unit 405. The control unit 408 can also generate a signal for controlling an external device such as the image generation device 160. In this case, the control unit 408 transmits the generated control signal to the control signal output unit 409. The control signal output unit 409 outputs the control signal for controlling the external device input from the control unit 408 to the device that is the destination of the control signal. The control signal output unit 409 can output, for example, a control device to the image generation device 160 or a predetermined camera in the camera group 100. For example, as described later, when the combination of the stereo cameras is changed according to a user instruction, the control unit 408 generates a control signal for notifying the image generating device 160 of the combination information before and after the change. Then, the control signal output unit 409 transmits the control signal to the image generating device 160.

[0031] Here, an example of a UI screen generated by the UI generating unit 405 and displayed by the display unit 406 will be described with reference to FIG. 7. In FIG. 7, a circle 702 indicating the installation position and state of the camera, a line 703 indicating the connection relationship and connection state between the cameras, and an area 704 indicating the target area 200 are shown in a UE screen 701. Here, a circle is used as a sign indicating a camera, and a straight line is used as a sign indicating a connection between the cameras, but signs of other formats may be used. According to the information of the "installation position" of each camera among the information shown in FIG. 5, a circle 702 corresponding to each camera is placed at a predetermined position in the screen. Note that a camera ID may be displayed inside or near the circle 702 to indicate which camera the circle 702 corresponding to each camera corresponds to. Also, if necessary, the camera ID may be displayed on the screen by a pop-up. Then, a line 703 connecting the circle corresponding to each camera to another circle or a HUB is placed according to the information of the "connection destination" shown in FIG. 7, L1 to L4 are information shown as the connection group information shown in FIG. 5. For example, a connection group 705 including cameras 141 to 147 is drawn as seven circles constituting a daisy chain. Other connection groups L1 to L3 are drawn in the same manner. By the information display device 170 performing such a display, the user can recognize information related to a multi-camera system having a plurality of cameras at a glance. Furthermore, the user can easily recognize the position where each camera is arranged and the connection destination of the camera. Furthermore, the information display device 170 can perform a display for distinguishing between a camera that is in a state where it can take pictures and a camera that is broken or being adjusted and is not in a state where it can take pictures. For example, the information display device 170 can change the shape of a figure indicating a camera that is not in a state where it can take pictures to a rectangle instead of a circle. Furthermore, the information display device 170 may display, for example, a circle indicating a camera that is not in a state where it can take pictures by filling it in. Note that the information display device 170 may display a camera that is in a state where it can take pictures by a rectangle or a filled circle, and a camera that is not in a state where it can take pictures by an unfilled circle.That is, the cameras may be represented by any figure as long as the difference in state can be identified. Such a display allows the user to easily identify the camera in which a malfunction has occurred. Also, the user can easily identify the location where such a camera is located. Note that the display example in FIG. 7 shows an example of a normal display that does not include information about the stereo camera. An example of a display that includes information about the stereo camera will be described later.

[0032] (Processing flow) The information display device 170 switches between a normal display as shown in Fig. 7 and a stereo camera configuration display that allows the configuration of the stereo camera to be recognized. The process of this switching will be described below.

[0033] First, the information display device 170 determines whether an instruction to display the stereo camera configuration has been received (S801). For example, the user input unit 407 receives input of operation information from a keyboard or the like, and outputs the information to the control unit 408. In one example, the information display device 170 assigns in advance, among the 10 keys (ten-key pad) of the keyboard, the "1" key to turn on the stereo camera configuration display, and the "2" key to turn off the stereo camera configuration display (normal display). Then, information indicating whether the key pressed by the user operation is "1" or "2" is output to the control unit 408. When the control unit 408 receives operation information indicating that "1" has been pressed, the control unit 408 may determine that an instruction to display the stereo camera configuration has been received. Also, when the control unit 408 receives operation information indicating that "2" has been pressed, the control unit 408 may determine that an instruction to display the normal display has been received. When an instruction to display the stereo camera configuration is received (YES in S801), the information display device 170 causes the UI generation unit 405 to acquire status information of all cameras from the status acquisition unit 404 (S802). Then, the UI generation unit 405 generates a UI image for displaying the stereo camera configuration based on the acquired status information, and displays the UI image on the display unit 406 (S803). Here, for example, when there is a camera whose status information indicates an abnormality, the UI generation unit 405 can generate a UI image to indicate that the camera has an abnormality.

[0034] When the information display device 170 does not receive an instruction to display the stereo camera configuration (NO in S801), the UI generation unit 405 acquires state information of all cameras from the state acquisition unit 404 (S804). Note that when the information display device 170 does not receive a particular instruction, it may determine that the instruction to display the stereo camera configuration has not been received. The UI generation unit 405 determines whether or not a camera in an abnormal state exists based on the acquired state information (S805). When a camera in an abnormal state does not exist (NO in S805), the UI generation unit 405 generates and displays a normal screen that does not include stereo camera information as shown in FIG. 7 (S808). On the other hand, when a camera in an abnormal state exists (YES in S805), the UI generation unit 405 checks the stereo camera configuration data acquired from the stereo camera information storage unit 402 (S806). Then, the UI generation unit 405 determines whether or not the camera in an abnormal state is used as a stereo camera based on the data (S807). When the camera in the abnormal state is used as a stereo camera (YES in S807), the UI generating unit 405 generates a UI image for displaying the stereo camera configuration to display information about the stereo camera (S803). Here, the other cameras used as a stereo camera in combination with the camera in the abnormal state may also be displayed to indicate that they are in an abnormal state. This allows the user to easily recognize that the camera itself is normal but that an abnormality has occurred in the stereo camera function. On the other hand, when the UI generating unit 405 determines that the camera in the abnormal state is not used as a stereo camera (NO in S807), it generates and displays a UI screen of a normal display that does not include information about the stereo camera (S808). In this case, the camera in the abnormal state is displayed in a format that allows the user to identify that it is in an abnormal state, and is displayed in such a way that it can be distinguished from other normal cameras. This allows the user to recognize that some cameras are not being used as stereo cameras while recognizing that the cameras are abnormal.Note that the processing example in FIG. 8 is just an example, and even if a camera in an abnormal state is used as a stereo camera, normal display may be performed unless an instruction to display the stereo camera configuration is accepted by a user operation.

[0035] (Stereo camera configuration display) Next, some examples of display forms of the stereo camera configuration are shown. The information display device 170 may display any one of a plurality of display forms described below. The information display device 170 may also switch the display form based on a user instruction. That is, any display may be performed as long as the user can appropriately check the information of the stereo camera.

[0036] FIG. 9 shows a first example of a screen display of a stereo camera configuration. This screen is displayed based on the receipt of an instruction to display the stereo camera configuration. The information display device 170 performs display control on this screen to enable the user to easily recognize a part of the camera group 100 to which a stereo camera ID is assigned. For such display control, an operation box 901 is prepared on the screen of FIG. 9. The operation box 901 includes information 902 of a stereo camera whose stereo camera ID is set to "StrCam1" and information 904 of a stereo camera whose stereo camera ID is set to "StrCam2". Note that, in FIG. 9, information on two stereo cameras set as stereo cameras by a user is shown, but three or more stereo cameras may be set, or only one stereo camera may be set. In addition, information on a stereo camera for which information is displayed may be limited by user selection from among the set stereo cameras. In addition, the information 902 and the information 904 are selected by a user operation, and information that can identify a stereo camera corresponding to the information is shown. 9 shows an example in which information 902 corresponding to StrCam1 is selected by the user via user input unit 407, and information 902 is displayed in a background color different from that of unselected information 904. As a result, cameras 112 and 131 forming the stereo camera corresponding to selected StrCam1 are indicated by adding, for example, dashed circles 906 and 907. On the other hand, dashed circles are not added to cameras 111, 121, and 141 forming the stereo camera corresponding to unselected StrCam2. By checking the dashed circle added so as to surround the circle indicating this camera, the user can recognize at a glance the multiple cameras forming the stereo camera corresponding to StrCam1.

[0037] The information indicating the camera belonging to the selected stereo camera may be expressed in a form other than a dashed circle. For example, the size of the circle (such as circle 702 in FIG. 7) indicating the camera belonging to the stereo camera may be changed. In one example, the diameter of the circle indicating the camera belonging to the stereo camera may be a predetermined multiple, such as twice, compared with the circles indicating the other cameras. Also, for example, the color of the circle indicating the camera may be changed. For example, the circle indicating the camera not belonging to the stereo camera may be drawn in black, and the circle indicating the camera belonging to the stereo camera may be drawn in red. Note that this is only one example, and the circle indicating the camera belonging to the stereo camera may be drawn in a color that the user can easily recognize. Also, the circle indicating the camera may be changed to another shape, such as a rectangle, for the camera belonging to the stereo camera. Also, for example, the transmittance of the circle indicating the camera may be changed. For example, the circle indicating the camera belonging to the stereo camera may be expressed darker by lowering the transmittance compared with the circles indicating the other cameras. Or, the circle indicating the camera belonging to the selected stereo camera may be blinked. Also, for example, when a character string indicating the identification information of the camera is displayed around the circle indicating the camera, the display form of the character string of the identification information may be changed. For example, "camera 113" may be displayed around a circle indicating camera 113 not belonging to the stereo camera, while "camera 121_StrCam1" may be displayed around a circle indicating camera 121 belonging to the stereo camera. That is, the stereo camera ID may be added to the end to indicate that camera 121 belongs to the stereo camera corresponding to StrCam1. Note that the stereo camera ID may be shortened to "S1" and displayed as "camera 121_S1". For example, the font used when displaying the identification information of the camera belonging to the stereo camera may be changed. Furthermore, the identification information of the camera belonging to the stereo camera may be displayed in bold. In this way, the cameras belonging to the stereo camera may be displayed in any format different from the cameras not belonging to the stereo camera.

[0038] Next to the information 902 and information 904 about each stereo camera, information 903 and information 905 are displayed indicating whether the stereo camera is operating normally. In Fig. 9, information 903 is displayed in white, which indicates that the stereo camera corresponding to StrCam1 is operating normally. Note that in Fig. 9, status information 905 for unselected StrCam2 is also displayed in white, which indicates that the stereo camera corresponding to StrCam2 is operating normally. This is just one example, and for unselected StrCamn2, information 905 may be displayed in a color indicating a non-selected state.

[0039] FIG. 10 shows an example of a screen when the camera 131 transitions from the display state of the screen shown in FIG. 9 to an abnormal state. Information 1001 indicates that the camera 131 has entered an abnormal state. In this case, since an abnormality has occurred in the camera 131, the function of the stereo camera with the stereo camera ID "StrCam1" to which the camera 131 belongs does not operate normally. To indicate this state, the information 903 is displayed differently from the normal state, for example, by changing the display color or blinking. This allows the user to recognize at a glance that an abnormality has occurred in the function of the stereo camera with the stereo camera ID "StrCam1". At this time, as in the information 1002, the display surrounding the circle indicating the camera 131 is drawn with a double line different from the normal dashed circle. Also, as in the information 1003, for the camera 112 constituting the stereo camera together with the camera 131, the display surrounding the circle indicating the camera 112 is also drawn with a double line different from the normal dashed circle. With these pieces of information, the user can easily recognize that the stereo camera with the stereo camera ID "StrCam1" is not operating normally. Furthermore, the user can recognize from the information 1001 which of the multiple cameras displaying such a status is unable to operate normally.

[0040] FIG. 11 shows a second example of a screen display of a stereo camera configuration. In this example, the display positions of the cameras belonging to the stereo camera ID are displayed close to each other, so that the configuration of the stereo camera is presented to the user in an easy-to-understand manner. In FIG. 11, a circle 1101 indicates the camera 112, and a circle 1102 indicates the camera 131. In this embodiment, the position of the circle 1101 corresponding to the camera 112 is moved so as to be close to the circle 1102 corresponding to the camera 131, so that it is expressed that the camera 112 and the camera 131 are a combination of cameras belonging to one stereo camera. Furthermore, a frame 1103 is added so as to surround the circle 1101 and the circle 1102 corresponding to the camera 112 and the camera 131 forming the stereo camera corresponding to the stereo camera ID "StrCam1". Moreover, a circle 1104 represents the camera 121, a circle 1105 represents the camera 111, and a circle 1106 represents the camera 141. By moving the positions of the circle 1105 corresponding to the camera 111 and the circle 1106 corresponding to the camera 141 so as to be close to the circle 1104 corresponding to the camera 121, it is expressed that the cameras 111, 121, and 141 belong to one stereo camera. In addition, a frame 1107 is added so as to surround the circles 1104 to 1106 corresponding to the cameras 111, 121, and 141 forming the stereo camera corresponding to the stereo camera ID "StrCam2". As described above, the UI generating unit 405 may move the circles corresponding to the other cameras belonging to the stereo camera to the initial display position of the circle corresponding to one camera belonging to the stereo camera so as to be close to the camera, but is not limited to this. For example, the UI generating unit 405 may display the circles corresponding to the cameras belonging to the stereo camera corresponding to the stereo camera ID at a position defined in advance for each stereo camera ID. In addition, the UI generating unit 405 may detect an empty space where the circles corresponding to the cameras are not displayed, and may use the empty space to display the circles corresponding to the cameras belonging to the stereo camera in an aggregated manner. Furthermore, the UI generation unit 405 can display the circles corresponding to the cameras belonging to the stereo camera in close proximity to each other using any method other than those mentioned above.In addition, in response to the selection of a stereo camera ID, the UI generation unit 405 may dynamically change the UI screen so that circles indicating the cameras belonging to the stereo camera corresponding to that stereo camera ID are displayed closely together.

[0041] FIG. 12 shows an example of a screen when the camera 131 transitions from the display state of the screen shown in FIG. 11 to an abnormal state. In FIG. 12, information 1201 indicates that the camera 131 is in an abnormal state. Also, a frame 1202 corresponds to the frame 1103 in FIG. 11, but is drawn with double lines, unlike the frame 1103 during normal operation, in order to indicate that the stereo camera corresponding to the stereo camera ID "StrCam1" cannot operate normally due to an abnormality in the camera 131. This representation allows the user to easily recognize that the stereo camera function is not operating normally, that is, that the multiple cameras constituting the stereo camera cannot operate in cooperation with each other. Also, the user can easily identify which camera's abnormality is causing the abnormality in the stereo camera function. Note that the shape of the information 1201 indicating that the camera 131 is in an abnormal state and the double lines of the frame 1202 indicating that the stereo camera is in an abnormal state may be shown with other shapes or other line types. For example, the information 1201 and the frame 1202 may indicate an abnormality in the camera alone or in the stereo camera by changing color or blinking.

[0042] FIG. 13 shows a third example of a screen display of a stereo camera configuration. In this example, cameras constituting a stereo camera corresponding to each stereo camera ID are displayed as in list 1301. This allows the stereo camera configuration to be presented so that the user can recognize it at a glance. Note that in FIG. 13, list 1301 is displayed at the bottom of camera layout diagram 1302, but list 1301 may be displayed above, to the left, or to the right of camera layout diagram 1302. Also, list 1301 may be displayed superimposed on camera layout diagram 1302 using a dynamic window. Also, only list 1301 may be displayed.

[0043] FIG. 14 shows an example of a screen when the camera 131 transitions to an abnormal state from the display state of the screen shown in FIG. 13. Information 1401 indicates that the camera 131 has entered an abnormal state. Information 1402 indicates that information on the stereo camera ID "StrCam1" corresponding to the stereo camera in which the camera 131 is used is highlighted in a different color in response to the occurrence of an abnormality in the camera 131. That is, the information 1402 indicates that the stereo camera corresponding to the stereo camera ID "StrCam1" is not operating normally. This expression allows the user to easily identify which camera abnormality is causing the abnormality in the stereo camera function. Note that, in the example of FIG. 14, an example is shown in which the background color of the information on the stereo camera in an abnormal state is changed, but the character color may be changed, or the occurrence of the abnormal state may be indicated by blinking the characters.

[0044] FIG. 15 shows a fourth example of a screen display of a stereo camera configuration. In this example, all of the cameras constituting one stereo camera are displayed surrounded by a closed space-shaped frame. This allows the user to recognize the cameras belonging to the stereo camera at a glance. For example, in FIG. 15, a frame 1501 is drawn so as to surround the circles indicating the cameras 112 and 131 constituting the stereo camera corresponding to the stereo camera ID "StrCam1". Similarly, a frame 1502 is configured so as to surround the circles indicating the cameras 111, 121, and 141 constituting the stereo camera corresponding to the stereo camera ID "StrCam2". This display allows the user to easily recognize the cameras constituting the stereo camera corresponding to each stereo camera ID. Note that in FIG. 15, the frames 1501 and 1502 are shown with dashed lines, but these frames may be expressed in another format, such as a solid line or a double line. Also, for example, a display in which a plurality of circles indicating a plurality of cameras constituting the same stereo camera are connected by a special line may be used. For example, in a case where the physical connection relationship between each camera is shown by a black line, multiple cameras constituting one stereo camera may be connected by a red line.

[0045] FIG. 16 shows an example of a screen when the camera 131 transitions from the display state of the screen shown in FIG. 15 to an abnormal state. In FIG. 16, information 1601 indicates that the camera 131 is in an abnormal state. Also, a frame 1602 corresponds to the frame 1501 in FIG. 15, but is drawn with double lines, unlike the frame 1501 during normal operation, in order to indicate that the stereo camera corresponding to the stereo camera ID "StrCam1" cannot operate normally due to an abnormality in the camera 131. This representation makes it possible for a user to easily identify that the stereo camera function is not operating normally and which camera is causing the abnormality in the stereo camera function. Note that the shape of the information 1601 indicating that the camera 131 is in an abnormal state and the double lines of the frame 1602 indicating that the stereo camera is in an abnormal state may be shown with other shapes or other line types. For example, the information 1601 and the frame 1602 may indicate an abnormality in the camera alone or the stereo camera by changing the color or blinking.

[0046] In the above examples, the cameras are connected in a daisy chain. However, even in the case of a star connection in which each camera is directly connected to a HUB, the display can be performed as in the above examples. FIG. 17 shows an example of a screen in which the stereo camera configuration display similar to that in FIG. 9 is performed in a situation in which the cameras are star-connected to the HUB. Other display examples can be similarly applied in a situation in which the cameras are star-connected. The camera connection method is not limited to a daisy chain connection and a star connection, and some cameras may be directly connected to the HUB, and the remaining cameras may be connected to the HUB via other cameras. In addition, the HUB is an example of a camera connection destination, and in the case in which the camera is directly connected to the image generating device 160, the "HUB" may be replaced with the "image generating device". In addition, the "HUB" may be omitted in the UI screen. In any of these cases, the screen display method corresponding to each of the above examples can naturally be applied.

[0047] In this way, in a multi-camera system in which two or more cameras, which are a part of a large number of cameras, work together as a stereo camera, the user can easily check which camera is being used as the stereo camera. In addition, the above-mentioned display allows the user to easily recognize which stereo camera is affected by an abnormality in the case of an abnormality occurring in a camera.

[0048] In the above example, the on / off of the stereo camera display is determined by a user input, but the present invention is not limited to this. For example, the stereo camera configuration display may always be performed. In the above example, a two-dimensional screen is shown as an example of a UI screen displayed by the information display device 170, but the present invention is not limited to this. For example, a stereoscopic camera arrangement may be displayed using computer graphics, and the stereo camera configuration display may be performed on the screen as described above. In the above example, an example in which the entire state of the camera group 100 included in the system is displayed is shown, but only a part of the camera group 100 may be displayed by, for example, a zoom operation. At this time, only when a circle (sign) indicating the camera constituting the stereo camera is displayed on the screen, the additional information indicating the camera belonging to the stereo camera may be displayed. However, this is only one example, and for example, in a configuration such as that shown in FIG. 13, even if a circle indicating the camera constituting the stereo camera is not displayed on the screen, the additional information indicating the camera belonging to the stereo camera may be displayed. Furthermore, in the above processing example, an example in which additional information indicating that an abnormal state has occurred for a camera operating as a stereo camera has been described, but an abnormal state may also be indicated for a camera not operating as a stereo camera in a similar manner. Moreover, the above-mentioned display examples may be combined.

[0049] (Changing the cameras that make up the stereo camera) If an abnormality occurs in a camera used as a stereo camera, another camera may be used to configure the stereo camera instead of the camera in which the abnormality occurs. In this case, the UI generation unit 405 may generate a UI screen that displays that the stereo camera is configured using such an alternative camera.

[0050] FIG. 18 shows an example of data stored by the stereo camera information storage unit 402 when a stereo camera is used with such an alternative camera. In this example, an abnormality occurs in "camera 131" of the stereo camera ID "StrCam1", so a new stereo camera ID "StrCam1-upd" is created. In the stereo camera with this stereo camera ID "StrCam1-upd", "camera 121" is used instead of "camera 131". The alternative camera can be selected by a user operation.

[0051] The UI generating unit 405 may generate and display a screen that lists cameras previously determined as candidates for the alternative camera and prompts the user to select a camera to be actually used as the alternative camera, as shown in FIG. 19. The user determines one camera as the alternative camera, for example, by inputting a numeric key corresponding to one of the numbers assigned to each of the candidate cameras. Note that inputting a numeric key is only an example, and the user's selection may be accepted by other methods. For example, a button corresponding to each of the alternative camera candidates may be displayed on the screen, and the alternative camera may be determined by the user clicking one of the buttons by operating the mouse. Alternatively, the alternative camera may be determined by the user uttering information such as a number indicating one of the candidate cameras, and the voice being recognized by voice.

[0052] The UI generating unit 405 may generate a UI screen that presents the overlap rate of the angles of view of cameras that can be used in combination with the stereo camera and prompts the user to select an alternative camera, as shown in FIG. 20, and display the UI screen on the display unit 406. Here, information 2001 indicates the name of a camera that is not abnormal in the stereo camera and is used in combination with the alternative camera. The button 2002 is a button that accepts an operation to switch the camera that is used in combination with the alternative camera to the camera immediately before the camera being displayed. The button 2003 is a button that accepts an operation to switch the camera that is used in combination with the alternative camera to the camera immediately after the camera being displayed. The information 2004 is information that displays a list of candidates for alternative cameras that can be used as a stereo camera in combination with the camera shown in the information 2001. The information 2004 indicates, as an overlap rate, the size of an area where the angle of view of the alternative camera candidate at the time of shooting and the angle of view of the camera shown in the information 2001 overlap. 20, the alternative camera candidates are listed in descending order of overlap rate, but the present invention is not limited to this. For example, the alternative camera candidates may be displayed in descending order of the cameras physically located closest to the camera indicated in the information 2001. The alternative camera candidates may also be arranged in order of the camera identification information. In other words, the display order of the alternative camera candidates in the list may be determined according to any rule.

[0053] In one example, an abnormality occurs in camera 131, one of the cameras that constitute the stereo camera with stereo camera ID "StrCam1", and as a result another camera that constitutes the stereo camera in combination with camera 112 is specified. In this case, camera 112 is shown in information 2001. Then, cameras that have an overlapping area with the angle of view of camera 112 are presented in descending order in information 2004. Note that in this case, camera 112 is the only camera that constituted the stereo camera together with camera 131 with the abnormality, and therefore information 2001 does not display anything other than camera 112. For this reason, buttons 2002 and 2003 may be disabled.

[0054] In FIG. 20, "Overlap" indicates the overlap rate of the angle of view, and "Status" indicates the state of the camera. The overlap rate of the angle of view is calculated, for example, using calibration parameters that represent the posture and orientation of each camera. In FIG. 20, it is shown that the camera with the highest overlap rate of the angle of view with the camera 112 is the camera 131. Therefore, when no abnormality occurs in the camera 131, the camera 131 is used as a stereo camera together with the camera 112. On the other hand, in FIG. 20, since the camera 131 is in an abnormal state, the "Status" of the camera 131 is displayed as "NG". Therefore, the user can select an alternative camera by, for example, referring to the information on the overlap rate. For example, the user can select a camera to be used as an alternative camera by, for example, inputting a numeric key on a keyboard by referring to the information on the overlap rate. Note that in FIG. 20, the candidates for the alternative camera are shown in descending order of the overlap rate of the angle of view, but this is not limited thereto. For example, the candidates for the alternative camera may be presented in order from a list that is defined and recorded in advance.

[0055] Also, the alternative camera may be determined automatically, not based on a user instruction. For example, a camera that constitutes the stereo camera and is operating normally and has the highest overlap rate of the angle of view with a camera that is operating normally may be automatically selected as the alternative camera. In this case, a message inquiring about switching to an alternative camera is presented to the user, and switching to the alternative camera may be performed in response to the user's approval of the switching to the alternative camera. FIG. 21 shows an example of a display of a message inquiring the user about switching to an alternative camera. FIG. 21 shows an example of a message inquiring about switching to an alternative camera. Based on the detection of an abnormality in the camera 131, FIG. 21 shows an example of a message for confirming whether the camera 121, which has the second highest overlap rate of the angle of view with the camera 112 after the camera 131, may be used as an alternative camera for the camera 131. When such a message is presented, the user may instruct whether or not to permit the switching of the camera by, for example, inputting a numeric key on the keyboard. Then, after an instruction to permit the switching of the camera is accepted from the user, the camera that constitutes the stereo camera is changed. In this example, an example in which switching to an alternative camera is performed after an instruction from the user is accepted is shown, but this is not limited to this. For example, the camera may be automatically switched to an alternative camera and a message indicating the alternative camera after the switch may be presented to the user. That is, a message may be presented to the user informing the user that the switch will be performed automatically, without a message asking the user whether or not to permit the switch to the alternative camera.

[0056] FIG. 22 shows an example of a screen when the camera 131 transitions to an abnormal state from the display state of the screen shown in FIG. 9 and an alternative camera is used. In this example, it is shown that the camera 121 is used as an alternative because an abnormality has occurred in the camera 131. Information 2201 shows that the stereo camera with the stereo camera ID "StrCam1" has stopped functioning normally because an abnormality has occurred in the camera 131. Information 2202 shows that a stereo camera function is realized by a new combination of cameras as an alternative to the stereo camera ID "StrCam1". At this time, in order to show that the stereo camera with the stereo camera ID "StrCam1" is not functioning normally, the information of StrCam1 may be displayed in a color different from that during normal operation or may be grayed out. In addition, when the information of the stereo camera that is not selected is grayed out and the information of the stereo camera that is not functioning normally is also grayed out, the gray of the information of the stereo camera that is not functioning normally may be made darker. Naturally, display in other formats is also possible. The information 2202 may be displayed when the stereo camera with the stereo camera ID "StrCam1" stops functioning. The information 2202 may be displayed at all times. The information 2203 indicates that the camera 131 has become abnormal. The dashed circle 2204 surrounding the circle corresponding to the camera 131 indicates that the camera 131 was used as a stereo camera, but is not currently being used as a stereo camera due to an abnormality that has occurred. The dashed circles 2205 and 2206 surrounding the circles corresponding to the camera 112 and the camera 121 indicate that the camera 112 and the camera 121 are being used as a stereo camera with the stereo camera ID "StrCam1-upd". The circle 2204 may be displayed with a different color, shape, size, transparency, or the like to distinguish it from the circle 2205 and the circle 2206.

[0057] As described above, when an abnormality occurs in a camera used as a stereo camera and an alternative camera is to be used to configure the stereo camera, the user can easily recognize the situation. In addition, the user can accurately and quickly recognize the functional status of the stereo camera at the time when the alternative camera is used. In the above example, a method for determining an alternative camera to be used in place of a camera in which an abnormality occurs when an abnormality occurs in any of the cameras constituting the stereo camera has been described. This method can also be used to change a camera in which an abnormality does not occur. That is, for example, some or all of the cameras constituting the stereo camera may be changed according to the user's intention. For example, by using a screen display such as that shown in FIG. 20, it is possible to accept the user's selection of a camera to be used in conjunction with one of the cameras constituting the stereo camera while leaving that one camera.

[0058] In addition, in the above example, an example of a UI image showing the status of the group of cameras used by image generation device 160 is shown, but the status of image generation device 160 may also be shown. This allows the user to easily recognize the status of the entire image generation system.

[0059] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0060] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure.

[0061] (Summary of the embodiment) At least some of the above-described embodiments can be summarized as follows. (Item 1) A display control device comprising: a display control means for generating a UI image showing a user interface (UI) image of a multi-camera system having a plurality of imaging devices, in which information showing two or more imaging devices among the plurality of imaging devices that perform a predetermined coordinated operation in which only a portion of the plurality of imaging devices are used is displayed in a format different from that of imaging devices that do not perform the predetermined coordinated operation; and for controlling the display of the UI image on a display means. (Item 2) The display control device according to item 1, characterized in that the display control means generates the UI image including additional information that indicates a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices, and that surrounds each of the graphic representing the two or more imaging devices. (Item 3) The display control device according to item 1, characterized in that the display control means generates the UI image in which a figure representing each of the plurality of imaging devices and a connection relationship of the plurality of imaging devices with other devices are shown, and the figure representing the two or more imaging devices is drawn differently from a figure representing another imaging device that does not perform the specified coordinated operation. (Item 4) The display control device according to item 3, wherein the display control means generates the UI image in which the size, color, shape, or transparency of figures representing the two or more imaging devices is drawn to be different from that of figures representing other imaging devices that do not perform the specified coordinated operation. (Item 5) The display control device according to item 1, characterized in that the display control means generates the UI image in which a graphic representing each of the plurality of imaging devices, a character string representing identification information of each of the plurality of imaging devices, and a connection relationship between the plurality of imaging devices and other devices are drawn such that the display form of the character string for the two or more imaging devices is different from the display form of the character string for the other imaging devices that do not perform the specified cooperative operation. (Item 6) The display control device according to item 5, wherein the display control means generates the UI image in which a string of identification information for the two or more imaging devices is drawn as the string of identification information for the two or more imaging devices, the string of identification information for the two or more imaging devices being added to the string of identification information for the two or more imaging devices, or a string of identification information for the two or more imaging devices that does not perform the specified coordinated operation, the string of identification information for the two or more imaging devices being drawn in a different font than the string of identification information for the other imaging devices that do not perform the specified coordinated operation. (Item 7) The display control device according to item 1, characterized in that the display control means generates the UI image in which a figure representing each of the plurality of imaging devices and a connection relationship of the plurality of imaging devices with other devices are shown, and the figures for the two or more imaging devices are displayed closer together than in a case in which the two or more imaging devices do not perform the specified coordinated operation. (Item 8) The display control device according to any one of items 1 to 8, characterized in that the display control means generates the UI image showing a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices, and displaying a list of information about the two or more imaging devices that perform the specified coordinated operation. (Item 9) The display control device according to any one of items 1 to 8, characterized in that the display control means generates the UI image in which a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices are displayed, and a frame is displayed surrounding the graphic representing each of the two or more imaging devices. (Item 10) The display control device according to any one of items 1 to 8, wherein the display control means generates the UI image in which a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices are displayed, and lines connecting the graphic representing each of the two or more imaging devices are displayed. (Item 11) The display control device according to any one of items 1 to 11, characterized in that, when there are multiple combinations of the two or more imaging devices that perform the specified collaborative operation, the display control means generates the UI image in which information indicating the two or more imaging devices included in the combination selected by the user is displayed in a format different from that of the imaging devices that do not perform the specified collaborative operation, and information indicating the two or more imaging devices included in the combination not selected by the user is displayed in the same format as that of the imaging devices that do not perform the specified collaborative operation. (Item 12) 12. The display control device according to any one of items 1 to 11, characterized in that the display control means generates the UI image showing additional information indicating that any one of the plurality of imaging devices has entered an abnormal state based on the abnormal state of the imaging devices. (Item 13) The display control device according to item 12, characterized in that, based on an abnormality occurring in any of the two or more imaging devices performing the specified coordinated operation, the display control means generates the UI image further displaying information indicating that the two or more imaging devices are unable to perform the specified coordinated operation. (Item 14) 13. The display control device according to item 12, further comprising an acquisition means for acquiring information indicating a state of each of the plurality of imaging devices. (Item 15) The display control means generating a screen for accepting a change in a combination of the two or more imaging devices performing the predetermined coordinated operation, the screen indicating an overlap rate between an angle of view of shooting by one imaging device and an angle of view of shooting by the other imaging device, for each of the two or more imaging devices that can be used in combination with the one imaging device, and displaying the screen on the display means; generating the UI image after the change when a user's operation indicating a change in the combination of the two or more imaging devices performing the predetermined cooperative operation on the screen is accepted; 15. A display control device according to any one of items 1 to 14. (Item 16) a selection means for selecting, when a combination of the two or more imaging devices performing the predetermined coordinated operation is to be changed, an imaging device that performs the predetermined coordinated operation with one of the two or more imaging devices from among other imaging devices that can perform the predetermined coordinated operation with the one imaging device, based on an overlap rate between an angle of view of shooting by the one imaging device and an angle of view of shooting by the other imaging device; 15. A display control device according to any one of items 1 to 14. (Item 17) 17. The display control device according to any one of items 1 to 16, further comprising a storage means for storing information indicating two or more of the plurality of imaging devices that are operating in cooperation with each other among the plurality of imaging devices. (Item 18) 18. The display control device according to any one of items 1 to 17, further comprising the display means. (Item 19) The display control device described in any one of items 1 to 18, characterized in that the multi-camera system is a system that generates a virtual viewpoint image using the multiple imaging devices, the multiple imaging devices are used to generate a three-dimensional model of a subject, and two or more of the multiple imaging devices perform imaging including determining a distance to a surface of the subject as the specified collaborative operation. (Item 20) A display control method executed by a display control device, comprising: generating a user interface (UI) image showing a multi-camera system including a plurality of imaging devices, in which information showing two or more imaging devices performing a predetermined cooperative operation in which only a portion of the plurality of imaging devices is used is displayed in a format different from that of imaging devices not performing the predetermined cooperative operation; performing control to display the UI image on a display means; A display control method comprising: (Item 21) A program for causing a computer to function as a display control device according to any one of items 1 to 19. [Explanation of symbols]

[0062] 10: image generation system, 100: camera group, 170: information display device, 401: data storage unit, 402: stereo camera information storage unit, 404: status acquisition unit, 405: UI generation unit, 406: display unit

Claims

1. A display control device characterized by having a display control means for generating a user interface (UI) image showing a multi-camera system equipped with a plurality of imaging devices, in which information showing two or more imaging devices among the plurality of imaging devices that perform a predetermined cooperative operation in which only a portion of the plurality of imaging devices are used is displayed in a format different from that of imaging devices that do not perform the predetermined cooperative operation, and for controlling the display of the UI image on a display means.

2. 2. The display control device according to claim 1, wherein the display control means generates the UI image including additional information that indicates a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices, and that surrounds each of the graphic representing the two or more imaging devices.

3. 2. The display control device according to claim 1, wherein the display control means generates the UI image showing a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices, and the graphic representing the two or more imaging devices is drawn differently from a graphic representing another imaging device that does not perform the predetermined cooperative operation.

4. The display control device according to claim 3, characterized in that the display control means generates the UI image in which the size, color, shape, or transparency of the figures representing the two or more imaging devices is drawn so as to be different from that of the figures representing other imaging devices that do not perform the specified cooperative operation.

5. 2. The display control device according to claim 1, wherein the display control means generates the UI image including a graphic representing each of the plurality of imaging devices, a character string representing identification information of each of the plurality of imaging devices, and a connection relationship between the plurality of imaging devices and other devices, and wherein the display form of the character string for the two or more imaging devices is drawn differently from the display form of the character string for other imaging devices that do not perform the predetermined cooperative operation.

6. The display control device according to claim 5, characterized in that the display control means generates the UI image in which the character strings for the two or more imaging devices are drawn as character strings in which a character string of identification information corresponding to a combination of the two or more imaging devices is added to a character string of identification information of each of the two or more imaging devices, or a character string in which a font different from a font of a character string for other imaging devices that do not perform the specified collaborative operation is changed.

7. 2. The display control device according to claim 1, wherein the display control means generates the UI image in which a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices is displayed such that the graphic representing each of the plurality of imaging devices is displayed closer to each other than when the two or more imaging devices do not perform the predetermined cooperative operation.

8. 2. The display control device according to claim 1, wherein the display control means generates the UI image that shows a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices, and that displays a list of information about the two or more imaging devices that perform the predetermined cooperative operation.

9. 2. The display control device according to claim 1, wherein the display control means generates the UI image drawn so that a frame is displayed surrounding the figures for the two or more imaging devices, the figure showing each of the plurality of imaging devices and the connection relationship between the plurality of imaging devices and other devices.

10. 2. The display control device according to claim 1, wherein the display control means generates the UI image drawn so that a graphic representing each of the plurality of imaging devices and a connection relationship between the plurality of imaging devices and other devices are displayed, and lines connecting the graphic representing each of the two or more imaging devices are displayed.

11. 2. The display control device according to claim 1, wherein, when there are multiple combinations of the two or more imaging devices that perform the predetermined collaborative operation, the display control means generates the UI image in which information indicating the two or more imaging devices included in the combination selected by the user is displayed in a format different from that of imaging devices that do not perform the predetermined collaborative operation, and information indicating the two or more imaging devices included in combinations not selected by the user is displayed in the same format as that of imaging devices that do not perform the predetermined collaborative operation.

12. 2. The display control device according to claim 1, wherein the display control means generates the UI image showing additional information indicating that any one of the plurality of imaging devices has entered an abnormal state, based on the abnormal state of the imaging devices.

13. 13. The display control device according to claim 12, wherein the display control means, based on an abnormal state of any of the two or more imaging devices performing the predetermined cooperative operation, generates the UI image on which information indicating that the two or more imaging devices are unable to perform the predetermined cooperative operation is displayed.

14. 13. The display control device according to claim 12, further comprising an acquisition unit that acquires information indicating the state of each of the plurality of image capture devices.

15. The display control means generating a screen for accepting a change in the combination of the two or more imaging devices performing the predetermined cooperative operation, wherein another imaging device that can be used in combination with one of the two or more imaging devices generates a screen indicating an overlap rate between an angle of view of imaging by the one imaging device and an angle of view of imaging by the other imaging device, and displaying the screen on the display means; When a user's operation indicating a change in the combination of the two or more imaging devices that perform the predetermined cooperative operation is accepted on the screen, the UI image after the change is generated.

2. The display control device according to claim 1.

16. a selection unit for selecting, when a combination of the two or more imaging devices performing the predetermined cooperative operation is to be changed, an imaging device that will perform the predetermined cooperative operation with one of the two or more imaging devices from among other imaging devices that can perform the predetermined cooperative operation with the one imaging device, based on an overlap rate between an angle of view of the one imaging device and an angle of view of the other imaging device; 2. The display control device according to claim 1.

17. 2. The display control device according to claim 1, further comprising: a storage unit for storing information indicating two or more of the plurality of image capture devices that are operating in cooperation with each other.

18. 2. The display control device according to claim 1, further comprising the display means.

19. The display control device according to claim 1, characterized in that the multi-camera system is a system that generates a virtual viewpoint image using the plurality of imaging devices, the plurality of imaging devices are used to generate a three-dimensional model of a subject, and two or more of the plurality of imaging devices perform imaging including specifying a distance to a surface of the subject as the predetermined cooperative operation.

20. A display control method executed by a display control device, comprising: generating a user interface (UI) image showing a multi-camera system including a plurality of imaging devices, in which information showing two or more imaging devices that perform a predetermined cooperative operation in which only a portion of the plurality of imaging devices is used is displayed in a format different from that of imaging devices that do not perform the predetermined cooperative operation; performing control to display the UI image on a display means; A display control method comprising:

21. A program for causing a computer to function as the display control device according to any one of claims 1 to 19.

Images