Information processing apparatus

Abstract

To provide an information processing apparatus that, when displaying an image from a wide-angle imaging unit and an image from an imaging unit having a narrower angle of view than the wide-angle imaging unit on one display, facilitates understanding of the positional relationship between the image from the wide-angle imaging unit and the image from the imaging unit other than the wide-angle imaging unit even when display of the image from the wide-angle imaging unit is rotated.SOLUTION: An information processing apparatus has: first acquisition means that acquires a first image from a first imaging unit that performs imaging at a predetermined angle of view; second acquisition means that acquires a second image from a second imaging unit that picks up an image of an area overlapping an area picked up by the first imaging unit at an angle of view narrower than the predetermined angle of view; and control means that displays the first image and the second image on a screen of a display. When the first image is rotated by a predetermined angle on the screen of the display, the control means changes the display position of the second image on the screen of the display on the basis of the predetermined angle.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Imaging devices incorporating multiple cameras in one housing are known. For example, Patent Document 1 describes an imaging device equipped with a PTZ camera and a predetermined number of wide-area cameras arranged around it. In this imaging device, the predetermined number of wide-area cameras can capture images of the surrounding situation, and the PTZ camera arranged in the center can focus on and capture an object of interest. The "P" in PTZ stands for Panoramic, the "T" stands for Tilt, and the "Z" stands for Zoom.

[0003] In Patent Document 1, when the pan / tilt angle of the PTZ camera is changed, the display of the image from the wide area camera is changed according to the imaging direction of the PTZ camera in order to make it easier to recognize the imaging positional relationship between the wide area camera and the PTZ camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-140566 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, when the display (image) of the wide-area camera is rotated, it may become difficult to recognize the positional relationship between the wide-area camera and the PTZ camera.

[0006] Therefore, an object of the present invention is to provide an information processing device that, when displaying an image from a wide-angle imaging unit and an image from an imaging unit with a narrower angle of view than the wide-angle imaging unit on a single display unit, makes it easy to understand the positional relationship with images from imaging units other than the wide-angle imaging unit even when the image display of the wide-angle imaging unit is rotated. [Means for solving the problem]

[0007] In order to solve the above problem, an information processing device according to one embodiment of the present invention is an information processing device having a first acquisition means for acquiring a first image from a first imaging unit that captures images at a predetermined angle of view, a second acquisition means for acquiring a second image from a second imaging unit that captures an area overlapping with the first imaging unit at an angle of view narrower than the predetermined angle of view, and a control means for displaying the first image and the second image on a screen of a display unit, wherein when the first image is rotated by a predetermined angle on the screen of the display unit, the control means changes the display position of the second image on the screen of the display unit based on the predetermined angle. Effect of the Invention

[0008] According to the present invention, when an image from a wide-angle imaging unit and an image from an imaging unit with a narrower angle of view than the wide-angle imaging unit are displayed on a single display unit, even if the image display of the wide-angle imaging unit is rotated, it is easy to understand the positional relationship with the images from imaging units other than the wide-angle imaging unit. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a system including a terminal device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a terminal device in FIG. [Diagram 3] FIG. 2 is a schematic diagram illustrating the appearance of the imaging device in FIG. [Figure 4] FIG. 4 is a flowchart showing a processing procedure in the first embodiment. [Diagram 5] 2 is a diagram illustrating a GUI screen of the terminal device of FIG. 1. [Figure 6] FIG. 6 is a diagram illustrating an example of how the GUI in FIG. 5 is displayed on a screen. [Figure 7] FIG. 7 is a diagram illustrating the GUI screen when the image is rotated from the state in FIG. 6. [Figure 8] 13 is a schematic external view of an imaging device according to a second embodiment of the present invention. [Figure 9]FIG. 11 is a block diagram illustrating a system including a terminal device according to a second embodiment. [Figure 10] FIG. 11 is a flowchart showing a processing procedure in the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a display on a GUI screen in the second embodiment. [Figure 12] FIG. 12 is a diagram illustrating the GUI screen when the image is rotated from the state in FIG. 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). The technical scope of the present invention is determined by the claims, and is not limited by the individual embodiments below. Also, parts of each embodiment may be combined as appropriate. In all figures, parts having the same function are given the same reference symbols, and repeated explanations thereof will be omitted.

[0011] First embodiment 1 shows a surveillance system 100 according to an embodiment of the present invention. The surveillance system 100 includes an imaging device 1000 and a terminal device 1600 connected to the imaging device 1000 via a network 1500. The terminal device 1600 is operated by, for example, a user of the surveillance system 100. The terminal device 1600 is, for example, a personal computer.

[0012] The imaging device 1000 includes an imaging unit 1001, an imaging unit 1002, an imaging unit 1003, an imaging unit 1004, an imaging unit 1005, a signal processing unit 1006, and a distribution unit 1007. The imaging units 1001 to 1005 are connected to the signal processing unit 1006. The signal processing unit 1006 is connected to the distribution unit 1007. The imaging unit 1001 is composed of a lens 1011 and an imaging element such as a CCD or CMOS (not shown in FIG. 1), and converts light incident from the lens 1011 into an electrical signal by the imaging element. The imaging unit 1001 is a wide-angle camera. The lens 1011 of the imaging unit 1001 in this embodiment is an ultra-wide-angle lens such as a fisheye lens, and has a horizontal angle of view of about 180 degrees. In this embodiment, the imaging unit 1001 is an omnidirectional camera (a camera capable of imaging in all directions). The imaging units 1002 to 1005 are gazing cameras that capture an object (object of interest) in an omnidirectional image captured by the imaging unit 1001. In other words, the imaging units 1002 to 1005 capture an area overlapping with the imaging unit 1001. In this embodiment, it is assumed that the areas captured by the four imaging units 1002 to 1005 do not overlap. CCD stands for Charge Coupled Device. CMOS stands for Complementary Metal Oxide Semiconductor.

[0013] The imaging units 1002 to 1005 have the same structure as the imaging unit 1001. Specifically, the imaging unit 1002 includes a lens 1012 and an imaging element (not shown), the imaging unit 1003 includes a lens 1013 and an imaging element (not shown), the imaging unit 1004 includes a lens 1014 and an imaging element (not shown), and the imaging unit 1005 includes a lens 1015 and an imaging element (not shown). The lenses 1012 to 1015 all have the same angle of view, and the horizontal angle of view is narrower than that of the lens 1011. In other words, the imaging units 1002 to 1005 are cameras with a narrower angle of view than the imaging unit 1001, which is a wide-angle camera. In this embodiment, the angles of view of the imaging units 1002 to 1005 are fixed.

[0014] A signal processing unit 1006 performs predetermined image processing and compression encoding processing on the electrical signals output from the imaging units 1001 to 1005, and generates image data. Furthermore, the signal processing unit 1006 can transmit the setting values ​​of the imaging conditions to the imaging units 1001 to 1005 to change the operation state of each of the imaging units 1001 to 1005. The imaging conditions include gain conditions, gamma conditions, dynamic range conditions, exposure conditions, focus conditions, and the like.

[0015] The distribution unit 1007 is connected to the network 1500 , converts the image data output from the signal processing unit 1006 into an appropriate signal format, and transmits it to the network 1500 . The network 1500 transmits information (including images) received from the imaging device 1000 to the terminal device 1600. The terminal device 1600 receives images captured by the imaging units 1001-1005 of the imaging device 1000, thereby acquiring the images captured by the imaging units 1001-1005. Furthermore, the network 1500 transmits information received from the terminal device 1600 to the imaging device 1000. When information from the terminal device 1600 (e.g., information input by a user) is input to the imaging device 1000 via the network 1500, the distribution unit 1007 receives the information, converts it into an appropriate format, and transmits the converted information (signal) to the signal processing unit 1006. The network 1500 may be any network that allows data to be exchanged between any number of devices. In this embodiment, the network 1500 is a wired LAN network that complies with Ethernet (registered trademark). LAN is an abbreviation for Local Area Network.

[0016] Some of the functional modules (1006, 1007) included in the imaging device 100 shown in FIG. 1 may be realized by hardware. In the case of hardware realization, for example, a specific compiler may be used to automatically generate a dedicated circuit on an FPGA from a program for realizing the function of each functional module. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as an FPGA and realized as hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit). Note that the configuration of the functional modules shown in FIG. 1 is an example, and a plurality of functional modules may constitute one functional module, or any of the functional modules may be divided into modules performing a plurality of functions.

[0017] 2(a) is a block diagram showing the hardware configuration of a terminal device 1600. The terminal device 1600 has a CPU 1601, a main memory device 1602, an auxiliary memory device 1603, an external I / F 1604, a GPU 1605, an input unit 1606, and a display unit 1607. The units (1601 to 1607) of the terminal device 1600 are connected to each other via a bus 1608 so as to be able to communicate with each other. The terminal device 1600 processes various types of information and may therefore be called an information processing device. The CPU 1601 is a central processing unit that is responsible for controlling the entire terminal device 1600, and performs various calculations, image processing, data input / output, etc. CPU is an abbreviation for Central Processing Unit.

[0018] The main memory device 1602 is a storage device that functions as a work area for the CPU 1601 and a temporary storage area for data. The main memory device 1602 is implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0019] The auxiliary storage device 1603 is a storage device that stores information such as various programs, various setting information, various image data, camera parameters, three-dimensional shape data, and two-dimensional maps. The auxiliary storage device 1603 is implemented using storage media such as nonvolatile memory such as a read only memory (ROM) or flash memory, a hard disk drive (HDD), a solid state drive (SSD), and a tape medium. Note that in order to increase capacity and speed, multiple nonvolatile memories, HDDs, and SSDs may be combined, or may be logically combined into one such as a redundant array of independent disks (RAID).

[0020] The external I / F 1604 is a general-purpose input / output interface. The external I / F 1604 includes a communication interface used for communication with the imaging device 1000. An external device that accepts various operations from a user may be connected to the I / F 1604. In addition, an external storage may be connected to the I / F 1604 to input and output files to and from the terminal device 1600. I / F is an abbreviation for Interface. The external I / F 1604 is configured with an interface having a physical cable connection terminal such as InfiniBand, Ethernet, or USB, or a wireless interface such as wireless LAN or Bluetooth (registered trademark). A signal input from the network 1500 is transmitted to the CPU 1601 via the Ethernet interface of the external I / F 1604. USB is an abbreviation for Universal Serial Bus.

[0021] The GPU 1605 is a computing device for performing image processing calculations at high speed, and converts the image (video) received from the imaging device 1000 into an appropriate format and outputs it. GPU is an abbreviation for Graphics Processing Unit. The GPU 1605 may include a separate main memory device different from the main memory device 1602. In addition, the GPU 1605 may perform some or all of the various calculations performed by the CPU 1601 other than those described above. The input unit 1606 includes a keyboard, a mouse, a joystick, and the like. A user can operate the input unit 1606 to input desired instructions, data, and the like to the terminal device 1600. User instructions, and the like input from the input unit 1606 are converted into appropriate signals and transmitted to the imaging device 1000. The input unit 1606 may be referred to as an operation unit since it is operated by the user. Note that the input unit 1606 does not need to be provided inside the terminal device 1600. In that case, the input unit (keyboard, mouse, joystick, and the like) 1606 is connected to the terminal device 1600 via the external I / F 1604.

[0022] The display unit 1607 is composed of a liquid crystal monitor or the like. The display unit 1607 displays an image transmitted from the imaging device 1000 to the terminal device 1600, allowing the user to view the image. The display unit 1607 can also display information input by the user from the input unit 1606. Since the display unit 1607 displays (outputs) an image, it may be called an output unit. The display unit 1607 may function as a touch panel. In this case, it may be said that the display unit 1607 functions as a part of the input unit 1606. When the display unit 1607 functions as a touch panel, the display unit 1607 becomes a graphical user interface (GUI).

[0023] The bus 1608 is configured by a bus such as PCI-Express, and allows each part of the hardware block (1601 to 1607) to perform bidirectional communication with each other. PCI is an abbreviation for Peripheral Component Interconnect. Note that the bus 1608 may be any bus that allows each part of the hardware block to perform bidirectional communication.

[0024] 2B is a block diagram showing the functional configuration of the terminal device 1600. The terminal device 1600 includes a system control unit 1701, a display control unit 1702, and an operation control unit 1703. The system control unit 1701 communicates with external devices such as the imaging device 1000, controls each function of the terminal device 1600, sets various parameters, and so on. More specifically, the system control unit 1701 executes processes such as receiving an image transmitted from the imaging device 1000, issuing an instruction to display the received image, and accepting an input based on an operation via the input unit 1606. The display control unit 1702 displays information corresponding to the instruction on the monitor of the display unit 1607 (screen 3000 in FIG. 5) in response to an instruction from the system control unit 1701. The display control unit 1702 also displays images transmitted from the imaging units 1001 to 1005 of the imaging device 1000 on the monitor of the display unit 1607. The user can check the image captured by the imaging device 1000 by visually checking the monitor of the display unit 1607. The operation control unit 1703 accepts an instruction from a user based on the user's operation via the input unit 1606, and transmits information on the accepted instruction to the system control unit 1701. The operation control unit 1703 receives an instruction to drive the wide-angle camera 1001 from the user based on, for example, an operation by the user via the input unit 1606, and transmits information on the received instruction to the system control unit 1701. Upon receiving an instruction to drive the wide-angle camera 1001, the system control unit 1701 transmits the received instruction to the imaging device 1000, thereby instructing the imaging device 1000 to drive the wide-angle camera 1001.

[0025] FIG. 3 is a schematic diagram showing the arrangement of the imaging units 1001 to 1005 in the imaging device 1000. FIG. 3(a) is a side view of the imaging device 1000, and FIG. 3(b) is a top view of the imaging device 1000. The imaging device 1000 has a base 1000a, a cover 1000b, and a support 1000c. The support 1000c is provided on the base 1000a. The cover 1000b is provided on the base 1000a so as to cover the support 1000c. The imaging unit 1001 is provided on the top of the support 1000c. The imaging units 1002 to 1005 are provided on the periphery of the support 1000c. The imaging units 1001 to 1005 are housed in a space partitioned by the base 1000a and the cover 1000b. 3(b), when the imaging device 1000 is viewed from above, the imaging units 1002-1005 are arranged concentrically around the imaging unit 1001, and the imaging units 1002-1005 are arranged at 90 degree intervals. In other words, the imaging units 1002-1005, which capture an area overlapping with the imaging unit 1001 at an angle of view narrower than that of the imaging unit 1001, are arranged around the imaging unit 1001. The optical axis directions of the imaging units 1001-1005 are fixed. The configuration of the imaging device 1000 is not limited to the configuration shown in Figures 2 and 3. The configurations shown in Figures 2 and 3 are merely examples, and may be appropriately changed and configured according to the imaging subject and functions.

[0026] Hereinafter, we will explain how an image from the imaging unit (wide-angle camera) 1001 and images from the four imaging units (multiple-lens cameras) 1002 to 1005 arranged around it are displayed on the screen 3000 of a single display unit 1607 in the imaging system 100 of the first embodiment. 4 is a flow diagram for explaining the operation and processing of the terminal device 1600. This flow illustrates a processing procedure executed by controlling each processing block 1701-1703 by the CPU 1601 of the terminal device 1600. This flow is realized by the CPU 1601 expanding and executing a program stored in the auxiliary storage device 1603. It is assumed that the terminal device 1600 acquires images captured by the imaging units 1001-1005 by receiving them from the imaging device 1009. It is also assumed that the display unit 1607 of the terminal device 1600 functions as a GUI.

[0027] When this flow starts, in S100, the CPU 1601 determines the display positions of the images of the imaging units 1001-1005 on the screen (display area) 3000 of the display unit (GUI) 1607. The display positions of the images are determined based on the imaging areas of the imaging units 1001-1005.

[0028] FIG. 5 shows a screen 3000 of the display unit (GUI) 1607 of the terminal device 1600. Five areas 3001, 3002, 3003, 3004, and 3005 are set on the screen 300. The area 3001 is a circular area, and the areas 3002 to 3005 are rectangular areas. The area 3001 is at the center of the screen 3000, and the areas 3002 to 3005 are around the area 3001. More specifically, the area 3002 is at the top left of the screen 3000, the area 3003 is at the top right of the screen 3000, the area 3004 is at the bottom right of the screen 3000, and the area 3005 is at the bottom left of the screen 3000. An image of the imaging unit 1001 is displayed in the area 3001. Any of the images of the imaging units 1002 to 1005 is displayed in the areas 3002 to 3005. A screen 3000 of the display unit 1607 has display areas 3002-3005 set in the same number as the number of the imaging units 1002-1005 (four in this embodiment) in order to display the images of the imaging units 1002-1005 individually.

[0029] Next, the determination of the display position will be described. The area 3001 is divided into areas 3001a, 3001b, 3001c, and 3001d in advance. The areas 3001a to 3001d are quarter-circular areas obtained by dividing the area 3001 into four. The CPU 1601 calculates the center position of the imaging areas of the imaging units 1002 to 1005 in the image of the imaging unit 1001, and determines whether the center position is located in the area 3001a, the area 3001b, the area 3001c, or the area 3001d. If the center position of the imaging area is within the area 3001a, the image of the imaging unit is displayed in the area 3002. If the center position of the imaging area is within the area 3001b, the image of the imaging unit is displayed in the area 3003. If the center position of the imaging area is within the area 3001c, the image of the imaging unit is displayed in the area 3004. If the center position of the imaging area is within area 3001d, the image of that imaging unit is displayed in area 3005. For example, if the imaging area of ​​one of the imaging units 1002 to 1005 is area 3500 and the center of area 3500 is point X, the image of that imaging unit is displayed in area 3002. The center of the imaging area in this embodiment is the coordinates of the optical axis of the imaging unit projected onto the imaging area. Note that the center of the imaging area is not limited to this. The center of the imaging area may be any point appropriately set within the imaging area (for example, the center of gravity of a person (subject) in the image).

[0030] The display positions determined in S100 are shown in FIG. 6. It is assumed that the subjects captured by the imaging units 1002 to 1005 are people. In the example of FIG. 6, the center of the imaging area 3502 of the imaging unit 1002 is set as point A, the center of the imaging area 3503 of the imaging unit 1003 is set as point B, the center of the imaging area 3504 of the imaging unit 1004 is set as point C, and the center of the imaging area 3505 of the imaging unit 1005 is set as point D. Since point A is located in area 3001a, the CPU 1601 determines that the image of the imaging unit 1002 is to be displayed in area 3002. Since point B is located in area 3001b, the CPU 1601 determines that the image of the imaging unit 1003 is to be displayed in area 3003. Since point C is located in area 3001c, the CPU 1601 determines that the image of the imaging unit 1004 is to be displayed in area 3004. Since point D is located in area 3001 d , the CPU 1601 determines that the image from the imaging unit 1005 is to be displayed in area 3005 .

[0031] In S101, the CPU 1601 displays the captured images of the imaging units 1002 to 1005 on the screen 3000 of the display unit (GUI) 1607 according to the display positions determined in S100. Specifically, the image of the imaging unit 1002 is displayed in the area 3002, the image of the imaging unit 1003 is displayed in the area 3003, the image of the imaging unit 1004 is displayed in the area 3004, and the image of the imaging unit 1005 is displayed in the area 3005. Also, the captured image of the imaging unit 1001 is displayed in the area 3001, and the imaging areas 3502 to 3505 of the imaging units 1002 to 1005 are displayed on the image. The imaging unit 1001 captures an image overlooking the whole (omnidirectional image), and the image of the imaging unit 1001 is complemented by the images of the imaging units 1002 to 1005 in areas (peripheral areas) where the details may be unclear. In FIG. 6, details of the image captured by the imaging unit 1001 (parts other than the subject) are omitted.

[0032] Next, the process in S102 will be described. The image of the imaging unit 1001 in the GUI screen 3000 can be rotated and displayed at any angle around point O, which is the center of the displayed image. In this embodiment, it is assumed that the user can input an instruction to rotate the image of the imaging unit 1001 clockwise (or counterclockwise) by angle θ via the input unit 1606. In S102, the CPU 1601 judges whether an instruction to rotate the image has been input from the user. If the judgment result is Yes, the process proceeds to S103, where the CPU 1601 analyzes the instruction (input signal), rotates the image of the imaging unit 1001 by angle θ, and displays it in the area 3001 of the GUI screen 3000. After rotating the image of the imaging unit 1001, the process proceeds to S104. If a rotation instruction from the user is not detected in S102, the process returns to S101. In the following description, it is assumed that a user inputs an instruction to rotate the image captured by the image capturing unit 1001 clockwise by an angle θ.

[0033] In S104, the CPU 1601 changes (determines) the display positions of the images of the imaging units 1002 to 1005 on the screen 3000 according to the rotation angle θ input by the user. When the image of the imaging unit 1001 is rotated clockwise by the angle θ, as shown in FIG. 7(a), the point A also rotates clockwise by the angle θ and moves to the point A1. The point B also rotates clockwise by the angle θ and moves to the point B1, the point C also rotates clockwise by the angle θ and moves to the point C1, and the point D also rotates clockwise by the angle θ and moves to the point D1. After the rotation, the point A1 is located in the area 3001b, so the CPU 1601 determines that the image of the imaging unit 1002 is to be displayed in the area 3003. The point B1 is located in the area 3001c, so the CPU 1601 determines that the image of the imaging unit 1003 is to be displayed in the area 3004. Since point C1 is located in area 3001d, the CPU 1601 determines that the image of the imaging unit 1004 is to be displayed in area 3005. Since point D1 is located in area 3001a, the CPU 1601 determines that the image of the imaging unit 1005 is to be displayed in area 3002. 7B, in S105, the CPU 1601 displays the images on the GUI screen 3000 at the display positions determined in S104. Specifically, the CPU 1601 displays the image of the imaging unit 1002 in an area 3003, the image of the imaging unit 1003 in an area 3004, the image of the imaging unit 1004 in an area 3005, and the image of the imaging unit 1005 in an area 3002. After performing the above processes, this flow ends.

[0034] According to the imaging device 1000 of this embodiment, in the monitoring device 1000 combining the multiple lens cameras 1002-1005 and the wide-angle camera 1001, the display positions of the multiple lens cameras 1002-1005 can be changed and displayed in response to a change in the display of the wide-angle camera 1001. If the images of the multiple lens cameras 1002-1005 remain unchanged when the image of the wide-angle camera 1001 is rotated, it becomes difficult to understand the correspondence (relationship) between the images of the wide-angle camera 1001 and the images of the multiple lens cameras 1002-1005. In this embodiment, the positions of the images of the multiple lens cameras 1002-1005 are changed in response to the rotation of the image of the wide-angle camera 1001, so that the correspondence between the images is easy to understand. In the above embodiment, the number of multi-lens cameras is four, but the number is not limited to four. In addition, the terminal device 1600 includes the display unit 1607, but the display unit 1607 may be provided as an independent device outside the terminal device 1600. In this case, the display unit (display device) 1607 is connected to the external I / F 1604.

[0035] Second embodiment A second embodiment of the present invention will be described below with reference to Figs. 8 to 12. In the second embodiment, an imaging device has one wide-angle camera (omnidirectional camera) and multiple gazing cameras with variable angle of view. In the second embodiment, a user specifies a subject from an image (video) captured by the wide-angle camera, and the image from the wide-angle camera rotates on the screen of the display unit 1607 in response to the specification, and the display position of the image from the gazing camera also changes. Note that the same reference symbols are used for configurations similar to those of the first embodiment, and descriptions thereof may be omitted.

[0036] 8 is a schematic diagram of an imaging device 2000 used in the second embodiment. The imaging device 2000 has a base 2000a, a cover 2000b, and a support 2000c provided on the base 2000a. The support 2000c is provided on the base 2000a. The cover 2000b is provided on the base 2000a so as to cover the support 2000c. The imaging device 2000 has imaging units 2001-2005. The imaging unit 2001 is an omnidirectional camera. The imaging unit 2001 is provided on the upper surface of the support unit 2000c. The imaging units 2002-2005 are gaze cameras. The imaging units 2002-2005 are provided on the upper surface of the base unit 2000a. The imaging units 2001-2005 are housed in a space partitioned by the base unit 2000a and the cover unit 2000b.

[0037] The imaging unit 2001 has the same configuration as the imaging unit 1001 of the first embodiment. The imaging units 2002 to 2004 are arranged at equal intervals around the imaging unit 2001, with the imaging unit 2001 at the center, as in the first embodiment. Each of the imaging units 2002 to 2005 is defined with an X-axis, a Y-axis, and a Z-axis. The Z-axis is the optical axis direction of each of the imaging units 2002 to 2005, and the X-axis and the Y-axis are axes perpendicular to the Z-axis. FIG. 8 shows the X-axis 2003a, the Y-axis 2003b, and the Z-axis 2003c of the imaging unit 2003. Each of the imaging units 2002 to 2005 can change its angle around the X-axis, the Y-axis, and the Z-axis. The angle can be changed independently for each axis (X-axis, Y-axis, and Z-axis). In the case of the imaging unit 2003, rotation around the X-axis corresponds to the panning operation of the imaging unit 2003, and rotation around the Y-axis corresponds to the tilting operation of the imaging unit 2003. Each of the imaging units 2002-2005 can change the imaging direction by changing the angle around the respective X-axis and Y-axis. The imaging units 2002-2005 may be called movable imaging units. The imaging unit 2001 may be called fixed imaging unit. The imaging units 2002-2005 can also change the imaging range by a zoom function installed in a lens (not shown).

[0038] FIG. 9 shows a monitoring system 200 according to this embodiment. The monitoring system 200 includes an imaging device 2000 and a terminal device 1600 connected to the imaging device 2000 via a network 1500. The imaging unit 2001 includes a lens 2011. The imaging unit 2002 includes a lens 2012 and a field of view detection unit 2022. The imaging unit 2003 includes a lens 2013 and a field of view detection unit 2023. The imaging unit 2004 includes a lens 2014 and a field of view detection unit 2024. The imaging unit 2005 includes a lens 2015 and a field of view detection unit 2025. Each of the field of view detection units 2022 to 2025 detects the imaging angle of each of the imaging units 2022 to 2025. Each of the field of view detection units 2022 to 2025 is configured with a device capable of detecting an angle, such as a rotary encoder. The imaging range of each of the imaging units 2002-2005 can be changed by a zoom function installed in the lenses 2012-2015. The terminal device 1600 acquires the imaging angles of the imaging units 2002-2005 from the angle-of-view detection units 2022-2025.

[0039] In this embodiment, a view angle detection unit 2022 detects angles with respect to the X-axis, Y-axis, and Z-axis for the imaging unit 2002. Similarly, a view angle detection unit 2023 detects angles with respect to the X-axis, Y-axis, and Z-axis for the imaging unit 2003, a view angle detection unit 2024 detects angles with respect to the X-axis, Y-axis, and Z-axis for the imaging unit 2004, and a view angle detection unit 2025 detects angles with respect to the X-axis, Y-axis, and Z-axis for the imaging unit 2005. Information on the angles detected by the view angle detection units 2022 to 2025 (angle information) is sent to the signal processing unit 2006 in an appropriate format. The signal processing unit 2006 and the distribution unit 2007 have the same functions as the signal processing unit 1006 and the distribution unit 1007 in the first embodiment.

[0040] Next, the processing flow of the second embodiment will be described with reference to Fig. 10. As in the first embodiment, this flow illustrates the processing procedure executed by controlling each processing block by the CPU 1601 of the terminal device 1600. This flow is realized by the CPU 1601 expanding and executing a program stored in the auxiliary storage device 1603.

[0041] When the flow starts, in S200, the CPU 1601 calculates the centers of the imaging areas of the imaging units 2002 to 2005 in the image captured by the imaging unit 2001 from the angles of the imaging units 2002 to 2005 acquired by the angle-of-view detection units 2022 to 2025. In this embodiment, as a result of this calculation, it is assumed that the center of the imaging area 4502 of the imaging unit 2002 is acquired as point E (FIG. 11), the center of the imaging area 4503 of the imaging unit 2003 is acquired as point F, the center of the imaging area 4504 of the imaging unit 2004 is acquired as point G, and the center of the imaging area 4505 of the imaging unit 2005 is acquired as point H.

[0042] In S201, the display positions of the images of the imaging units 2002 to 2005 are determined with respect to the display areas 4100 and 4200 (FIG. 11) on the screen 4000 (FIG. 11) of the display unit (GUI) 1607 of the terminal device 1600. The display positions are determined based on the imaging areas of the imaging units 2002 to 2005. FIG. 11 shows the screen 4000 of the display unit 1607. A circular area 4001 is set in the center of the GUI screen 4000. The image of the imaging unit 2001 is displayed in the area 4001. A rectangular area 4100 is set above the area 4001, and a rectangular area 4200 is set below the area 4001. The areas 4100 and 4200 are indicated by dashed lines.

[0043] In this embodiment, the area 4001 is divided into two parts, vertically (divided into semicircular areas 4001a and 4001b). When the center of the captured image of the imaging units 2002 to 2005 is located in the area 4001a, the captured image is displayed in the display area 4100, and when the center is located in the area 4001b, the captured image is displayed in the display area 4200. When multiple images are displayed in the display area 4100 and the display area 4200, each image is enlarged or reduced, and the enlarged or reduced images are aligned at equal intervals and of the same size and displayed. In addition, the display position is determined while maintaining the horizontal positional relationship (which image is displayed to the left or right of which image). For example, as shown in FIG. 11, it is assumed that points E, F, and G exist in the area 4001a, and point H exists in the area 4001b. 11 (images 4002, 4003, and 4004 are arranged from left to right) in the area 4100. The CPU 1601 also determines that the image 4005 captured by the imaging unit 2005 is to be displayed in the area 4200. In this embodiment, display areas 4100 and 4200 capable of displaying two or more images among images 4002 to 4005 are set on screen 4000 of display unit 1607. When displaying two or more images in display areas 4100 and 4200, CPU 1601 displays the two or more images in the same size and evenly spaced. When displaying one image in display areas 4100 and 4200, CPU 1601 displays the one image by arranging it in the center.

[0044] In S202, the CPU 1601 displays the images captured by the image capturing units 2002 to 2005 on the GUI screen 4000 in accordance with the display positions determined in S201. Specifically, images 4002, 4003, and 4004 are displayed at equal intervals and of the same size from the left of the area 4100. Also, an image 4005 is displayed in the center of the area 4200.

[0045] Next, the process of S203 will be described. The captured image of the imaging unit 2001 in the GUI screen 4000 can be rotated and displayed by any angle around the point O in FIG. 11 as in the first embodiment. In this embodiment, it is assumed that the user specifies the subject of the image 4004 and the subject of the image 4005 from the image of the imaging unit 2001 (FIG. 11), and the images 4004 and 4005 are displayed in the display area 4200 in response to the specification. In order to display the images 4004 and 4005 in the display area 4200, the image 4001 is rotated on the GUI screen 4000. The CPU 1601 determines the angle φ by which the image 4001 should be rotated on the GUI screen 4000. In the following description, it is assumed that an instruction is input via the input unit 1606 that the image of the imaging unit 2001 is rotated clockwise by the angle φ by specifying the subject of the images 4004 and 4005. In S203, the CPU 1601 determines whether an instruction to rotate the image captured by the imaging unit 2001 in response to a subject designation by the user has been input. If the determination result in S203 is Yes, the process proceeds to S204, where the CPU 1601 analyzes the instruction (input signal), rotates the image of the imaging unit 2001 by the angle φ, and displays it in the area 4001 of the GUI screen 4000. After rotating the image of the imaging unit 2001, the process proceeds to S205. If a rotation instruction from the user is not detected in S203, the process returns to S202. In the following description, it is assumed that a user has input an instruction to rotate the image captured by the imaging unit 2001 clockwise by an angle φ.

[0046] In S205, the CPU 1601 changes (determines) the display positions of the images of the imaging units 2002 to 2005 on the screen 4000 according to the rotation angle φ input by the user. When the image of the imaging unit 2001 is rotated clockwise by the angle φ, as shown in FIG. 12(a), the point E also rotates clockwise by the angle φ and moves to a point E1. The point F also rotates clockwise by the angle φ and moves to a point F1, the point G also rotates clockwise by the angle φ and moves to a point G1, and the point H also rotates clockwise by the angle φ and moves to a point H1. After the rotation, the points E1 and F1 are located in the area 4001a, so the CPU 1601 determines that the image 4002 of the imaging unit 2002 and the image 4003 of the imaging unit 2003 are displayed at equal intervals and of the same size in the display area 4100. Furthermore, since points G1 and H1 are located in area 4001b, the CPU 1601 determines that an image 4004 from the imaging unit 2004 and an image 4005 from the imaging unit 2005 are to be displayed in the display area 4200 at equal intervals and with the same size.

[0047] 12B, the CPU 1601 displays the images at the display positions determined in S205 on the GUI screen 4000. Specifically, the CPU 1601 displays the images 4002 of the imaging unit 2002 and the images 4003 of the imaging unit 2003 at equal intervals and of the same size in the display area 4100, and displays the images 4004 of the imaging unit 2004 and the images 4004005 of the imaging unit 2005 at equal intervals and of the same size in the display area 4200. After performing the above processes, this flow ends.

[0048] According to the imaging device 2000 of this embodiment, in the monitoring device 2000 combining the multiple lens cameras 2002-2005 and the wide-angle camera 2001, the display positions of the multiple lens cameras 2002-2005 can be changed and displayed in response to a change in the display of the wide-angle camera 2001. If the images of the multiple lens cameras 2002-2005 remain unchanged when the image of the wide-angle camera 2001 is rotated, it becomes difficult to understand the correspondence (relationship) between the image of the wide-angle camera 2001 and the images of the multiple lens cameras 2002-2005. In this embodiment, the positions of the images of the multiple lens cameras 2002-2005 are changed in response to the rotation of the image of the wide-angle camera 2001, so that the correspondence between the images is easy to understand.

[0049] The arrangement of the GUI screen 3000 and the GUI screen 4000 and the image display positions shown in the first and second embodiments are merely examples, and any display device that changes the display in response to a change in the imaging position of another imaging unit relative to an image captured by a wide-angle imaging unit may be used. Also, the imaging units 2002 to 2005 may not have a zoom function, and may be able to change the imaging direction only in the pan direction (or only in the tilt direction). Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0050] The above-described embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0051] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

[0052] The disclosure of this embodiment includes the following configuration, method, and program. Configuration 1 a first acquisition means for acquiring a first image from a first imaging unit that captures an image at a predetermined angle of view; a second acquisition means for acquiring a second image from a second imaging unit that captures an image of an area overlapping with the first imaging unit at an angle of view narrower than the predetermined angle of view; A control means for displaying the first image and the second image on a screen of a display unit; An information processing device having An information processing device characterized in that, when the first image is rotated by a predetermined angle on the screen of the display unit, the control means changes the display position of the second image on the screen of the display unit based on the predetermined angle. Configuration 2 2. The information processing device according to configuration 1, wherein the first imaging unit has a super-wide-angle lens and can capture images in all directions. Configuration 3 3. The information processing device according to configuration 1 or 2, wherein the second imaging section captures an image of a narrower range than the first imaging section.

[0053] Configuration 4 The information processing device of any one of configurations 1 to 3, wherein the second imaging units are arranged around the first imaging unit, and each of the second imaging units images an area that overlaps with the first imaging unit, and the areas imaged by the second imaging units do not overlap with each other. Configuration 5 the second acquisition means acquires the second images from the second imaging units, 5. The information processing device according to configuration 4, wherein the control means displays the first image in the center of a screen of the display unit, and displays the plurality of second images around the first image. Configuration 6 The information processing device described in configuration 4 or 5, characterized in that the control means determines the display positions of the multiple second images on the screen of the display unit based on the positions where the areas imaged by each of the multiple second imaging units overlap with the area imaged by the first imaging unit. Configuration 7 The information processing device according to any one of configurations 4 to 6, characterized in that the control means determines the display positions of the multiple second images on the screen of the display unit based on the center positions of the areas captured by each of the multiple second imaging units. Configuration 8 8. The information processing device according to any one of configurations 4 to 7, wherein each of the plurality of second imaging sections is capable of changing the imaging direction in at least one of a pan direction and a tilt direction.

[0054] Configuration 9 the information processing device further includes a third acquisition means for acquiring imaging angles of the second imaging units, The information processing device according to any one of configurations 4 to 8, wherein the control means determines a center position of an area imaged by each of the plurality of second imaging units based on the imaging angle acquired by the third acquisition means. Configuration 10 a screen of the display unit is provided with display areas the number of which is equal to the number of the second images, in order to display the second images individually; An information processing device as described in any one of configurations 5 to 9, characterized in that when the first image is rotated by a predetermined angle on the screen of the display unit, the control means determines a display area in which each of the multiple second images should be displayed based on the predetermined angle.

[0055] Configuration 11 An information processing device as described in any one of configurations 5 to 9, characterized in that a display area is set on the screen of the display unit in which two or more of the multiple second images can be displayed, and when two or more of the second images are displayed in the display area, the control means displays the two or more second images of the same size and evenly arranged, and when one second image is displayed in the display area, the control means displays the one second image by arranging it in the center. Configuration 12 The information processing device according to any one of configurations 1 to 11, characterized in that when a subject is specified in the first image, the control means determines an angle by which the first image should be rotated in accordance with the specification, and the determined angle becomes the specified angle. Method 1 acquiring a first image from a first imaging unit that captures an image at a predetermined angle of view; acquiring a second image from a second imaging unit that captures an image of an area overlapping with the first imaging unit at an angle of view narrower than the predetermined angle of view; displaying the first image and the second image on a screen of a display unit; changing a display position of the second image on the screen of the display unit based on the predetermined angle when the first image is rotated on the screen of the display unit; The method according to claim 1, further comprising: Program 1 13. A program for causing a computer to function as each of the means included in the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0056] 1000 imaging device, 1001 to 1005 imaging units, 1600 terminal device, 1601 CPU, 1607 display unit, 1701 system control unit, 1702 display control unit, 2000 imaging device

Claims

1. a first acquisition means for acquiring a first image from a first imaging unit that captures an image at a predetermined angle of view; a second acquisition means for acquiring a second image from a second imaging unit that captures an image of an area overlapping with the first imaging unit at an angle of view narrower than the predetermined angle of view; a control means for displaying the first image and the second image on a screen of a display unit; An information processing device having An information processing device characterized in that when the first image is rotated by a predetermined angle on the screen of the display unit, the control means changes the display position of the second image on the screen of the display unit based on the predetermined angle.

2. The information processing apparatus according to claim 1 , wherein the first imaging unit has a super-wide-angle lens and is capable of capturing images in all directions.

3. The information processing apparatus according to claim 1 , wherein the second imaging section captures an image of a narrower range than the first imaging section.

4. The information processing device according to claim 1, characterized in that the second imaging unit is arranged in a plurality of units around the first imaging unit, and each of the plurality of second imaging units images an area that overlaps with the first imaging unit, and the areas imaged by the plurality of second imaging units do not overlap with each other.

5. the second acquisition means acquires the second images from the second imaging units, 5. The information processing apparatus according to claim 4, wherein the control means displays the first image in the center of a screen of the display unit, and displays the plurality of second images around the first image.

6. The information processing device according to claim 5, characterized in that the control means determines the display positions of the multiple second images on the screen of the display unit based on the position where the area imaged by each of the multiple second imaging units overlaps with the area imaged by the first imaging unit.

7. The information processing device according to claim 6, characterized in that the control means determines display positions of the plurality of second images on the screen of the display unit based on center positions of areas captured by each of the plurality of second imaging units.

8. The information processing apparatus according to claim 7 , wherein the imaging direction of each of the second imaging units can be changed in at least one of a pan direction and a tilt direction.

9. the information processing device further includes a third acquisition means for acquiring imaging angles of the second imaging units, 9. The information processing apparatus according to claim 8, wherein the control means determines a center position of an area captured by each of the plurality of second imaging units based on the imaging angle acquired by the third acquisition means.

10. a screen of the display unit is provided with display areas the number of which is equal to the number of the second images, in order to display the second images individually; The information processing device according to claim 5, characterized in that when the first image is rotated by a predetermined angle on the screen of the display unit, the control means determines a display area in which each of the plurality of second images is to be displayed based on the predetermined angle.

11. The information processing device described in claim 5, characterized in that a display area is set on the screen of the display unit in which two or more of the multiple second images can be displayed, and when two or more of the second images are displayed in the display area, the control means displays the two or more second images of the same size and evenly arranged, and when one second image is displayed in the display area, the control means displays the one second image by arranging it in the center.

12. 2. The information processing device according to claim 1, wherein, when a subject is specified in the first image, the control means determines an angle by which the first image should be rotated in accordance with the specification, and the determined angle becomes the predetermined angle.

13. acquiring a first image from a first imaging unit that captures an image at a predetermined angle of view; acquiring a second image from a second imaging unit that captures an image of an area overlapping with the first imaging unit at an angle of view narrower than the predetermined angle of view; displaying the first image and the second image on a screen of a display unit; when the first image is rotated by a predetermined angle on the screen of the display unit, changing a display position of the second image on the screen of the display unit based on the predetermined angle; The method according to claim 1, further comprising:

14. A program for causing a computer to function as each of the means possessed by the information processing device according to any one of claims 1 to 12.

Images