Information processing apparatus, information processing method, storage medium, and computer program product

Abstract

The present invention provides an information processing apparatus, an information processing method, a storage medium, and a computer program product, the information processing apparatus including: a first obtaining unit configured to obtain a first image from a first imaging unit configured to capture an image at a predetermined viewing angle; a second obtaining unit configured to obtain a second image from a second imaging unit configured to capture an image of a region overlapping a region in which the first imaging unit captures the image at a viewing angle narrower than the predetermined viewing angle; and a control unit configured to perform control to display the first image and the second image on the display unit, in which the control unit changes a display position of the second image on a screen of the display unit based on a predetermined angle in a case where the first image is rotated by the predetermined angle on the display unit.

Description

Technical Field

[0001] The present disclosure relates to an information processing device. Background Art

[0002] There are known imaging devices having a configuration in which a plurality of cameras are incorporated in a single housing. For example, Japanese Patent Laid-Open No. 2019-140566 discusses an imaging device including a pan-tilt-zoom (PTZ) camera and a predetermined number of wide-area cameras disposed around the PTZ camera. Such an imaging device is configured to capture images of the surrounding environment using a predetermined number of wide-area cameras, thereby focusing on a subject of interest and capturing an image of the subject of interest using a centrally disposed PTZ camera.

[0003] In the case of Japanese Patent Laid-Open No. 2019-140566, if the pan angle and pitch angle of the PTZ camera change, the display of the image captured by the wide-area camera is changed based on the shooting direction of the PTZ camera, so that the user can easily identify the relationship between the shooting position of each wide-area camera and the shooting position of the PTZ camera.

[0004] However, in the case of Japanese Patent Laid-Open No. 2019-140566, if the display (image) of each wide area camera is rotated, it may be difficult for the user to recognize the positional relationship between the wide area camera and the PTZ camera. Summary of the invention

[0005] Accordingly, the present disclosure aims to provide an information processing device, which is configured to allow a user to easily identify the positional relationship between the image captured by the wide-angle camera unit and the image captured by a camera unit other than the wide-angle camera unit, even when the display of the image captured by the wide-angle camera unit is rotated, when the image captured by the wide-angle camera unit and the image captured by a camera unit having a narrower angle of view than the wide-angle camera unit are displayed on a single display unit.

[0006] According to one aspect of the present invention, an information processing device includes: a first acquisition unit, which is configured to obtain a first image from a first camera unit, wherein the first camera unit is configured to capture an image at a predetermined angle of view; a second acquisition unit, which is configured to obtain a second image from a second camera unit, wherein the second camera unit is configured to capture an image of an area overlapping with an area where the image is captured by the first camera unit at a angle of view narrower than the predetermined angle of view; and a control unit, which is configured to control to display the first image and the second image on a display unit, wherein when the first image is rotated by a predetermined angle on the display unit, the control unit changes the display position of the second image on the screen of the display unit based on the predetermined angle.

[0007] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram illustrating a system including a terminal device according to a first exemplary embodiment of the present disclosure.

[0009] Figure 2A and Figure 2B Each shows Figure 1 A block diagram of the configuration of the terminal device shown.

[0010] Figure 3A and Figure 3B yes Figure 1 The external schematic diagram of the camera device shown.

[0011] Figure 4 is a flowchart illustrating a processing procedure according to the first exemplary embodiment.

[0012] Figure 5 Show Figure 1 A screen of a graphical user interface (GUI) of a terminal device is shown.

[0013] Figure 6 Show Figure 5 An example of a display of a GUI screen is shown.

[0014] Fig. 7A and Figure 7B Each shows that when the image is from Figure 6 The GUI screen is shown in the state of rotation.

[0015] Figure 8 is a schematic external view of an imaging apparatus according to a second exemplary embodiment of the present disclosure.

[0016] Fig. 9 is a block diagram illustrating a system including a terminal device according to a second exemplary embodiment of the present disclosure.

[0017] Fig.10 is a flowchart illustrating a processing procedure according to the second exemplary embodiment.

[0018] Fig.11 A display example of a screen of a GUI according to the second exemplary embodiment is shown.

[0019] Fig. 12A and Fig. 12B Each shows that when the image is from Fig.11 The GUI screen is shown in the state of rotation. DETAILED DESCRIPTION

[0020] Typical embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The following typical embodiments are not intended to limit the present disclosure, and not all feature combinations described in the typical embodiments are essential to the solutions of the present disclosure. The configuration of the typical embodiments can be appropriately modified or changed according to the specifications and various conditions (usage conditions, usage environment, etc.) of the equipment to which the present disclosure is applied. The technical scope of the present disclosure is defined by the claims and is not limited by the following typical embodiments. Parts of the typical embodiments can be appropriately combined. Throughout the accompanying drawings, components with the same function are represented by the same figure marks, and their repeated descriptions are omitted.

[0021] Figure 1 A monitoring system 100 according to a first exemplary embodiment of the present disclosure is shown. The monitoring system 100 includes an imaging apparatus 1000 and a terminal apparatus 1600 connected to the imaging apparatus 1000 via a network 1500. The terminal apparatus 1600 is operated by, for example, a user of the monitoring system 100. The terminal apparatus 1600 is, for example, a personal computer.

[0022] The imaging apparatus 1000 includes an imaging unit 1001, an imaging unit 1002, an imaging unit 1003, an imaging unit 1004, an imaging unit 1005, a signal processing unit 1006, and a delivery unit 1007. The imaging units 1001 to 1005 are each connected to the signal processing unit 1006. The signal processing unit 1006 is connected to the delivery unit 1007.

[0023] The camera unit 1001 includes a lens 1011 and Figure 1 An image sensor such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor, not shown, is used. The image sensor converts light incident from the lens 1011 into an electrical signal. The imaging unit 1001 is a wide-angle camera. In the first exemplary embodiment, it is assumed that the lens 1011 of the imaging unit 1001 is an ultra-wide-angle lens such as a fisheye lens, and the horizontal viewing angle of the lens 1011 is approximately 180 degrees. In the first exemplary embodiment, the imaging unit 1001 is a 360-degree camera (configured to capture a 360-degree image). The imaging units 1002 to 1005 are gaze cameras each configured to capture an image of a subject (a subject of interest) in a 360-degree image captured by the imaging unit 1001. In other words, the imaging units 1002 to 1005 each capture an image of an area overlapping with the area where the imaging unit 1001 captures an image. In the first exemplary embodiment, it is assumed that the images of the areas captured by the four imaging units 1002 to 1005 do not overlap with each other.

[0024] The imaging units 1002 to 1005 each have a structure similar to that of the imaging unit 1001. Specifically, the imaging unit 1002 includes a lens 1012 and an image sensor (not shown), the imaging unit 1003 includes a lens 1013 and an image sensor (not shown), the imaging unit 1004 includes a lens 1014 and an image sensor (not shown), and the imaging unit 1005 includes a lens 1015 and an image sensor (not shown). It is assumed that all the lenses 1012 to 1015 have the same angle of view, and the horizontal angle of view of each of the lenses 1012 to 1015 is narrower than the horizontal angle of view of the lens 1011. In other words, the imaging units 1002 to 1005 are cameras having an angle of view narrower than that of the imaging unit 1001 which is a wide-angle camera. In the first exemplary embodiment, the angles of view of each of the imaging units 1002 to 1005 are the same.

[0025] The signal processing unit 1006 is configured to perform predetermined image processing and compression encoding processing on the electrical signals output from the imaging units 1001 to 1005 , thereby generating image data.

[0026] The signal processing unit 1006 is also configured to transmit setting values ​​of imaging conditions to the imaging units 1001 to 1005 and change the operation states of the respective imaging units 1001 to 1005. The imaging conditions include gain conditions, gamma conditions, dynamic range conditions, exposure conditions, and focus conditions.

[0027] The delivery unit 1007 is connected to the network 1500. The delivery unit 1007 converts the image data output from the signal processing unit 1006 into a signal in an appropriate format and transmits the signal to the network 1500.

[0028] The network 1500 transmits information (including images) received from the imaging apparatus 1000 to the terminal apparatus 1600. The terminal apparatus 1600 receives captured images captured by the imaging units 1001 to 1005 of the imaging apparatus 1000, thereby obtaining images captured by the imaging units 1001 to 1005.

[0029] The network 1500 also transmits information received from the terminal device 1600 to the imaging device 1000. When information received from the terminal device 1600 (for example, information input by a user) is input to the imaging device 1000 via the network 1500, the delivery unit 1007 receives the information, converts the information into an appropriate format, and transmits the converted information (signal) to the signal processing unit 1006.

[0030] The network 1500 may be a network capable of exchanging data between any plurality of devices. In the first exemplary embodiment, it is assumed that the network 1500 is a network compliant with Ethernet. A wired local area network (LAN).

[0031] It can be realized through hardware Figure 1 A portion of the functional modules (1006, 1007) included in the camera device 1000 shown in FIG. 1 is a portion of the functional modules (1006, 1007) included in the camera device 1000 shown in FIG. In the case where a portion of the functional modules is implemented by hardware, a predetermined compiler may be used, for example, to automatically generate a dedicated circuit on a field programmable gate array (FPGA) according to a program for implementing each function of each functional module. The gate array circuit may be formed in the same manner as the FPGA, and the gate array circuit may be implemented as hardware. Alternatively, each function may be implemented by an application specific integrated circuit (ASIC). Figure 1 The configuration of each functional module shown is only an example. A plurality of functional modules may constitute a single functional module, or one of the functional modules may be divided into modules each configured to implement a plurality of functions.

[0032] Figure 2A 1600 is a block diagram showing the hardware configuration of the terminal device 1600. The terminal device 1600 includes a central processing unit (CPU) 1601, a main storage device 1602, an auxiliary storage device 1603, an external interface (I / F) 1604, a graphics processing unit (GPU) 1605, an input unit 1606, and a display unit 1607. The units (1601 to 1607) of the terminal device 1600 are interconnected via a bus 1608 so that the units can communicate with each other. The terminal device 1600 for processing various types of information may also be referred to as an information processing device. The CPU 1601 is a central processing unit for controlling the overall operation of the terminal device 1600, and performs various types of arithmetic processing, image processing, and data input-output processing.

[0033] The main storage device 1602 is a storage device used as a work area or a temporary storage area of ​​data for the CPU 1601. The main storage device 1602 is implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0034] The auxiliary storage device 1603 is a storage device for storing various programs, various types of setting information, various types of image data, camera parameters, three-dimensional shape data, and information related to a two-dimensional map, etc. The auxiliary storage device 1603 is implemented using a nonvolatile memory such as a read-only memory (ROM) or a flash memory, or a storage medium such as a hard disk drive (HDD), a solid-state drive (SSD), or a magnetic tape medium. In order to increase capacity and processing speed, a plurality of nonvolatile memories, HDDs, and SSDs may be combined into a single logical unit such as a redundant array of independent disks (RAID).

[0035] The external I / F 1604 is a general-purpose input-output interface. The external I / F 1604 includes a communication interface for communicating with the camera 1000. The external I / F 1604 can be connected to an external device for receiving various operations from a user. The external I / F 1604 can be connected to an external memory, and a file can be input from the external memory to the terminal device 1600 or a file can be output from the terminal device 1600 to the external memory.

[0036] The external I / F 1604 is configured to include, for example, a wireless bandwidth (InfiniBand), or a physical cable connection terminal such as a universal serial bus (USB), or an interface using a wireless LAN or Bluetooth The signal input from the network 1500 is transmitted via the external I / F 1604. The data is transmitted to the CPU 1601 through an interface such as the like.

[0037] The GPU 1605 is a computing unit that performs image processing operations at high speed. The GPU 1605 converts the image (video image) received from the camera device 1000 into an appropriate format and outputs the image. The GPU 1605 may include a main storage device independent of the main storage device 1602. In addition to the above operations, the GPU 1605 may perform part or all of the various computing operations performed by the CPU 1601.

[0038] The input unit 1606 includes a keyboard, a mouse, and a joystick. The user can operate the input unit 1606 to input desired instructions and data, etc. to the terminal device 1600. The user's instructions, etc. input from the input unit 1606 are converted into appropriate signals, and the signals are transmitted to the camera device 1000. The input unit 1606 operated by the user may also be referred to as an operation unit. The input unit 1606 does not necessarily have to be set in the terminal device 1600. In this case, the input unit 1606 such as a keyboard, a mouse, or a joystick is connected to the terminal device 1600 via the external I / F 1604.

[0039] The display unit 1607 includes a liquid crystal monitor. The display unit 1607 is configured to display an image transmitted from the camera device 1000 to the terminal device 1600 so that the user can view the image. The display unit 1607 is also configured to display information input by the user from the input unit 1606. The display unit 1607 for displaying (outputting) an image may also be referred to as an output unit. The display unit 1607 may also be used as a touch panel. In this case, it can be said that the display unit 1607 is also used as a part of the input unit 1606. In the case where the display unit 1607 is also used as a touch panel, the display unit 1607 is a graphical user interface (GUI).

[0040] The bus 1608 is a bus based on Peripheral Component Interconnect Express (PCI Express) or the like, and enables the units (1601 to 1607) represented by the hardware blocks to establish bidirectional communication. Any type of bus can be used as the bus 1608 as long as the units represented by the hardware blocks can establish bidirectional communication via the bus.

[0041] Figure 2B 1700 is a block diagram showing a functional configuration of a terminal device 1600. The terminal device 1600 includes a system control unit 1701, a display control unit 1702, and an operation control unit 1703.

[0042] The system control unit 1701 performs processing such as establishing communication with an external device such as the imaging device 1000, controlling each function of the terminal device 1600, and setting various parameters. More specifically, the system control unit 1701 performs processing such as receiving an image transmitted from the imaging device 1000, issuing an instruction for displaying the received image, and receiving an input based on an operation via the input unit 1606. The display control unit 1702 displays the monitor ( Figure 5 The display control unit 1702 also causes the monitor of the display unit 1607 to display images transmitted from the imaging units 1001 to 1005 of the imaging apparatus 1000. This enables the user to check the images captured by the imaging apparatus 1000 by viewing the monitor of the display unit 1607. The operation control unit 1703 receives an instruction from the user based on the user operation via the input unit 1606, and transmits information related to the received instruction to the system control unit 1701. For example, the operation control unit 1703 receives an instruction from the user to drive the wide-angle camera (imaging unit) 1001 based on the user operation via the input unit 1606, and transmits information related to the received instruction to the system control unit 1701. Upon receiving the instruction to drive the wide-angle camera 1001, the system control unit 1701 transmits the received instruction to the imaging apparatus 1000, thereby instructing the imaging apparatus 1000 to drive the wide-angle camera 1001.

[0043] Figure 3A and Figure 3B 1000 are schematic diagrams each showing the layout of image pickup units 1001 to 1005 in the image pickup apparatus 1000 .

[0044] Figure 3A is a side view of the imaging apparatus 1000, and Figure 3B 1 is a top view of the imaging device 1000. The imaging device 1000 includes a base portion 1000a, a cover portion 1000b, and a support portion 1000c. The support portion 1000c is provided on the base portion 1000a. The cover portion 1000b is provided on the base portion 1000a to cover the support portion 1000c.

[0045] The camera unit 1001 is disposed on the top of the support portion 1000c. The camera units 1002 to 1005 are disposed on the outer peripheral surface of the support portion 1000c. The camera units 1001 to 1005 are accommodated in a space defined by the base portion 1000a and the cover portion 1000b. Figure 3B As shown, as seen from the top of the imaging device 1000, the imaging units 1002 to 1005 are arranged on the circumference of a circle centered on the imaging unit 1001, and the imaging units 1002 to 1005 are arranged at intervals of 90 degrees. In other words, a plurality of imaging units 1002 to 1005, each configured to capture an image of an area overlapping with an area where the imaging unit 1001 captures an image, at a narrower angle of view than the angle of view of the imaging unit 1001, are arranged around the imaging unit 1001. The optical axis direction of each of the imaging units 1001 to 1005 is fixed.

[0046] The configuration of the imaging device 1000 is not limited to FIG. 2A to FIG. 3B Configuration shown. FIG. 2A to FIG. 3B The configuration shown is merely an example. The configuration of the imaging apparatus 1000 can be appropriately changed and configured according to the imaging target and function.

[0047] The following describes how to display an image captured by the camera unit (wide-angle camera) 1001 and images captured by four camera units (multiple cameras) 1002 to 1005 arranged around the image captured by the camera unit 1001 on a single screen 3000 of the display unit 1607 in the monitoring system 100 according to the first typical embodiment.

[0048] Figure 4 1701 to 1703. The flowchart shows the processing procedure to be performed by the CPU 1601 of the terminal device 1600 by controlling the control units 1701 to 1703. The processing flow is realized by the CPU 1601 loading the program stored in the auxiliary storage device 1603 and executing the program.

[0049] It is assumed that the terminal device 1600 receives images captured by the imaging units 1001 to 1005 from the imaging device 1000, thereby obtaining the captured images. Furthermore, it is assumed that the display unit 1607 of the terminal device 1600 is used as a GUI.

[0050] When the processing flow starts, in step S100, the CPU 1601 determines the display position of the images captured by the imaging units 1001 to 1005 on the screen (display area) 3000 of the display unit (GUI) 1607. The display position of the image is determined based on the imaging area of ​​the imaging units 1001 to 1005.

[0051] Figure 5 The screen 3000 of the display unit (GUI) 1607 of the terminal device 1600 is shown. Five areas 3001, 3002, 3003, 3004 and 3005 are set on the screen 3000. Area 3001 is a circular area, and areas 3002 to 3005 are rectangular areas. Area 3001 is located in the center of the screen 3000, and areas 3002 to 3005 are located around area 3001. Specifically, area 3002 is located at the upper left position of the screen 3000, area 3003 is located at the upper right position of the screen 3000, area 3004 is located at the lower right position of the screen 3000, and area 3005 is located at the lower left position of the screen 3000. The image captured by the camera unit 1001 is displayed in area 3001. An image captured by one of the camera units 1002 to 1005 is displayed in one of the areas 3002 to 3005.

[0052] In order to individually display images captured by the imaging units 1002 to 1005 on the screen 3000 of the display unit 1607 , the same number of areas 3002 to 3005 as the number of imaging units 1002 to 1005 (four in the first exemplary embodiment) are provided.

[0053] Next, the determination of the display position will be described. The area 3001 is divided into the area 3001a, the area 3001b, the area 3001c, and the area 3001d in advance. The areas 3001a to 3001d are each an area of ​​a 1 / 4 circle obtained by dividing the area 3001 into four areas. The CPU 1601 calculates the center position of each imaging area of ​​the imaging units 1002 to 1005 in the image captured by the imaging unit 1001, and determines in which area of ​​the areas 3001a, 3001b, 3001c, and 3001d the center position is located. If the center position of the imaging area exists in the area 3001a, the image captured by the imaging unit is displayed in the area 3002. If the center position of the imaging area exists in the area 3001b, the image captured by the imaging unit is displayed in the area 3003. If the center position of the imaging area exists in area 3001c, the image captured by the imaging unit is displayed in area 3004. If the center position of the imaging area exists in area 3001d, the image captured by the imaging unit is displayed in area 3005. For example, in a case where the imaging area of ​​one of the imaging units 1002 to 1005 is set to area 3500 and the center of area 3500 is set to point X, the image captured by the imaging unit is displayed in area 3002. In the first typical embodiment, the center of each imaging area is represented by the coordinates projected onto the imaging area by the optical axis of the imaging unit. However, the center of each imaging area is not limited to this point. Any point appropriately set in the imaging area (for example, the center of mass of a person (subject) in the image) can be the center of the imaging area.

[0054] Figure 6 The display position determined in step S100 is shown. It is assumed that the imaging units 1002 to 1005 each capture an image of a person as a subject. Figure 6 In the example shown, the center of the imaging area 3502 of the imaging unit 1002 is set to point A, the center of the imaging area 3503 of the imaging unit 1003 is set to point B, the center of the imaging area 3504 of the imaging unit 1004 is set to point C, and the center of the imaging area 3505 of the imaging unit 1005 is set to point D. Point A is located in the area 3001a, so the CPU 1601 determines to display the image captured by the imaging unit 1002 in the area 3002. Point B is located in the area 3001b, so the CPU 1601 determines to display the image captured by the imaging unit 1003 in the area 3003. Point C is located in the area 3001c, so the CPU 1601 determines to display the image captured by the imaging unit 1004 in the area 3004. Point D is located in the area 3001 d , and therefore the CPU 1601 determines to display the image captured by the imaging unit 1005 in the area 3005 .

[0055] In step S101, the CPU 1601 displays the images captured by the imaging units 1002 to 1005 on the screen 3000 of the display unit (GUI) 1607 based on the display positions determined in step S100. Specifically, the image captured by the imaging unit 1002 is displayed in the area 3002, the image captured by the imaging unit 1003 is displayed in the area 3003, the image captured by the imaging unit 1004 is displayed in the area 3004, and the image captured by the imaging unit 1005 is displayed in the area 3005. In addition, the image captured by the imaging unit 1001 is displayed in the area 3001, and the imaging areas 3502 to 3505 of the imaging units 1002 to 1005 are respectively displayed on the image. The camera unit 1001 captures an image of the entire area from a bird's eye view (360-degree image), and uses the images captured by the camera units 1002 to 1005 to supplement the portion (peripheral portion) in which details may be blurred in the image captured by the camera unit 1001. Figure 6 In FIG. 1 , illustration of details (parts other than the subject) of the image captured by the imaging unit 1001 is omitted.

[0056] Next, the processing of step S102 will be described. The image captured by the camera unit 1001 and displayed on the screen 3000 can be displayed in a manner rotated by any angle around the point O which is the center of the displayed image. In the first exemplary embodiment, it is assumed that the user can input an instruction for rotating the image captured by the camera unit 1001 clockwise (or counterclockwise) by an angle θ via the input unit 1606. In step S102, the CPU 1601 determines whether an image rotation instruction is input from the user. If the CPU 1601 determines that an image rotation instruction is input from the user ("Yes" in step S102), the processing proceeds to step S103. In step S103, the CPU 1601 analyzes the instruction (input signal), rotates the image captured by the camera unit 1001 by an angle θ, and displays the rotated image in the area 3001 of the screen 3000. After the image captured by the camera unit 1001 is rotated, the processing proceeds to step S104.

[0057] If an image rotation instruction from the user is not detected (NO in step S102 ), the process returns to step S101 .

[0058] In the following description, it is assumed that the user inputs an instruction for rotating the image captured by the imaging unit 1001 clockwise by an angle θ.

[0059] In step S104, the CPU 1601 changes (determines) the display position of the images captured by the imaging units 1002 to 1005 on the screen 3000 based on the rotation angle θ input by the user. When the image captured by the imaging unit 1001 is rotated clockwise by the angle θ, as shown in FIG. Fig. 7A As shown, point A also rotates clockwise by an angle θ and moves to point A1. Point B also rotates clockwise by an angle θ and moves to point B1. Point C also rotates clockwise by an angle θ and moves to point C1. Point D also rotates clockwise by an angle θ and moves to point D1. After the rotation, point A1 is located in area 3001b, whereby the CPU 1601 determines that the image captured by the camera unit 1002 is displayed in area 3003. Point B1 is located in area 3001c, whereby the CPU 1601 determines that the image captured by the camera unit 1003 is displayed in area 3004. Point C1 is located in area 3001d, whereby the CPU 1601 determines that the image captured by the camera unit 1004 is displayed in area 3005. Point D1 is located in area 3001a, whereby the CPU 1601 determines that the image captured by the camera unit 1005 is displayed in area 3002.

[0060] In step S105, Figure 7B As shown, the CPU 1601 displays the image at the display position determined in step S104 on the screen 3000. Specifically, the CPU 1601 displays the image captured by the camera unit 1002 in the area 3003, displays the image captured by the camera unit 1003 in the area 3004, displays the image captured by the camera unit 1004 in the area 3005, and displays the image captured by the camera unit 1005 in the area 3002. After the processing is performed as described above, the processing flow ends.

[0061] Regarding the imaging apparatus 1000 according to the first exemplary embodiment, the imaging apparatus 1000 including the combination of the wide-angle camera 1001 and the plurality of cameras 1002 to 1005 is configured to display images captured by the plurality of cameras 1002 to 1005 by changing the display positions of the images captured by the plurality of cameras 1002 to 1005 based on the change in the display of the wide-angle camera 1001. If the images captured by the plurality of cameras 1002 to 1005 do not change when the image captured by the wide-angle camera 1001 is rotated, it is difficult to understand the correspondence relationship between the image captured by the wide-angle camera 1001 and the images captured by the plurality of cameras 1002 to 1005. In the first exemplary embodiment, since the positions of the images captured by the plurality of cameras 1002 to 1005 change according to the rotation of the image captured by the wide-angle camera 1001, the correspondence relationship between the images can be easily understood.

[0062] In the above exemplary embodiment, four cameras are used as multiple cameras. However, the number of multiple cameras to be used is not limited to four. Although the terminal device 1600 includes the display unit 1607 in the above exemplary embodiment, the display unit 1607 can be set outside the terminal device 1600 as an independent device. In this case, the display unit (display device) 1607 is connected to the external I / F 1604.

[0063] Reference below Figures 8 to 12B A second typical embodiment of the present disclosure is described. An imaging device according to the second typical embodiment includes a single wide-angle camera (360-degree camera) and a plurality of gaze cameras each having a variable viewing angle. In the second typical embodiment, a user specifies a subject from an image (video image) captured by the wide-angle camera, the image captured by the wide-angle camera is rotated on the screen of the display unit 1607 based on the specification, and the display position of the image captured by the gaze camera is also changed. Components according to the second typical embodiment that are the same as those according to the first typical embodiment are denoted by the same reference numerals, and descriptions thereof may be omitted.

[0064] Figure 8 2 is a schematic diagram of an image pickup apparatus 2000 used in the second exemplary embodiment. The image pickup apparatus 2000 includes a base portion 2000a, a cover portion 2000b, and a support portion 2000c provided on the base portion 2000a. The support portion 2000c is provided on the base portion 2000a. The cover portion 2000b is provided on the base portion 2000a to cover the support portion 2000c.

[0065] The camera device 2000 includes camera units 2001 to 2005. The camera unit 2001 is a 360-degree camera. The camera unit 2001 is arranged on the top surface of the support portion 2000c. The camera units 2002 to 2005 are gaze cameras. The camera units 2002 to 2005 are arranged on the top surface of the base portion 2000a. The camera units 2001 to 2005 are accommodated in a space defined by the base portion 2000a and the cover portion 2000b.

[0066] The imaging unit 2001 has the same configuration as that of the imaging unit 1001 according to the first exemplary embodiment. As in the first exemplary embodiment, the imaging unit 2001 is located at the center, and the imaging units 2002 to 2005 are arranged at regular intervals around the imaging unit 2001. An X-axis, a Y-axis, and a Z-axis are defined for each of the imaging units 2002 to 2005. The Z-axis corresponds to the optical axis direction of each of the imaging units 2002 to 2005. The X-axis and the Y-axis are orthogonal to the Z-axis. Figure 8The X-axis 2003a, Y-axis 2003b and Z-axis 2003c of the camera unit 2003 are shown. The angles of the camera units 2002 to 2005 around the X-axis, Y-axis and Z-axis can be changed. The angles of the camera units 2002 to 2005 around the axes (X-axis, Y-axis and Z-axis) can be changed independently. In the case of the camera unit 2003, the rotation around the X-axis corresponds to the panning operation of the camera unit 2003, and the rotation around the Y-axis corresponds to the pitching operation of the camera unit 2003. The camera direction of the camera units 2002 to 2005 can be changed by changing the angles of the camera units 2002 to 2005 around the X-axis and Y-axis. The camera units 2002 to 2005 can also be called movable camera units. The camera unit 2001 can also be called a fixed camera unit. The imaging range of each of the imaging units 2002 to 2005 can be changed using a zoom function incorporated in a corresponding lens (not shown).

[0067] Fig. 9 A monitoring system 200 according to a second exemplary embodiment is shown. The monitoring system 200 includes an imaging device 2000 and a terminal device 1600 connected to the imaging device 2000 via a network 1500. An imaging unit 2001 includes a lens 2011. An imaging unit 2002 includes a lens 2012 and an angle of view detection unit 2022. An imaging unit 2003 includes a lens 2013 and an angle of view detection unit 2023. An imaging unit 2004 includes a lens 2014 and an angle of view detection unit 2024.

[0068] The camera unit 2005 includes a lens 2015 and a viewing angle detection unit 2025. The viewing angle detection units 2022 to 2025 detect the camera angles of the camera units 2002 to 2005, respectively. The viewing angle detection units 2022 to 2025 each include a device configured to detect an angle, such as a rotary encoder. The camera units 2002 to 2005 are each configured to change the camera range using a zoom function incorporated in each lens 2012 to 2015. The terminal device 1600 obtains the camera angles of the camera units 2002 to 2005 from the viewing angle detection units 2022 to 2025, respectively.

[0069] In the second exemplary embodiment, the angle of view detection unit 2022 detects the angle of the camera unit 2002 with respect to the X-axis, the Y-axis, and the Z-axis. Likewise, the angle of view detection unit 2023 detects the angle of the camera unit 2003 with respect to the X-axis, the Y-axis, and the Z-axis, the angle of view detection unit 2024 detects the angle of the camera unit 2004 with respect to the X-axis, the Y-axis, and the Z-axis, and the angle of view detection unit 2025 detects the angle of the camera unit 2005 with respect to the X-axis, the Y-axis, and the Z-axis. Information (angle information) related to the angles detected by the angle of view detection units 2022 to 2025 is transmitted to the signal processing unit 2006 in an appropriate format.

[0070] It is assumed that the signal processing unit 2006 has the same function as that of the signal processing unit 1006 according to the first exemplary embodiment, and the delivery unit 2007 has the same function as that of the delivery unit 1007 according to the first exemplary embodiment.

[0071] Next, we will refer to Fig.10 The processing flow according to the second exemplary embodiment is described. As in the first exemplary embodiment, the flowchart shows the processing procedure to be performed by the CPU 1601 of the terminal device 1600 by controlling the control unit. The processing flow is realized by the CPU 1601 loading the program stored in the auxiliary storage device 1603 and executing the program.

[0072] When the processing flow starts, in step S200, the CPU 1601 calculates the center of the imaging area of ​​the imaging units 2002 to 2005 in the image captured by the imaging unit 2001 based on the angles of the imaging units 2002 to 2005 obtained by the angle of view detection units 2022 to 2025, respectively. In the second exemplary embodiment, as a result of the calculation, the center of the imaging area of ​​the imaging units 2002 to 2005 is obtained as the point E ( Fig.11 ) is obtained as the center of the imaging area 4502 of the imaging unit 2002, the center of the imaging area 4503 of the imaging unit 2003 is obtained as point F, the center of the imaging area 4504 of the imaging unit 2004 is obtained as point G, and the center of the imaging area 4505 of the imaging unit 2005 is obtained as point H.

[0073] In step S201, it is determined that the image captured by the camera units 2002 to 2005 is displayed on the screen 4000 ( Fig.11 ) on the display areas 4100 and 4200 ( Fig.11 The display position is determined based on the imaging area of ​​the imaging units 2002 to 2005. Fig.11 A screen 4000 of the display unit 1607 is shown. A circular area 4001 is provided at the center of the screen 4000. An image captured by the camera unit 2001 is displayed in the area 4001. A rectangular display area 4100 is provided above the area 4001, and a rectangular display area 4200 is provided below the area 4001. The display areas 4100 and 4200 are each indicated by a dotted line.

[0074] In the second typical embodiment, area 4001 is divided into two areas as an upper area and a lower area (semicircular area 4001a and semicircular area 4001b). If the center of each of the images captured by the camera units 2002 to 2005 is located in area 4001a, the captured image is displayed in display area 4100. If the center of each of the images captured by the camera units 2002 to 2005 is located in area 4001b, the captured image is displayed in display area 4200. If a plurality of images are displayed in each of display area 4100 and display area 4200, each image is scaled and displayed in the same size and arranged at regular intervals. The display position is determined while maintaining the positional relationship between the images in the horizontal direction (i.e., which image of the images is displayed on the left or right of another image). For example, as Fig.11 As shown, it is assumed that point E, point F, and point G are located in area 4001a, and point H is located in area 4001b. In this case, CPU 1601 determines that image 4002 captured by camera unit 2002, image 4003 captured by camera unit 2003, and image 4004 captured by camera unit 2004 are displayed in display area 4100 in the same size and at regular intervals. Fig.11 The CPU 1601 determines to display the image 4005 captured by the imaging unit 2005 in the display area 4200 (the image 4002 , the image 4003 , and the image 4003 are arranged in this order from the left).

[0075] In the second exemplary embodiment, display areas 4100 and 4200 capable of displaying two or more images among images 4002 to 4005 are set on the screen 4000 of the display unit 1607. In the case where two or more images are displayed in each of the display areas 4100 and 4200, the CPU 1601 displays the two or more images in the same size and at regular intervals. In the case where one image is displayed in each of the display areas 4100 and 4200, the CPU 1601 displays the one image at the center of the corresponding display area.

[0076] In step S202, the CPU 1601 displays the images captured by the imaging units 2002 to 2005 on the screen 4000 based on the display positions determined in step S201. Specifically, the images 4002, 4003, and 4004 are displayed in the same size and at regular intervals from the left side of the display area 4100. The image 4005 is displayed at the center of the display area 4200.

[0077] Next, the processing of step S203 will be described. Fig.11The image captured by the camera unit 2001 and displayed on the screen 4000 is displayed in a manner rotated by any angle from the point O shown. In the second exemplary embodiment, it is assumed that the user views the image captured by the camera unit 2001 ( Fig.11 ) and, in response to the designation, display the images 4004 and 4005 in the display area 4200. In order to display the images 4004 and 4005 in the display area 4200, the area 4001 is to be rotated on the screen 4000. The CPU 1601 determines the angle φ by which the area 4001 is to be rotated on the screen 4000. In the following description, it is assumed that an instruction to rotate the image captured by the imaging unit 2001 by the angle φ clockwise when the user designates the objects in the images 4004 and 4005 is input via the input unit 1606.

[0078] In step S203 , the CPU 1601 determines whether an instruction for rotating an image captured by the imaging unit 2001 is input in response to designation of a subject from the user.

[0079] If the CPU 1601 determines that an image rotation instruction is input ("Yes" in step S203), the process proceeds to step S204. In step S204, the CPU 1601 analyzes the instruction (input signal), rotates the image captured by the camera unit 2001 by the angle φ, and displays the rotated image in the area 4001 of the screen 4000. After the image captured by the camera unit 2001 is rotated, the process proceeds to step S205.

[0080] If an image rotation instruction from the user is not detected (NO in step S203 ), the process returns to step S202 .

[0081] In the following description, it is assumed that an instruction for rotating the image captured by the imaging unit 2001 clockwise by an angle φ is input from the user.

[0082] In step S205, the CPU 1601 changes (determines) the display position of the images captured by the camera units 2002 to 2005 on the screen 4000 based on the rotation angle φ input by the user. When the image captured by the camera unit 2001 is rotated clockwise by the angle φ, as shown in FIG. Fig. 12AAs shown, point E also rotates clockwise by an angle φ and moves to point E1. Point F also rotates clockwise by an angle φ and moves to point F1. Point G also rotates clockwise by an angle φ and moves to point G1. Point H also rotates clockwise by an angle φ and moves to point H1. After the rotation, point E1 and point F1 are located in area 4001a, so the CPU 1601 determines to display image 4002 captured by camera unit 2002 and image 4003 captured by camera unit 2003 in the same size and at regular intervals in display area 4100. On the other hand, point G1 and point H1 are located in area 4001b, so the CPU 1601 determines to display image 4004 captured by camera unit 2004 and image 4005 captured by camera unit 2005 in the same size and at regular intervals in display area 4200.

[0083] In step S206, if Fig. 12B As shown, the CPU 1601 displays the image at the display position determined in step S205 on the screen 4000. Specifically, the CPU 1601 displays the image 4002 captured by the camera unit 2002 and the image 4003 captured by the camera unit 2003 in the same size and at regular intervals in the display area 4100, and displays the image 4004 captured by the camera unit 2004 and the image 4005 captured by the camera unit 2005 in the same size and at regular intervals in the display area 4200. After the processing is performed as described above, the processing flow ends.

[0084] Regarding the imaging apparatus 2000 according to the second exemplary embodiment, the imaging apparatus 2000 including a combination of a wide-angle camera (imaging unit) 2001 and a plurality of cameras 2002 to 2005 is configured to display images captured by the plurality of cameras 2002 to 2005 by changing the display positions of the images based on a change in the display of the wide-angle camera 2001. If the images captured by the plurality of cameras 2002 to 2005 do not change when the image captured by the wide-angle camera 2001 is rotated, it is difficult to understand the correspondence between the image captured by the wide-angle camera 2001 and the images captured by the plurality of cameras 2002 to 2005. In the second exemplary embodiment, since the positions of the images captured by the plurality of cameras 2002 to 2005 are changed according to the rotation of the image captured by the wide-angle camera 2001, the correspondence between the images can be easily understood.

[0085] The layout and display positions of the images on the screen 3000 and the screen 4000 described in the first exemplary embodiment and the second exemplary embodiment are merely examples. Any display device may be used as long as the display device can change the display of an image based on a change in the display position of an image captured by another imaging unit relative to the image captured by the wide-angle imaging unit. The imaging units 2002 to 2005 do not necessarily have to include a zoom function, and the imaging direction may be changed only in the pan direction (or only in the tilt direction).

[0086] Although the exemplary embodiments of the present disclosure are described above, the present disclosure is not limited to the exemplary embodiments described above, and various modifications and changes can be made within the gist of the present disclosure.

[0087] The above exemplary embodiments are merely examples for implementing the present disclosure. The technical scope of the present disclosure should not be interpreted in a limited manner due to the above exemplary embodiments. In other words, the present disclosure may be implemented in various ways without departing from the technical concept and main features of the present disclosure.

[0088] The present disclosure may also be implemented by providing a program for implementing one or more functions according to the above-described typical embodiments to a system or device via a network or storage medium, and reading and executing the program by one or more processors in a computer of the system or device. The present disclosure may also be implemented by a circuit (e.g., ASIC) for implementing one or more functions according to the above-described typical embodiments.

[0089] The above exemplary embodiments are merely examples for implementing the present disclosure. The technical scope of the present disclosure should not be interpreted in a limited manner due to the above exemplary embodiments. In other words, the present disclosure may be implemented in various ways without departing from the technical concept and main features of the present disclosure.

[0090] Other embodiments

[0091] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0092] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: a first obtaining unit configured to obtain a first image from a first camera unit, wherein the first camera unit is configured to capture an image at a predetermined viewing angle; a second obtaining unit configured to obtain a second image from a second camera unit, wherein the second camera unit is configured to capture an image of a region overlapping with a region where the first camera unit captures an image at an angle of view narrower than the predetermined angle of view; as well as a control unit configured to control so that the first image and the second image are displayed on a display unit, Wherein, when the first image is rotated by a predetermined angle on the display unit, the control unit changes a display position of the second image on the screen of the display unit based on the predetermined angle.

2. The information processing device according to claim 1, wherein: The first camera unit includes an ultra-wide-angle lens and is configured to capture a 360-degree image.

3. The information processing device according to claim 1, wherein: The second imaging unit captures an image in a range narrower than a range in which the first imaging unit captures an image.

4. The information processing device according to claim 1, in, A plurality of second camera units are arranged around the first camera unit, wherein each of the plurality of second camera units captures an image of an area overlapping with an area captured by the first camera unit, and The regions where the images are captured by the multiple second camera units do not overlap with each other.

5. The information processing device according to claim 4, in, The second obtaining unit obtains the second image from each of the plurality of second imaging units, and The control unit performs control to display the first image at the center of the screen of the display unit and to display a plurality of second images around the first image.

6. The information processing device according to claim 5, wherein: The control unit determines a display position of each of the plurality of second images on a screen of the display unit based on a position where an area where an image is captured by each of the plurality of second camera units overlaps with an area where an image is captured by the first camera unit.

7. The information processing device according to claim 6, wherein: The control unit determines a display position of each of the plurality of second images on a screen of the display unit based on a center position of an area where each of the plurality of second imaging units captures an image.

8. The information processing device according to claim 7, wherein: Each of the plurality of second camera units is configured to change a camera direction of at least one of a pan direction and a tilt direction.

9. The information processing device according to claim 8, further comprising a third obtaining unit configured to obtain an imaging angle of each of the plurality of second imaging units, in, The control unit determines a center position of an area where each of the plurality of second imaging units captures an image based on the imaging angle obtained by the third obtaining unit.

10. The information processing device according to claim 5, in, setting display areas of the same number as the number of the plurality of second images on the screen of the display unit to individually display the plurality of second images, and Wherein, when the first image is rotated by a predetermined angle on the screen of the display unit, the control unit determines a display area in which each of the plurality of second images is to be displayed based on the predetermined angle.

11. The information processing device according to claim 5, in, A display area capable of displaying two or more second images among the plurality of second images is set on the screen of the display unit, and in a case where the two or more second images are displayed in the display area, the control unit performs control to display the two or more second images so that the two or more second images have the same size and are arranged at regular intervals, and Wherein, in a case where one second image among the plurality of second images is displayed in the display area, the control unit performs control to display the one second image at the center of the display area.

12. The information processing device according to claim 1, wherein: In a case where a subject is specified in the first image, the control unit determines an angle by which the first image is to be rotated based on the specification, and the angle is used as the predetermined angle.

13. An information processing method, comprising: obtaining a first image from a first camera unit, wherein the first camera unit is configured to capture an image at a predetermined viewing angle; obtaining a second image from a second camera unit, wherein the second camera unit is configured to capture an image of an area overlapping with an area where the first camera unit captures an image, at an angle of view narrower than the predetermined angle of view; displaying the first image and the second image on a screen of a display unit; as well as In a case where the first image is rotated by a predetermined angle on the display unit, a display position of the second image on the screen of the display unit is changed based on the predetermined angle.

14. A non-transitory computer-readable storage medium storing a program configured to cause a computer to execute a method, the method comprising: obtaining a first image from a first camera unit, wherein the first camera unit is configured to capture an image at a predetermined viewing angle; obtaining a second image from a second camera unit, wherein the second camera unit is configured to capture an image of an area overlapping with an area where the first camera unit captures an image, at an angle of view narrower than the predetermined angle of view; displaying the first image and the second image on a display unit; as well as In a case where the first image is rotated by a predetermined angle on the screen of the display unit, a display position of the second image on the screen of the display unit is changed based on the predetermined angle.

15. A computer program product comprising a program configured to cause a computer to perform a method comprising: obtaining a first image from a first camera unit, wherein the first camera unit is configured to capture an image at a predetermined viewing angle; obtaining a second image from a second camera unit, wherein the second camera unit is configured to capture an image of an area overlapping with an area where the first camera unit captures an image, at an angle of view narrower than the predetermined angle of view; displaying the first image and the second image on a display unit; as well as In a case where the first image is rotated by a predetermined angle on the screen of the display unit, a display position of the second image on the screen of the display unit is changed based on the predetermined angle.

Images