Information processing system and program

Abstract

To enable work for collecting information associated with positional information of an image to be further easily performed.SOLUTION: An input device, which accepts input of a diagnostic result of a diagnostic target in a structure, comprises display means for displaying an entire celestial sphere image of the structure and an acceptance means for accepting input of a position indicating the diagnostic target with respect to the entire celestial sphere image and storing positional information indicating the accepted position in storage means. The display means displays the entire celestial sphere image and a diagnostic information input screen of accepting input of diagnostic information including the diagnostic result of the diagnostic target, on one screen. The acceptance means accepts input of the diagnostic information according to the diagnostic information input screen and stores the diagnostic information and the positional information in association with each other in the storage means.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an input device and an input method for an input device, as well as an output device and an output method for an output device. [Background technology]

[0002] In recent years, the importance of on-site inspections of buildings has been increasing. In particular, there are many things that cannot be understood without on-site inspections, such as inspections for repairs, construction work planning, equipment renewal, and design. There is a known technology that captures images of the target of an on-site inspection using an imaging device and stores the captured image data, thereby reducing the labor required for inspection work and sharing information.

[0003] Furthermore, Patent Document 1 discloses a technique in which location information of a predetermined location on a three-dimensional space model is included in search conditions, and the search conditions are configured to be extractable, thereby enabling the surrounding situation to be shared from the location information. Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when inspecting for defects such as cracks, the inspection results must be compiled and reported in a report. In this case, the inspector can search for associated information based on location information indicating the location on the 3D space model. However, in the past, when compiling the inspection results in a report, the task of matching the location information on the 3D space model with the defects was performed manually, which made the report creation process cumbersome and time-consuming.

[0005] The present invention has been made in view of the above, and has an object to make it easier to consolidate information associated with position information on an image. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides an input device for inputting a diagnosis result of a diagnosis target in a structure, the input device including: a display means for displaying a panoramic image of the structure; and a receiving means for receiving an input of a position indicating the diagnosis target relative to the panoramic image and storing position information indicating the received position in a storage means, wherein the display means displays, on one screen, the panoramic image and a diagnostic information input screen for receiving an input of diagnostic information including a diagnosis result of the diagnosis target, and the receiving means receives an input of diagnostic information according to the diagnostic information input screen and stores the diagnostic information and the position information in association with each other in the storage means. Effect of the Invention

[0007] According to the present invention, it is possible to more easily aggregate information associated with position information on an image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an outline of an external appearance of an example of an imaging device applicable to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the structure of an imaging body applicable to the first embodiment. [Diagram 3] FIG. 3 is a three-sided view that illustrates a schematic external view of the imaging device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of a configuration of the imaging device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the arrangement of a battery and a circuit unit in the imaging device according to the first embodiment. [Figure 6] FIG. 6 is a block diagram showing a configuration of an example of an information processing device serving as an input device for inputting annotations, which is applicable to the first embodiment. [Figure 7] FIG. 7 is a functional block diagram of an example for explaining functions of the information processing device according to the first embodiment. [Figure 8]FIG. 8 is a diagram for explaining how the imaging lens applicable to the first embodiment projects three-dimensional incident light into two dimensions. [Figure 9] FIG. 9 is a diagram for explaining the inclination of the imaging device according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a format of a celestial sphere image applicable to the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining correspondence between each pixel position of a hemispherical image and each pixel position of a celestial sphere image according to a conversion table, which is applicable to the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining vertical correction applicable to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of the configuration of an information processing system applicable to the first embodiment. [Figure 14] FIG. 14 is a diagram for explaining an example of imaging using the imaging device according to the first embodiment. [Figure 15-1] FIG. 15-1 is a diagram for roughly explaining the annotation input process in the information processing system according to the first embodiment. [Figure 15-2] FIG. 15-2 is a diagram for roughly explaining the annotation input process in the information processing system according to the first embodiment. [Figure 15-3] FIG. 15-3 is a diagram for roughly explaining the annotation input process in the information processing system according to the first embodiment. [Figure 16] FIG. 16 is a flowchart illustrating an example of an annotation input process according to the first embodiment. [Figure 17] FIG. 17 is a diagram for explaining an image cropping process applicable to the first embodiment. [Figure 18] FIG. 18 is a diagram illustrating an example of a screen displayed on a display device by a UI unit according to the first embodiment. [Figure 19]FIG. 19 is a diagram showing an example of a floor plan image applicable to the first embodiment. [Figure 20] FIG. 20 is a diagram showing an example in which an annotation input screen according to the first embodiment is displayed. [Figure 21] FIG. 21 is a diagram showing an example of an annotation input screen that has been switched to a region designation screen, which is applicable to the first embodiment. [Figure 22] FIG. 22 is a diagram showing an example of a display of a screen for designating a cutout region according to the first embodiment. [Diagram 23] FIG. 23 is a diagram showing an example in which a marker is displayed on a cut-out image displayed in a cut-out image display area according to the first embodiment. [Figure 24] FIG. 24 is a diagram showing an example of a display screen relating to report data creation according to the first embodiment. [Diagram 25] FIG. 25 is a diagram illustrating an example of report data applicable to the first embodiment. [Figure 26] FIG. 26 is a diagram showing an example in which a crack is observed on a wall surface, which is applicable to the modified example of the first embodiment. [Figure 27] FIG. 27 is a diagram showing an example of an annotation input screen for inputting an annotation for a specified crack according to a modified example of the first embodiment. [Figure 28] FIG. 28 is a diagram showing an example of an annotation input screen that has been switched to an area designation screen, which is applicable to the modified example of the first embodiment. [Figure 29] FIG. 29 is a diagram showing an example of a display of a screen for designating a cutout region according to a modification of the first embodiment. [Diagram 30] FIG. 30 is a diagram illustrating an imaging device according to the second embodiment. [Diagram 31] FIG. 31 is a diagram showing an example of an imaging range that can be imaged by each imaging body applicable to the second embodiment. [Diagram 32]FIG. 32 is a functional block diagram of an example illustrating functions of an information processing device serving as an input device for inputting annotations according to the second embodiment. [Diagram 33] FIG. 33 is a diagram showing an example of images captured from five different viewpoints and synthesized for each imaging body in an imaging device applicable to the second embodiment. [Diagram 34] FIG. 34 is a flowchart illustrating an example of a process for creating a 3D reconstruction model applicable to the second embodiment. [Diagram 35] FIG. 35 is a diagram for explaining triangulation applicable to the second embodiment. [Diagram 36] FIG. 36 is a diagram for explaining the principle of EPI applicable to the second embodiment. [Figure 37] FIG. 37 is a diagram for explaining the principle of EPI applicable to the second embodiment. [Figure 38] FIG. 38 is a diagram for explaining the point that the slope m becomes a value based on a curve when the EPI is configured using a panoramic image, which is applicable to the second embodiment. [Figure 39] FIG. 39 is a diagram for explaining the point that the slope m becomes a value based on a curve when the EPI is configured using a panoramic image, which is applicable to the second embodiment. [Diagram 40] FIG. 40 is a diagram showing an example of a plane for which parallax is preferentially calculated based on an image captured by an imaging device according to the second embodiment. [Diagram 41] FIG. 41 is a diagram showing an example of a large space including a large building as a target for creating a 3D reconstruction model according to the second embodiment. [Diagram 42] FIG. 42 is a block diagram illustrating an example of a configuration of an imaging device according to an embodiment. [Diagram 43] FIG. 43 is a block diagram showing an example of a configuration of a control unit and a memory in the imaging device according to the second embodiment. As shown in FIG. [Diagram 44] FIG. 44 is a diagram for explaining position designation according to the second embodiment. [Diagram 45]FIG. 45 is a diagram showing an example in which a position is specified by a line according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an input device, an input method for an input device, and an output device and an output method for an output device will be described in detail with reference to the accompanying drawings.

[0010] (First embodiment) In the first embodiment, an annotation is input by specifying a position on a spherical image of a diagnosis target captured by an imaging device capable of capturing an image of the celestial sphere. The input annotation is stored in association with the position.

[0011] By using a spherical image to capture an image of a diagnosis target, the diagnosis target can be captured without omission with a small number of captures. Furthermore, since an annotation is input by specifying a position on the spherical image, the target to which the annotation is input can be intuitively grasped, and input omissions can be suppressed. Furthermore, since the input annotation is associated with a position on the spherical image and saved, in the task of creating a report based on the annotation, it becomes easy to grasp the correspondence between the annotation and the target to which the annotation is input, and the burden associated with the report creation process can be reduced. This also suppresses the occurrence of human errors in report creation.

[0012] The omnidirectional image is an image captured with a 360° angle of view in each of two orthogonal planes (e.g., a horizontal plane and a vertical plane) relative to the imaging position, that is, with a 4π steradian angle of view relative to the imaging position. An annotation is information based on the results of observing and diagnosing a diagnostic object. An annotation can include one or more items predetermined for each type of diagnostic object and an arbitrary comment. An annotation can further include an image.

[0013] [Image capture device applicable to the first embodiment] Fig. 1 is a diagram showing a schematic external view of an example of an imaging device applicable to the first embodiment. In Fig. 1, an imaging device 1a has an imaging lens 20a provided on a first surface of a substantially rectangular parallelepiped housing 10a, and an imaging lens 20b provided on a second surface opposite to the first surface at a position corresponding to the imaging lens 20a. Imaging elements are provided in the housing 10a in correspondence with the imaging lenses 20a and 20b, respectively.

[0014] The light incident on each of the imaging lenses 20a and 20b is irradiated onto the corresponding imaging element via an imaging optical system including the imaging lenses 20a and 20b, which is provided in the housing 10a. The imaging element is, for example, a CCD (Charge Coupled Device), which is a light receiving element that converts the irradiated light into an electric charge. The imaging element is not limited to this, and may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0015] Although details will be described later, each driver that drives each image sensor performs shutter control for each image sensor in response to a trigger signal, and reads out electric charges converted from light from each image sensor. Each driver converts the electric charges read out from each image sensor into electric signals, and converts the electric signals into captured images as digital data and outputs them. Each captured image output from each driver is stored in a memory, for example.

[0016] In the following description, a configuration including the pair of imaging lenses 20a and 20b, and the image forming optical systems and imaging elements corresponding to the pair of imaging lenses is referred to as imaging body 21. For convenience, the operation of outputting an image captured based on light incident on imaging lenses 20a and 20b in response to a trigger signal is referred to as imaging by imaging body 21. When it is necessary to distinguish between imaging lenses 20a and 20b, it is referred to as imaging by imaging lens 20a.

[0017] The shutter button 30 is a button for instructing imaging by the imaging body 21 in response to operation. When the shutter button 30 is operated, imaging is synchronously performed by the imaging lenses 20a and 20b in the imaging body 21.

[0018] 1, the housing 10a of the imaging device 1a includes an imaging section area 2a in which the imaging body 21 is disposed, and an operation section area 3a in which the shutter button 30 is disposed. The operation section area 3a includes a grip section 31 with which a user holds the imaging device 1a. The grip section 31 is formed to have a less slippery surface so that the user can easily hold and operate it. In addition, a fixing section 32 is provided on the bottom of the imaging device 1a, i.e., on the bottom of the grip section 31, for fixing the imaging device 1a to a tripod or the like.

[0019] Next, the structure of the imaging body 21 will be described in more detail. FIG. 2 is a diagram showing an example of the structure of the imaging body 21 applicable to the first embodiment. In FIG. 2, the imaging body 21 includes imaging optical systems 201a and 201b including imaging lenses 20a and 20b, respectively, and imaging elements 200a and 200b using CCD or CMOS sensors. Each of the imaging optical systems 201a and 201b is configured as, for example, a fisheye lens with 6 groups and 7 lenses. This fisheye lens has a total angle of view of 180° (=360° / n, n is the number of optical systems=2) or more, preferably greater than 180°, more preferably 185° or more, and more preferably 190° or more.

[0020] Each of the imaging optical systems 201a and 201b includes a prism 202a or 202b that changes the optical path by 90°. The 6 groups and 7 fisheye lenses included in each of the imaging optical systems 201a and 201b can be divided into a group on the entrance side of the prisms 202a and 202b and a group on the exit side (image pickup elements 200a and 200b side). For example, in the imaging optical system 201a, light incident on the imaging lens 20a is incident on the prism 202a via each lens belonging to the group on the entrance side of the prism 202a. The light incident on the prism 202a has its optical path changed by 90° and is irradiated onto the image pickup element 200a via each lens, aperture stop, and filter belonging to the group on the exit side of the prism 202a.

[0021] The optical elements (lenses, prisms 202a and 202b, filters, and aperture stops) of the two imaging optical systems 201a and 201b are positioned relative to the image sensors 200a and 200b. More specifically, the optical elements of the imaging optical systems 201a and 201b are positioned so that the optical axes of the optical elements are positioned perpendicular to the centers of the light receiving areas of the corresponding image sensors 200a and 200b, and the light receiving areas are positioned as the image forming planes of the corresponding fisheye lenses. In the imaging body 21, the imaging optical systems 201a and 201b are of the same specifications, and are combined in opposite directions so that their optical axes match.

[0022] Fig. 3 is a three-sided view showing a schematic external view of an imaging device 1a according to an embodiment. Fig. 3(a), Fig. 3(b) and Fig. 3(c) are examples of a top view, a front view and a side view of the imaging device 1a, respectively. As shown in Fig. 3(c), an imaging body 21 is formed by an imaging lens 20a and an imaging lens 20b arranged at a position corresponding to the imaging lens 20a on the rear side of the imaging lens 20a.

[0023] In Fig. 3(a) and Fig. 3(c), the angle α indicates an example of the angle of view (imaging range) of the imaging lenses 20a and 20b. As described above, each of the imaging lenses 20a and 20b included in the imaging body 21 captures an image with an angle α larger than 180°. Therefore, in order to prevent the housing 10a from being reflected in each captured image captured by each of the imaging lenses 20a and 20b, the first and second surfaces of the housing 10a are chamfered on both sides of the center line C of each of the imaging lenses 20a and 20b according to the angle of view of each of the imaging lenses 20a and 20b, as shown as surfaces 231, 232, 233, and 234 in Fig. 3(a), Fig. 3(b), and Fig. 3(c).

[0024] In addition, center line C is a line that passes vertically through the center of each imaging lens 20a and 20b when the orientation of imaging device 1a is set so that the direction of the apex (pole) of each hemispherical image captured by imaging device 1a is parallel to the vertical direction.

[0025] The imaging body 21, by combining the imaging lenses 20a and 20b, has an imaging range of the entire celestial sphere centered on the center of the imaging body 21. That is, as described above, the imaging lenses 20a and 20b each have an angle of view of 180° or more, preferably greater than 180°, and more preferably 185° or more. Therefore, by combining these imaging lenses 20a and 20b, for example, the imaging range of a plane perpendicular to the first plane and the imaging range of a plane parallel to the first plane can each be 360°, and by combining these, an imaging range of the entire celestial sphere is realized.

[0026] [Configuration of signal processing in the imaging device according to the first embodiment] Fig. 4 is a block diagram showing an example of the configuration of an imaging device 1a according to the first embodiment. In Fig. 4, the same reference numerals are used for parts corresponding to those in Fig. 1 and Fig. 2 described above, and detailed description thereof will be omitted.

[0027] 4, the imaging device 1a includes, as an imaging system configuration, an imaging body 21 including imaging elements 200a and 200b and driving units 210a and 210b, and signal processing units 211a and 211b. The imaging device 1a also includes, as an imaging control and signal processing system configuration, a CPU (Central Processing Unit) 2000, a ROM (Read Only Memory) 2001, a memory controller 2002, a RAM (Random Access Memory) 2003, a trigger I / F 2004, a SW (switch) circuit 2005, a data I / F 2006, a communication I / F 2007, and an acceleration sensor 2008, each of which is connected to a bus 2010. The imaging device 1a also includes a battery 2020 for supplying power to each of these units.

[0028] First, the configuration of the imaging system will be described. In the imaging system, the imaging element 200a and the driving section 210a correspond to the imaging lens 20a. Similarly, the imaging element 200b and the driving section 210b correspond to the imaging lens 20b. The imaging element 200a and the driving section 210a, as well as the imaging element 200b and the driving section 210b, are included in the imaging body 21.

[0029] In the imaging body 21, the driving unit 210a drives the imaging element 200a and reads out charges from the imaging element 200a in response to a trigger signal supplied from the trigger I / F 2004. The driving unit 210a outputs one frame of an imaged image based on the charges read out from the imaging element 200a in response to one trigger signal. The driving unit 210a converts the charges read out from the imaging element 200a into an electric signal and outputs it. The signal output from the driving unit 210a is supplied to the signal processing unit 211a. The signal processing unit 211a performs predetermined signal processing such as noise removal and gain adjustment on the signal supplied from the driving unit 210a, converts the analog signal into a digital signal, and outputs it as an imaged image that is digital data. This imaged image is a hemispherical image (fisheye image) in which a hemispherical area corresponding to the angle of view of the imaging lens 20a is captured out of the entire celestial sphere.

[0030] In the imaging body 21, the imaging element 200b and the driving section 210b have the same functions as the imaging element 200a and the driving section 210a described above, and therefore their description will be omitted here. Also, the signal processing section 211b has the same functions as the signal processing section 211a described above, and therefore their description will be omitted here.

[0031] The hemispherical images output from the signal processing units 211 a and 211 b are stored in the RAM 2003 via the memory controller 2002 .

[0032] Next, the configuration of the imaging control and signal processing system will be described. The CPU 2000 operates according to a program stored in advance in, for example, the ROM 2001, using a part of the storage area of ​​the RAM 2003 as a work memory, and controls the overall operation of this imaging device 1a. The memory controller 2002 controls the storage and reading of data in the RAM 2003 according to instructions from the CPU 2000.

[0033] The SW circuit 2005 detects an operation of the shutter button 30 and passes the detection result to the CPU 2000. When the CPU 2000 receives a detection result indicating that an operation of the shutter button 30 has been detected from the SW circuit 2005, it outputs a trigger signal. The trigger signal is output via a trigger I / F 2004, branched, and supplied to each of the drive units 210a and 210b.

[0034] The data I / F 2006 is an interface for performing data communication with an external device. For example, a USB (Universal Serial Bus) or Bluetooth (registered trademark) can be used as the data I / F 2006. The communication I / F 2007 is connected to a network and controls communication with the network. The network connected to the communication I / F 2007 may be either a wired or wireless network, or may be connectable to both a wired and wireless network.

[0035] In the above description, the CPU 2000 outputs a trigger signal in response to the detection result by the SW circuit 2005, but this is not limited to this example. For example, the CPU 2000 may output a trigger signal in response to a signal supplied via the data I / F 2006 or the communication I / F 2007. Furthermore, the trigger I / F 2004 may generate a trigger signal in response to the detection result of the SW circuit 2005 and supply it to each of the driving units 210a and 210b.

[0036] The acceleration sensor 2008 detects three-axis acceleration components and passes the detection results to the CPU 2000. The CPU 2000 detects the vertical direction based on the detection results of the acceleration sensor 2008, and obtains the inclination of the imaging device 1a with respect to the vertical direction. The CPU 2000 adds inclination information indicating the inclination of the imaging device 1a to each hemispherical image captured by the imaging lenses 20a and 20b in the imaging body 21 and stored in the RAM 2003.

[0037] The battery 2020 is a secondary battery such as a lithium ion secondary battery, and serves as a power supply unit that supplies power to each unit in the imaging device 1a that requires power supply. The battery 2020 includes a charge / discharge control circuit that controls charging and discharging of the secondary battery.

[0038] FIG. 5 is a diagram showing an example of the arrangement of the battery 2020 and the circuit section 2030 in the imaging device 1a according to the embodiment. In this way, the battery 2020 and the circuit section 2030 are provided inside the housing 10a. Furthermore, at least the battery 2020 out of the battery 2020 and the circuit section 2030 is fixed inside the housing 10a by a fixing means such as an adhesive or a screw. Note that FIG. 5 corresponds to the above-mentioned FIG. 3(b) and shows an example of the imaging device 1a as viewed from the front. The circuit section 2030 includes, for example, each element of the above-mentioned imaging control and signal processing system, and is configured, for example, on one or more circuit boards.

[0039] Note that, although it has been described here that the battery 2020 and the circuit section 2030 are arranged in that position, this is not limited to this example, and for example, if the circuit section 2030 is sufficiently small, it is sufficient that at least the battery 2020 is arranged in that position.

[0040] In this configuration, a trigger signal is output from the trigger I / F 2004 in response to an operation of the shutter button 30. The trigger signal is supplied to the driving units 210a and 210b at the same timing. The driving units 210a and 210b read out electric charges from the imaging elements 200a and 200b in synchronization with the supplied trigger signal.

[0041] The driving units 210a and 210b convert the electric charges read from the imaging elements 200a and 200b into electric signals and supply them to the signal processing units 211a and 211b, respectively. The signal processing units 211a and 211b perform the above-mentioned predetermined processing on the signals supplied from the driving units 210a and 210b, convert these signals into hemispherical images, and output them. Each hemispherical image is stored in the RAM 2003 via the memory controller 2002.

[0042] Each hemispherical image stored in the RAM 2003 is sent to an external information processing device via the data I / F 2006 or the communication I / F 2007.

[0043] [Overview of image processing according to the first embodiment] Next, image processing for a hemispherical image according to the first embodiment will be described. Fig. 6 is a block diagram showing an example of a configuration of an information processing device as an input device for inputting annotations, applicable to the first embodiment. In Fig. 6, the information processing device 100a includes a CPU 1000, a ROM 1001, a RAM 1002, a graphic I / F 1003, a storage 1004, a data I / F 1005, a communication I / F 1006, and an input device 1011, each of which is connected by a bus 1030, and further includes a display device 1010 connected to the graphic I / F 1003.

[0044] The storage 1004 is a non-volatile memory such as a flash memory, and stores programs for operating the CPU 1000 and various data. A hard disk drive may be applied to the storage 1004. The CPU 1000 operates according to the programs stored in the ROM 1001 and the storage 1004, using the RAM 1002 as a work memory, and controls the overall operation of the information processing device 100a.

[0045] The graphic I / F 1003 generates a display signal that can be displayed by the display device 1010 based on a display control signal generated by the CPU 1000 according to a program, and sends the generated display signal to the display device 1010. The display device 1010 includes, for example, an LCD (Liquid Crystal Display) and a drive circuit for driving the LCD, and displays a screen according to the display signal sent from the graphic I / F 1003.

[0046] The input device 1011 outputs a signal according to a user operation and accepts a user input. In this example, the information processing device 100a is a tablet personal computer, and is configured as a touch panel 1020 in which the display device 1010 and the input device 1011 are integrally formed. The input device 1011 allows the display of the display device 1010 to be seen through, and outputs position information according to a contact position. The information processing device 100a is not limited to a tablet personal computer, and may be, for example, a desktop personal computer.

[0047] The data I / F 1005 is an interface for performing data communication with an external device. For example, USB or Bluetooth (registered trademark) can be used as the data I / F 1005. The communication I / F 1006 is connected to a network via wireless communication and controls communication with the network. The communication I / F 1006 may be connected to the network via wired communication. Here, it is assumed that the imaging device 1a and the information processing device 100a are connected via a wired connection via the data I / F 1005.

[0048] Fig. 7 is a functional block diagram of an example for explaining functions of the information processing device 100a according to the first embodiment. In Fig. 7, the information processing device 100a includes an image acquisition unit 110, an image processing unit 111, an auxiliary information generation unit 112, a UI (User Interface) unit 113, a communication unit 114, and an output unit 115.

[0049] The image acquisition unit 110, image processing unit 111, incidental information generation unit 112, UI unit 113, communication unit 114, and output unit 115 are realized by an input program according to the first embodiment running on the CPU 1000. Without being limited to this, some or all of the image acquisition unit 110, image processing unit 111, incidental information generation unit 112, UI unit 113, communication unit 114, and output unit 115 may be configured by hardware circuits operating in cooperation with each other.

[0050] The UI unit 113 is a receiving unit that receives a user input in response to a user operation on the input device 1022 and executes a process in response to the received user input. The UI unit 113 can store information received in response to the user input in the RAM 1002 or the storage 1004. The UI unit 113 also serves as a display unit that generates a screen to be displayed on the display device 1010. The communication unit 114 controls communication via the data I / F 1005 and the communication I / F 1006.

[0051] The image acquisition unit 110 acquires each hemispherical image captured by the imaging lenses 20a and 20b from the imaging device 1a and tilt information added to each hemispherical image via the data I / F 1005. The image processing unit 111 executes image conversion processing to generate one omnidirectional image by connecting each hemispherical image acquired by the image acquisition unit 110. The incidental information generation unit 112 generates incidental information (annotation) at a specified position for the omnidirectional image generated by the image processing unit 111 in response to a user input accepted by the UI unit 113. The output unit 115 creates and outputs a report in a predetermined format based on the annotation generated by the incidental information generation unit 112.

[0052] An input program for realizing each function according to the first embodiment in the information processing device 100a is provided by being recorded in a computer-readable recording medium such as a CD (Compact Disk), a flexible disk (FD), or a DVD (Digital Versatile Disk) in an installable or executable format. Without being limited to this, the input program may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The input program may also be configured to be provided or distributed via a network such as the Internet.

[0053] The input program has a modular configuration including an image acquisition unit 110, an image processing unit 111, an incidental information generation unit 112, a UI unit 113, a communication unit 114, and an output unit 115. In terms of actual hardware, the CPU 1000 reads out the input program from a storage medium such as storage 1004 and executes it, whereby the above-mentioned units are loaded onto a main storage device such as RAM 1002, and the image acquisition unit 110, the image processing unit 111, the incidental information generation unit 112, the UI unit 113, the communication unit 114, and the output unit 115 are generated on the main storage device.

[0054] [Image conversion applicable to the first embodiment] Next, image conversion processing applicable to the first embodiment will be described with reference to Figures 8 to 12. The image conversion processing described below is, for example, processing executed by the image processing unit 111 in the information processing device 100a.

[0055] 8 is a diagram for explaining how the imaging lenses 20a and 20b applicable to the first embodiment project three-dimensional incident light into two dimensions. In the following, unless otherwise specified, the imaging lens 20a will be used as a representative of the imaging lenses 20a and 20b.

[0056] In Fig. 8(a), the imaging lens 20a includes a fisheye lens 24 (imaging optical system 201a) and an imaging element 200a. The axis perpendicular to the light receiving surface of the imaging element 200a is defined as the optical axis. In the example of Fig. 6(a), the incidence angle φ is expressed as the angle with respect to the optical axis when the intersection point between the plane tangent to the edge of the fisheye lens 24 and the optical axis is defined as the vertex.

[0057] A fisheye image (hemispherical image) captured by a fisheye lens 24 with an angle of view exceeding 180° is an image of a scene equivalent to a hemisphere from the capture position. However, as shown in FIG. 8(a) and FIG. 8(b), a hemispherical image 22 is generated at an image height h corresponding to an incident angle φ, the relationship of which is determined by a projection function f(φ). In FIG. 8(b), an area on the imaging surface of the image sensor 200a where light from the fisheye lens 24 is not irradiated is shown as an invalid area filled in black. The projection function f(φ) varies depending on the properties of the fisheye lens 24. For example, there is a fisheye lens 24 that has a projection method called an equidistant projection method, which is expressed by the following formula (1) where the image height h, focal length f, and angle (incident angle) between the incident direction and the optical axis are φ, and this will be used in this example.

[0058]

number

[0059] Fig. 9 is a diagram for explaining the inclination of the imaging device 1a according to the first embodiment. In Fig. 9, the vertical direction coincides with the z-axis in the Cartesian coordinate system of the xyz three-dimensional directions of the global coordinate system. When this direction is parallel to the center line C of the imaging device 1a shown in Fig. 3(b), the camera is not inclined. When these are not parallel, the imaging device 1a is in a tilted state.

[0060] The imaging device 1a associates each hemispherical image captured by the imaging lenses 20a and 20b with an output value output from the acceleration sensor 2008 at the time of capturing the image, and stores the associated images in, for example, the RAM 2003. The information processing device 100a acquires each hemispherical image and the output value of the acceleration sensor 2008 stored in the RAM 2003 from the imaging device 1a.

[0061] In the information processing device 100a, the image processing unit 111 calculates the tilt angle from the gravity vector (hereinafter, tilt angle α) and the tilt angle β in the xy plane (hereinafter, tilt angle β) by the following formulas (2) and (3) using the output value of the acceleration sensor 2008 acquired from the imaging device 1a. Here, the value Ax indicates the value of the x0 axis direction component of the output value of the acceleration sensor 2008 in the camera coordinate system, the value Ay indicates the value of the y0 axis direction component of the output value of the acceleration sensor 2008 in the camera coordinate system, and the value Az indicates the value of the z0 axis direction component of the output value of the acceleration sensor 2008 in the camera coordinate system. The image processing unit 111 calculates the tilt angles α and β from the values ​​of the axial components of the acceleration sensor 2008 in response to the trigger signal. The image processing unit 111 stores the calculated tilt angles α and β in, for example, the storage 1004 in association with each hemispherical image acquired from the imaging device 1a.

[0062]

number

[0063]

number

[0064] The image processing unit 111 generates a spherical image based on each hemispherical image acquired from the imaging device 1a and the tilt angles α and β associated with each hemispherical image.

[0065] Fig. 10 is a diagram for explaining a format of a celestial sphere image applicable to the first embodiment. Fig. 10(a) shows an example of a format when the celestial sphere image is represented on a plane, and Fig. 10(b) shows an example of a format when the celestial sphere image is represented on a spherical surface. When represented on a plane, the format of the celestial sphere image is an image having pixel values ​​corresponding to angle coordinates (φ, θ) with a horizontal angle of 0° to 360° and a vertical angle of 0° to 180° as shown in Fig. 10(a). The angle coordinates (φ, θ) are associated with each point on the spherical surface shown in Fig. 10(b), and are similar to latitude and longitude coordinates on a globe.

[0066] The relationship between the planar coordinate values ​​of an image captured by a fisheye lens and the spherical coordinate values ​​of a celestial sphere image can be established by using the projection function f (h=f(θ)) as described in Fig. 8. This makes it possible to create a celestial sphere image based on a plane as shown in Fig. 10(a) by converting and combining (composing) two partial images (hemispherical images) captured by a fisheye lens.

[0067] In the first embodiment, a conversion table that associates each pixel position of a hemispherical image with each pixel position in a celestial sphere image based on the plane shown in Fig. 10(a) is created in advance and stored in, for example, the storage 1004 of the information processing device 100a. Table 1 shows an example of this conversion table. Fig. 11 is a diagram for explaining the association between each pixel position of a hemispherical image and each pixel position of a celestial sphere image based on the conversion table, which is applicable to the first embodiment.

[0068] [Table 1]

[0069] The conversion table has a data set of coordinate values ​​(θ,φ) [pix: pixels] of the converted image and corresponding coordinate values ​​(x,y) [pix] of the pre-converted image for all coordinate values ​​of the converted image, as shown in Table 1. A converted image can be generated from a captured hemispherical image (pre-converted image) according to the conversion table shown in Table 1. Specifically, as shown in FIG. 11, from the correspondence between pre-converted coordinates and post-converted coordinates shown in the conversion table of Table 1, each pixel of the converted image can be generated by referring to the pixel value of the coordinate value (x,y) [pix] of the pre-converted image corresponding to the coordinate value (θ,φ) [pix].

[0070] The conversion table shown in Table 1 reflects distortion correction assuming that the direction of the center line C of the imaging device 1a is parallel to the vertical direction. By applying a correction process to this conversion table according to the tilt angles α and β, a correction (called vertical correction) can be performed to make the center line C of the imaging device 1a parallel to the vertical direction.

[0071] Fig. 12 is a diagram for explaining vertical correction applicable to the first embodiment. Fig. 12(a) shows a camera coordinate system, and Fig. 12(b) shows a global coordinate system. In Fig. 12(b), the three-dimensional orthogonal coordinates of the global coordinate system are expressed as (x1, y1, z1), and the spherical coordinates are expressed as (θ1, φ1). In Fig. 12(a), the three-dimensional orthogonal coordinates of the camera coordinate system are expressed as (x0, y0, z0), and the spherical coordinates are expressed as (θ0, φ0).

[0072] Image processing unit 111 performs conversion from spherical coordinates (θ1, φ1) to spherical coordinates (θ0, φ0) using the following formulas (4) to (9) through vertical correction calculation. First, since rotational conversion using three-dimensional orthogonal coordinates is necessary to correct the tilt, image processing unit 111 performs conversion from spherical coordinates (θ1, φ1) to three-dimensional orthogonal coordinates (x1, y1, z1) using formulas (4) to (6).

[0073]

number

[0074]

number

[0075]

number

[0076] Next, the image processing unit 111 performs rotational coordinate transformation shown in equation (7) using the tilt angle (α, β) to transform the global coordinate system (x1, y1, z1) into the camera coordinate system (x0, y0, z0). In other words, equation (6) gives the definition of the tilt angle (α, β).

[0077]

number

[0078] This means that the global coordinate system is first rotated by α around the z-axis and then rotated by β around the x-axis to become the camera coordinate system. Finally, the image processor 111 converts the three-dimensional orthogonal coordinates (x0, y0, z0) of the camera coordinate system back to spherical coordinates (θ0, φ0) using equations (8) and (9).

[0079]

number

[0080]

number

[0081] In the above description, the coordinate conversion is performed by executing the vertical correction calculation, but this is not limited to this example. For example, a plurality of conversion tables corresponding to the tilt angles (α, β) may be created and stored in advance. This makes it possible to omit the vertical correction calculation process and to speed up the process.

[0082] [Outline of input processing according to the first embodiment] Next, the annotation input process according to the first embodiment will be described in more detail. Fig. 13 is a diagram showing an example of the configuration of an information processing system applicable to the first embodiment. In Fig. 13(a), the information processing system includes an imaging device 1a, an information processing device 100a connected to the imaging device 1a by wired or wireless communication, and a server 6 connected to the information processing device 100a via a network 5.

[0083] Each hemispherical image captured by each imaging lens 20a and 20b in the imaging device 1a is transmitted to the information processing device 100a. The information processing device 100a converts each hemispherical image transmitted from the imaging device 1a into a celestial sphere image as described above. Furthermore, the information processing device 100a adds annotations to the converted celestial sphere image in response to a user input. For example, the information processing device 100a transmits the celestial sphere image and the annotation to the server 6 via the network 5. The server 6 stores and manages the celestial sphere image and the annotation transmitted from the information processing device 100a.

[0084] Furthermore, the information processing device 100a acquires a celestial sphere image and annotations from the server 6, and creates report data based on the acquired celestial sphere image and annotations. Without being limited to this, the information processing device 100a can also create report data based on the celestial sphere image and annotations stored in the information processing device 100a. The information processing device 100a transmits the created report data to the server 6.

[0085] 13(a) and 13(b), the server 6 is shown as being configured by a single computer, but this is not limited to this example. That is, the server 6 may be configured to realize its functions by distributing them among a plurality of computers. The server 6 may also be configured as a cloud system on the network 5.

[0086] 13(b), a configuration in which the imaging device 1a and the information processing device 100a are each connected to a network 5 is also possible. In this case, each hemispherical image captured by the imaging device 1a is transmitted to the server 6 via the network 5. The server 6 performs conversion processing on each hemispherical image as described above, converts each hemispherical image into a celestial sphere image, and stores the converted image. The information processing device 100a adds annotations to the celestial sphere images stored in the server 6.

[0087] However, in the first embodiment, the information processing device 100a may not be connected to the server 6, and the information processing device 100a may perform the processing locally.

[0088] FIG. 14 is a diagram for explaining an example of imaging using the imaging device 1a according to the first embodiment. For example, inside a building 4 that is a diagnosis target, the imaging device 1a is used to capture an image of the celestial sphere. Imaging may be performed at multiple positions inside the building 4. Since the imaging range of the celestial sphere can be captured by one imaging operation, it is possible to suppress omissions in imaging, including the ceiling inside the building 4. In addition, in FIG. 14, imaging is performed inside the building 4 using the imaging device 1a, but this is not limited to this example, and the external surface of the building 4 can also be captured using the imaging device 1a. In this case, for example, by capturing images multiple times while moving around the entire circumference of the building 4, it is possible to capture images of the external surface (for example, wall surface) of the building 4 without omissions.

[0089] Annotation input processing in the information processing system according to the first embodiment will be briefly described with reference to Figs. 15-1 to 15-3. The information processing system according to the first embodiment provides several functions for assisting annotation input to a spherical image. Among the auxiliary functions provided, a function for automatically estimating an imaging position, a 3D panorama automatic tour function for realizing movement of a viewpoint between a plurality of spherical images, and an annotation input and confirmation function will be briefly described here.

[0090] Fig. 15-1 is a diagram for explaining the automatic estimation function of the imaging position according to the first embodiment. In Fig. 15-1, an image 5010 of a predetermined area of ​​a celestial sphere image based on each hemispherical image obtained by celestial sphere capturing, for example, the inside of a building 4 to be diagnosed is displayed on a screen 500 displayed on a display device 1010. Also displayed on the screen 500 is a floor plan image 5011 of the building 4 acquired in advance.

[0091] In the example of FIG. 15-1, pin markers 50401, 50402, 50403, . . . indicating the image capturing positions where images were captured are displayed on a floor plan image 5011.

[0092] For example, the imaging device 1a is used to capture images of the diagnosis target multiple times while moving the imaging position, and multiple sets of hemispherical images are acquired by the imaging lenses 20a and 20b. In the information processing device 100a, the image processing unit 111 generates multiple omnidirectional images based on each of the multiple sets of hemispherical images, and performs a matching process between each of the generated omnidirectional images by a known method. The image processing unit 111 estimates the relative imaging positions of each omnidirectional image based on the result of the matching process and each hemispherical image before conversion of each omnidirectional image. While maintaining the relative positional relationship between each estimated imaging position, each imaging position set is rotated and enlarged / reduced to correspond to the floor plan image 5011, thereby estimating each imaging position in the building 4.

[0093] Furthermore, in the information processing device 100a, the UI unit 113 displays a cursor 5041 that can be moved to any position in response to a user operation on the screen 500. The UI unit 113 displays, on the screen 500, an image 5010 based on a celestial sphere image that corresponds to a pin marker designated by the cursor 5041 among the pin markers 50401, 50402, 50403, .

[0094] In this way, by automatically estimating the imaging positions at which a plurality of omnidirectional images are captured, the user can easily grasp the relationship between the plurality of omnidirectional images, thereby improving work efficiency.

[0095] Note that the UI unit 113 can display a tag at a position designated by the user on the image 5010. In the example of Fig. 15-1, tags 5030a and 5030b are displayed for the image 5010. Although details will be described later, the tags 5030a and 5030b indicate the position of an object to which an annotation is input, and are associated with the coordinates of the omnidirectional image including the image 5010.

[0096] FIG. 15-2 is a diagram for explaining the 3D panorama automatic tour function according to the first embodiment. For example, referring to FIG. 15-2(a), when the pin marker 5040 in the floor plan image 5011 is 10 is specified, pin marker 5040 10 It is assumed that an image 5010a (see FIG. 15-2(b)) corresponding to the position of is displayed on the screen 500. In this state, when, for example, a movement to the left diagonal front (indicated by an arrow in the figure) is instructed by the cursor 541, the UI unit 133 instructs the specification to the pin marker by the pin marker 5040. 10 From the pin marker 5040 10 The pin marker 5040 is displayed diagonally to the right of the 11 and move the pin marker 5040 11 In the same manner, the UI unit 113 displays an image 5010b based on the spherical image corresponding to the pin marker 5040 on the screen 500. 10 From the pin marker 5040 10 The pin marker 5040 is displayed diagonally to the right of the 12 When the pin marker 5040 is moved to the 12 An image 5010c based on the omnidirectional image corresponding to the image 5010 is displayed on the screen 500.

[0097] In this way, by specifying a movement within screen 500, the specified pin marker moves to an adjacent pin marker in the direction of movement, for example, pin markers 50401, 50402, 50403, ..., and accordingly, image 5010 displayed on screen 500 switches. This allows the user to operate as if he or she were observing the diagnosis target in sequence, making it possible to improve the efficiency of the diagnosis work.

[0098] FIG. 15-3 is a diagram for explaining an annotation input and confirmation function according to the first embodiment. In the UI unit 113, a desired position of an image 5010 is specified on a screen 500 using a cursor 5041, and a tag 5030a indicating the position is displayed superimposed on the image 5010. In addition, an annotation input screen 600 for inputting an annotation corresponding to the position of the tag 5030a is displayed on the screen 500 together with the image 5010. That is, the annotation input screen 600 is a diagnostic information input screen for inputting diagnostic information related to a diagnostic target. Note that the annotation input screen 600 is highlighted in FIG. 15-3 for the purpose of explanation.

[0099] Although details will be described later, this annotation input screen 600 allows input to preset items and input of arbitrary comments. Also, the annotation input screen 600 allows editing of information that has already been input. Furthermore, the annotation input screen 600 allows input, as an annotation, of an image in a specified range including the position indicated by the tag 5030a in the image 5010, or an image acquired from outside.

[0100] In this way, a position is specified on the spherical image, and an annotation is input for the specified position, so that it is easy to associate the annotation with the image, and work efficiency can be improved.

[0101] [Details of input processing according to the first embodiment] Next, the annotation input process according to the first embodiment will be described in more detail. Fig. 16 is a flowchart of an example showing the annotation input process according to the first embodiment.

[0102] 16, the information processing device 100a acquires each hemispherical image of the diagnosis target captured by the imaging lenses 20a and 20b in the imaging device 1a and an output value of the acceleration sensor 2008, generates a celestial sphere image based on each hemispherical image, and stores the image in, for example, the storage 1004. In the following, in order to avoid complexity, a description will be given assuming one celestial sphere image.

[0103] In step S100, in the information processing device 100a, the image acquisition unit 110 acquires a celestial sphere image stored in the storage 1004. In the next step S101, the UI unit 113 cuts out an image of a predetermined region from the celestial sphere image acquired by the image acquisition unit 110 in step S100, and displays the image on the screen 500.

[0104] Fig. 17 is a diagram for explaining an image cropping process by the UI unit 113 applicable to the first embodiment. Note that Fig. 17 corresponds to Fig. 10(a) described above. The UI unit 113 crops an image of a part of an area 511 of a celestial sphere image 510 generated from two hemispherical images, and displays the image as an image 5010 on the screen 500. In step S101, the image of the area 511 that is predetermined as an initial value is cropped. Here, the UI unit 113 replaces, for example, the angular coordinates (φ, θ) of the celestial sphere image 510 with coordinates (x, y) suitable for display on the screen 500, and displays the image 5010.

[0105] In the next step S102, the UI unit 113 moves the region 511 within the spherical image 510 in response to a user operation. Furthermore, the UI unit 113 can change the size of the region 511 relative to the spherical image 510 in response to a user operation.

[0106] Fig. 18 is a diagram showing an example of a screen 500 displayed on the display device 1010 by the UI unit 113 according to the first embodiment. In this example, the UI unit 113 displays an image 5010 of the area 511 shown in Fig. 17 over the entire surface of the screen 500. On the screen 500, buttons 5020, 5021, 5022, and 5023, and menu buttons 5050 and 5051 are further displayed.

[0107] The button 5020 is a button for reading the spherical image 510. The button 5021 is a button for saving annotation data, which is set on the screen 500 and will be described later. The current screen 500 itself may be further saved in response to an operation of the button 5021. The button 5022 is a button for changing the scale, and an instruction screen for instructing enlargement and reduction of the image 5010 in the screen 500 in response to the operation is displayed. The button 523 is a button for displaying an annotation input screen 600, which will be described later, on the screen 500.

[0108] The menu button 5050 is a button for displaying a menu for switching the display mode of the screen 500. The menu button 5051 is a button for switching the viewpoint for the image 5010.

[0109] Also, on the screen 500, an area 5024 displays each imaging position estimated using the function of automatically estimating the imaging position described with reference to FIG. 15-1, in terms of relative position using pin markers 50411, 50412, . . .

[0110] Here, the information processing device 100a acquires in advance a floor plan image 5011 including a diagnosis target as exemplified in FIG. 19, and stores it in, for example, the storage 1004. The UI unit 113 can display this floor plan image 5011 on the screen 500. For example, the UI unit 113 can display the floor plan image 5011 by superimposing it on an area 5024 on the screen 500. The UI unit 113 can also display pin markers 50411, 50412, ... indicating imaging positions on the floor plan image 5011 displayed on the screen 500. The user can adjust the positions of the pin markers 50411, 50412, ... on the floor plan image 5011 by a predetermined input operation so as to match the positions where imaging was actually performed while maintaining the relative positional relationship between the pin markers 50411, 50412, ....

[0111] In the next step S103, the UI unit 113 waits for an operation on the button 5023, and when the button 5023 is operated, it displays the annotation input screen 600 on the screen 500. Then, in response to a predetermined operation on the annotation input screen 600, a position on the image 5010 is specified.

[0112] FIG. 20 is a diagram showing an example in which an annotation input screen 600 according to the first embodiment is displayed on a screen 500. In FIG. 20, the same reference numerals are used for parts common to those in FIG. 19 described above, and detailed description thereof will be omitted. In FIG. 20, a marker 602 is displayed superimposed on the position specified in step S103. In FIG. 20, the annotation input screen 600 is displayed superimposed on an image 5010. The UI unit 113 associates position information indicating the specified position with identification information (called an annotation ID) for identifying the annotation, and stores the information in, for example, the RAM 1002. Here, for example, the coordinates in the omnidirectional image 510 including the image 5010 are used as the position information.

[0113] Note that the annotation input screen 600 only needs to be displayed on the same screen 500 as the image 5010, and is not limited to the example of being displayed superimposed on the image 5010. For example, the annotation input screen 600 and the image 5010 may be displayed in different areas within the screen 500, or the annotation input screen 600 may be displayed in a window separate from the image 5010. Furthermore, the display position of the annotation input screen 600 on the screen 500 can be changed in response to a user operation.

[0114] 20, the annotation input screen 600 is provided with tabs 601a, 601b, and 601c for switching functions in the annotation input screen 600. The tab 601a is a tab for editing annotations. The tab 601b is a tab for inserting detailed images, and in response to an operation, for example, a file selection screen provided by the OS (Operating System) of the information processing device 100a is displayed, allowing a desired image to be selected. The tab 601c is a tab for selecting a cutout area, and in response to an operation, the display of the screen 500 is switched to a display for specifying an area for the image 5010.

[0115] 20, a button 6011 is a button for adding a tag to an image 5010. When the button 6011 is operated, for example, a position designation operation on the image 5010 is performed with a cursor 5041, the UI unit 113 displays a marker 602 at the designated position.

[0116] In the next step S104, the UI unit 113 waits for an operation on each of the tabs 601a to 601c on the annotation input screen 600 and the button 5021 on the screen 500, and when an operation is performed on any of them, executes the corresponding process.

[0117] When annotation input screen 600 is displayed (initial screen) in response to the operation of button 5023, or when tab 601a is specified, UI unit 113 determines that annotation input is to be performed, and moves the process to step S110. In the next step S111, UI unit 113 displays annotation input screen 600 including annotation input area 6010 for inputting annotation, as shown in FIG.

[0118] In the example of Fig. 20, the annotation input area 6010 includes input areas 6010a to 6010d. In this example, the input area 6010a is an input area for selecting a predetermined item using a pull-down menu, and each type included in the predetermined item of "equipment type" can be selected. The input area 6010b is an input area for inputting text information. This input area 6010b is used to input text information with a certain number of characters, such as a product model number.

[0119] The input area 6010c is an input area in which predetermined items are listed and a check mark is input into a check box of an appropriate item. A plurality of items can be specified in the input area 6010c. Also, the input area 6010c displays items according to the content selected in the input area 6010a, for example. In this example, the input area 6010c displays items according to "Evacuation Light" selected in "Facility Type" in the input area 6010a.

[0120] The input area 6010d is an input area for inputting, for example, remarks, and allows input of text information in a free format. The number of characters is unlimited, or a certain length of text can be input. The UI unit 113 stores the content input in the annotation input area 6010 in the RAM 1002 in association with position information indicating the position specified in step S103.

[0121] Note that even during display of annotation input screen 600 in step S111, it is possible to operate buttons 5020 to 5023, tabs 601a to 601c, and menu buttons 5050 and 5051 arranged on screen 500. In addition, it is also possible to perform an operation of changing the position within spherical image 510 of area 511 into which image 5010 is cut out.

[0122] Furthermore, while annotation input screen 600 is being displayed in step S111, an operation for designating the next position on image 5010 is also possible. In step S150, UI unit 113 determines whether or not the next position on image 5010 has been designated. If UI unit 113 determines that the next position has been designated (step S150, "Yes"), it returns the process to step S102.

[0123] On the other hand, when the UI unit 113 determines that the next position is not specified (step S150, "No"), the process returns to step S104. In this case, the state immediately before the determination in step S150 is maintained.

[0124] When the tab 601b is specified in step S104, the UI unit 113 determines that detailed image insertion is to be performed, and moves the process to step S120. In the next step S121, the UI unit 113 displays an image file selection screen on the screen 500. For example, the UI unit 113 can use a file selection screen provided as a standard function by the OS (Operating System) mounted on the information processing device 100a as the image file selection screen. At this time, it is preferable to filter file names based on, for example, the file extension so that only the file names of corresponding image files are displayed.

[0125] In the next step S122, the UI unit 113 reads the image file selected on the image file selection screen. In the next step S123, the UI unit 113 displays the image of the image file read in step S122 in a predetermined area of ​​the annotation input screen 600. The UI unit 113 associates the file name, including the path, of the read image file with the location information indicating the location specified in step S103, and stores the file name in, for example, the RAM 1002. After the process of step S123, the process proceeds to the above-mentioned step S150.

[0126] When the tab 601c is selected in step S104, the UI unit 113 determines that a cutout region is to be selected for the image 5010, and moves the process to step S130. In the next step S131, the UI unit 113 switches the display of the annotation input screen 600 to a display of a region designation screen for designating a cutout region as a diagnostic region including an image to be diagnosed. Fig. 21 is a diagram showing an example of the annotation input screen 600 switched to the region designation screen, which is applicable to the first embodiment.

[0127] 21, the annotation input screen 600 switched to the area designation screen includes a cut image display area 6020, a check box 6021, and a selection button 6022. The cut image display area 6020 displays a cut image cut out from the area designated for the image 5010. In an initial state where no area is designated for the image 5010, the cut image display area 6020 displays, for example, blank. However, the present invention is not limited to this, and an image of an area previously set for the position designated in step S103 may be cut out from the image 5010 and displayed in the cut image display area 6020.

[0128] The check box 6021 is a button for displaying a marker image as an icon in the cut image display area 6020. The selection button 6022 is a button for starting to specify a cut area for the image 5010.

[0129] Fig. 22 is a diagram showing an example of display of a screen 500 for specifying a cutout area in response to an operation on a selection button 6022 according to the first embodiment. As shown in Fig. 22, the UI unit 113 displays a frame 603 indicating the cutout area on the screen 500 in response to an operation on the selection button 6022, and erases the display of the annotation input screen 600 from the screen 500. The frame 603 is displayed, for example, corresponding to an area that is set in advance for the position specified in step S103.

[0130] The UI unit 113 can change the size, shape, and position of the frame 603 in response to a user operation. The shape is limited to, for example, a rectangle, but the ratio of the lengths of the long and short sides can be changed. When the size, shape, and position of the frame 603 is changed, if the changed frame 603 does not include the position specified in step S103, the UI unit 113 can display a warning screen indicating that fact.

[0131] Returning to the explanation of the flowchart in Fig. 16, the UI unit 113 determines whether or not the designation of the area is completed in step S132. For example, when the display device 1010 and the input device 1011 are configured as a touch panel 1020, and the designation of the area using the frame 603 is performed by a drag operation of moving a user's finger while the finger is in contact with the touch panel 1020, it can be determined that the drag operation has ended and the designation of the area using the frame 603 has been completed when the finger is released from the touch panel 1020.

[0132] When the UI unit 113 determines that the area designation is not complete (step S132, “No”), the process returns to step S132. On the other hand, when the UI unit 113 determines that the area designation is complete (step S132, “Yes”), the process proceeds to step S133.

[0133] In step S133, the UI unit 113 acquires, as a cut-out image, an image of the area specified by the frame 603 from the image 5010. At this time, the UI unit 113 acquires coordinates in the spherical image 510 including the image 5010, for example, of each vertex of the frame 603, and stores each acquired coordinate in, for example, the RAM 1002 in association with position information indicating the position specified in step S103.

[0134] In the next step S134, the UI unit 113 displays the cut image acquired in step S133 in the cut image display area 6020 in the annotation input screen 600. The UI unit 113 also stores the cut image acquired in step S133 in a file with a predetermined file name to create a cut image file. The UI unit 113 stores the cut image file in, for example, the RAM 1002.

[0135] In the next step S135, the UI unit 113 determines whether or not to display an icon for the cut image displayed in the cut image display area 6020. If the check box 6021 on the annotation input screen 600 is checked, the UI unit 113 determines that an icon is to be displayed (step S135, "Yes"), and moves the process to step S136.

[0136] In step S136, UI unit 113 displays marker 602' at a position corresponding to the position specified in step S103 in the cut-out image displayed in cut-out image display area 6020, as exemplified in Fig. 23. In this example, marker 602' has the same shape as marker 602 displayed on image 5010, but this is not limited to this example, and marker 602' and marker 602 may have different shapes.

[0137] After the process of step S136, the process proceeds to the above-mentioned step S150. Also, if it is determined in step S135 that the icon is not to be displayed (step S135, “No”), the process proceeds to step S150.

[0138] When, for example, button 5021 is operated in step S104, UI unit 113 determines that the input contents on annotation input screen 600 are to be saved, and shifts the process to step S140. In step S140, UI unit 113 instructs supplementary information generation unit 112 to save each piece of data input on annotation input screen 600.

[0139] In the next step S141, the incidental information generating unit 112, in accordance with an instruction from the UI unit 113, creates annotation data based on the data stored in the RAM 1002 by the UI unit 113 in each of the above-mentioned processes (e.g., step S103, step S111, step S123, step S133, step S134, etc.). The annotation data is created for each position specified in step S103. The incidental information generating unit 112 stores the created annotation data in, for example, the storage 1004 of the information processing device 100a. Alternatively, the incidental information generating unit 112 may transmit the created annotation data to the server 6 via the network 5 and have the server 6 store it therein.

[0140] Tables 2 to 7 show examples of the configuration of annotation data according to the first embodiment.

[0141] [Table 2]

[0142] [Table 3]

[0143] [Table 4]

[0144] [Table 5]

[0145] [Table 6]

[0146] [Table 7]

[0147] Table 2 shows the overall structure of annotation data. The annotation data includes shape data, attribute data, image region data, and annotation data ID. Examples of the structure of the shape data, attribute data, and image region data are shown in Tables 3, 4, and 5, respectively.

[0148] An example of shape data is shown in Table 3. The shape data indicates the shape formed by the positions specified by the user in step S103 of Fig. 16, and includes the items "primitive type", "number of vertices", and "list of vertex coordinates".

[0149] Table 6 shows examples of values ​​defined in the item "primitive type". In this example, the values ​​"point", "line" and "polygon" are defined for the item "primitive type". When the value of the item "primitive type" is "point", the value of the item "number of vertices" is set to "1". When the value of the item "primitive type" is "line", the value of the item "number of vertices" is set to "2 or more". When the value of the item "primitive type" is "polygon", the value of the item "number of vertices" is set to "3 or more". The item "vertex coordinate list" defines each vertex shown in the item "number of vertices" in the (φ, θ) coordinate system. Here, the closed area enclosed by the straight lines connecting the coordinates listed in the item "vertex coordinate list" in order from the top becomes the range of the "polygon" defined by the primitive type.

[0150] In the above example, since one point is specified in step S103 in Fig. 16, the item "primitive type" is set to the value "point" and the number of vertices is set to "1." Also, the item "vertex coordinate list" describes only the coordinates of the one specified point.

[0151] Table 4 shows an example of attribute data. The attribute data includes the following items: "creator," "creation date and time," "location," "subject," "source image," "investigation date and time," "response method," "attribute information list," and "tag."

[0152] The value of the item "Creator" is obtained, for example, from login information for the information processing device 100a or an input program for realizing the functions of the information processing device 100a according to the first embodiment. The value of the item "Creation Date and Time" is obtained from the system time based on the clock of the information processing device 100a. The value of the item "Location" is obtained from the imaging position on the floor plan image 5011 if the floor plan image 5011 can be obtained. If the floor plan image 5011 cannot be obtained, the value of the item "Location" is obtained from user input.

[0153] The value of the item "Target" is obtained from user input. The value of the item "Source Image" is the image name (file name) referenced when the annotation was created on the annotation input screen 600. For example, the file name of the cut image file displayed in the above-mentioned cut image display area 6020 is used as the value of the item "Source Image". The value of the item "Investigation Date and Time" is obtained from the timestamp of the image file described in the item "Source Image". The value of the item "Response Method" is obtained from user input.

[0154] The value of the item "attribute information list" is described as a list of attribute information exemplified in Table 7. In Table 7, the attribute information includes the items "type", "name" and "attribute value list". The value of the item "type" is the type of the diagnosis target such as equipment or abnormality, and for example, the value of the input field 6010a in the annotation input screen 600 shown in FIG. 20 for inputting the equipment type is used. The value of the item "name" is the specific name of the diagnosis target, and for example, the value of the input field 6010b in the annotation input screen 600 shown in FIG. 20 for inputting the product model number is used. The value of the item "attribute value list" is, for example, the name corresponding to each check box in the input field 6010c in the annotation input screen 600 shown in FIG. 20 as the attribute name, and the value of the check box as the attribute value, and a list of pairs of these attribute names and attribute values ​​is obtained. Each attribute name included in the item "attribute value list" is different depending on the value of the item "type".

[0155] Returning to the explanation of Table 4, the value of the item "tag" is acquired from a user input. For example, the value (for example, a note) input in the input area 6010d on the annotation input screen 600 is used as the value of the item "tag".

[0156] Table 5 shows an example of image region data. The image region data indicates, for example, the coordinates of the top left, bottom left, top right, and bottom right vertices of the cropped image specified in step S132 of FIG. 16 by angular coordinates (φ, θ) in the spherical image 510.

[0157] Returning to the description of Fig. 16, when the storage of the annotation data in step S141 is completed, the auxiliary information generating unit 112 shifts the process to step S142. Here, the auxiliary information generating unit 112 associates, with the annotation data, information indicating each of the omnidirectional images 510 acquired at the time of annotation input (for example, a file name including a path). Taking Fig. 15-1 as an example, the auxiliary information generating unit 112 associates, with the annotation data, information indicating each of the omnidirectional images 510, 510, ... corresponding to each of the pin markers 50401, 50402, 50403, ....

[0158] In step S142, the UI unit 113 determines whether or not an instruction has been given to end the annotation input process by the input program. For example, after the save process in step S141, the UI unit 113 displays an end instruction screen for instructing the user to end or continue the annotation input process. If the UI unit 113 determines that an instruction has been given to end the annotation input process (step S142, "Yes"), it ends the series of processes according to the flowchart in Fig. 16. On the other hand, if the UI unit 113 determines that an instruction has been given to continue the annotation input process (step S142, "No"), it transitions the process to the above-mentioned step S150.

[0159] [Output processing according to the first embodiment] Next, a description will be given of the output process according to the first embodiment. In the information processing device 100a, the output unit 115 creates report data that summarizes the diagnosis results of the diagnosis target, based on the annotation data created as described above.

[0160] As an example, in the information processing device 100a, the UI unit 113 reads each piece of annotation data and each omnidirectional image 510, 510, ... associated with each piece of annotation data in response to, for example, an operation of a button 5020 on a screen 500. The UI unit 113 cuts out a predetermined area 511 of one omnidirectional image 510 from among the read omnidirectional images 510, 510, ..., and displays it on the screen 500 as an image 5010. At this time, the UI unit 113 displays, for example, a tag corresponding to each piece of annotation data at a position on the image 5010 according to the value of the item "list of vertex coordinates" in the shape data of each piece of annotation data.

[0161] Fig. 24 is a diagram showing an example of display of a screen 500 related to report data creation according to the first embodiment. In Fig. 24, for example, tags 5030c and 5030d are displayed on the screen 500 according to the item "tag" included in the attribute data of each annotation data. Also, corresponding to these tags 5030c and 5030d, values ​​(e.g., remarks) inputted to the item "tag" are displayed as comments 604a and 604b. The output unit 115 creates report data based on annotation data corresponding to the specified tags in response to the specifications for these tags 5030c and 5030d.

[0162] Without being limited to this, the output unit 115 can also perform a narrowed-down search for each item of attribute data of each annotation data using conditions specified by the user, and create report data based on each annotation data obtained as a result of the search.

[0163] 25 is a diagram showing an example of report data applicable to the first embodiment. In this example, the report data includes, for each annotation data, a record including the items of "inspection date" and "inspector name", and the items of "room name", "target", "site photo", "content", "necessity of action", and "remarks".

[0164] In the report data record, the items "Inspection date" and "Name of inspector" can be obtained from the items "Creation date and time" and "Creator" in the attribute data of the annotation data, for example, by referring to Table 4 above. The item "Subject" in the report data can be obtained from the item "Name" in the item "Attribute information list" in the attribute data of the annotation data, for example, by referring to Tables 4 and 7. Furthermore, the items "Room name," "Content," and "Response required" in the report data can be obtained from the items "Location," "Tag," and "Response method" in the attribute data of the annotation data, respectively. Furthermore, the item "Current status photo" in the report data is embedded with an image obtained based on the image name described in the item "Source image" in the attribute data of the annotation data, for example, by referring to Table 4.

[0165] In addition, in the report data, the item "Notes" can be obtained, for example, according to a user input when the report data is created.

[0166] 25 is an example, and each item included in the record is not limited to the above example. Each item included in the record can be changed or set according to user input.

[0167] The output unit 115 outputs the report data created in this way in a predetermined data format. For example, the output unit 115 can output the report data according to a data format of a commercially available document creation application program, a table creation application program, a presentation material creation application program, etc. The output unit 115 can also output the report data according to a data format specialized for printing and display, such as PDF (Portable Document Format).

[0168] Note that the report data shown in FIG. 25 is not limited to being output by the output unit 115. For example, an image embedded in the item "site photo" can be associated with link information for calling up a panoramic automatic tour function (see FIG. 15-2) that starts from an image capturing position corresponding to the image. The UI unit 113 displays an image representing the report data shown in FIG. 25 on the screen 500. When an image to be displayed in the item "site photo" is specified by a user operation, the UI unit 113 switches the display of the screen 500 to, for example, the display shown in FIG. 15-2(a) in accordance with the link information associated with the image.

[0169] In addition, it is also possible to associate link information to a specific website with the image embedded in the "site photo" item.

[0170] Also, in the report data, an item (hereinafter, "time-series photo item") in which a time-series image is embedded instead of the item "current state photo" can be provided. Note that the time-series image, for example, an image in which a plurality of images taken at different times in a certain position and imaging range are arranged in order of the time of taking, is taken in advance and stored in, for example, the information processing device 100a in association with location information indicating the position. When displaying a screen based on the report data, the output unit 115 displays a time-series image including a plurality of images for the item "time-series photo". This makes it possible to easily grasp the progress of work such as construction work. Also, by comparing images taken at different times, it is possible to easily check changes over time.

[0171] 24, annotation data used to create report data is specified by specifying each tag 5030c and 5030d displayed on the screen 500, but this is not limited to this example. For example, it is also possible to create and display a list including thumbnail images, comments, attribute information, etc. for each annotation data. The thumbnail images can be used by reducing the cut images.

[0172] Furthermore, comments by people other than the report creator (inspector) can be associated with report data. This allows for a thread function, for example, where a designer responds to the inspector's report data, and the inspector then responds with a comment.

[0173] [Modification of the first embodiment] Next, a modified example of the first embodiment will be described. In the above-mentioned first embodiment, as explained using Fig. 20 to Fig. 23, etc., the position is specified by a point in step S103 of Fig. 16. This is not limited to this example, and for example, as explained in the item "Primitive type" of Tables 3 and 6, the position can also be specified by a line or a polygon.

[0174] An example of specifying a position by a line in step S103 in Fig. 16 will be described below. Fig. 26 is a diagram showing an example in which a crack 610 is observed on a wall surface, which is applicable to a modified example of the first embodiment. Note that in Fig. 26 and Figs. 27 to 29 described below, parts corresponding to Figs. 20 to 23 above are given the same reference numerals, and detailed description thereof will be omitted.

[0175] 26, a crack 610 having a linear shape is observed in an image 5010 displayed on a screen 500. In step S103 in FIG. 16, the user specifies the crack 610 by, for example, tracing (dragging) the image of the crack 610 on the image 5010 on the touch panel 1020. The position information representing the crack 610 is, for example, a collection of position information of a plurality of points where the positions of successively adjacent points are smaller than a threshold value.

[0176] Fig. 27 corresponds to Fig. 20 described above, and is a diagram showing an example of an annotation input screen 600 for inputting an annotation for a specified crack 610 according to a modified example of the first embodiment. In Fig. 27, the annotation input screen 600 includes an annotation input area 6010' for inputting an annotation. In this example, the annotation input area 6010' includes input areas 6010d and 6010e.

[0177] The input area 6010e is an area for inputting information about the state of deformation of the crack 610, etc., and includes the input items of "location," "type of deformation," "width," "length," and "status." Of these, the items "type of deformation" and "status" are input areas for selecting a predetermined item from a pull-down menu. Furthermore, the user inputs names and values ​​for the items "location," and the items "width," and "length." The items "width" and "length" are not fixed items, but can be items according to the content selected in the item "type of deformation."

[0178] 27, a button 6012 is a button for adding a deformation type. In response to an operation on the button 6012, the UI unit 113 adds and displays, for example, the above-mentioned item "deformation type", the items "width" and "length", and the item "status" to the annotation input screen 600.

[0179] The designation of the cutout area for this linearly designated position can be performed in the same manner as the method described in the first embodiment with reference to Figures 21 and 22. Figure 28 is a diagram showing an example of an annotation input screen 600 switched to an area designation screen that can be applied to the modified example of the first embodiment. In this example, a crack 610' corresponding to the crack 610 on the image 5010 is displayed in the cutout image display area 6020. There are no other differences from Figure 21 described above.

[0180] 29 is a diagram showing an example of the display of a screen 500 for specifying a cutout area in response to an operation on a selection button 6022, according to a modified example of the first embodiment. In this case as well, a frame 603 is specified so as to include the crack 610. If a part or all of the crack 610 is not included in the frame 603, the UI unit 113 can display a warning screen indicating that fact.

[0181] Similarly, it is also possible to specify the position and cutout area for polygons with more than one side (including concave polygons) in a triangle.

[0182] [Second embodiment] Next, a second embodiment will be described. In the above-described first embodiment, annotation data is created based on a spherical image 510 having two-dimensional coordinate information. In contrast, in the second embodiment, a three-dimensional image having three-dimensional coordinate information is further used when creating annotation data.

[0183] [Imaging device according to the second embodiment] Fig. 30 is a diagram showing an image pickup device according to the second embodiment. In Fig. 30, parts common to Fig. 1 described above are given the same reference numerals, and detailed description is omitted. In Fig. 30, an image pickup device 1b is provided with a plurality of (five in this example) image pickup lenses 20a1, 20a2, 20a3, 20a4, and 20a5, and a shutter button 30 on a first surface of a substantially rectangular parallelepiped housing 10b. Within the housing 10b, image pickup elements are provided corresponding to the image pickup lenses 20a1, 20a2, ..., 20a5, respectively.

[0184] Furthermore, a plurality of imaging lenses 20b1, 20b2, 20b3, 20b4, and 20b5 are provided on a second surface of the housing 10b on the rear side of the first surface. Similar to the above-mentioned imaging lenses 20a1, 20a2, ..., 20a5, imaging elements corresponding to the imaging lenses 20b1, 20b2, ..., 20b4, and 20b5 are provided within the housing 10b.

[0185] In each of the imaging lenses 20a1, 20a2, ..., 20a5 and the imaging lenses 20b1, 20b2, ..., 20b5, a pair of imaging lenses having the same height from the bottom surface of the housing 10b constitutes imaging bodies 211, 212, 213, 214 and 215, respectively.

[0186] Since the same configuration as the imaging lenses 20a and 20b described in the first embodiment can be applied to the imaging lenses 20a1, 20a2, ..., 20a5 and the imaging lenses 20b1, 20b2, ..., 20b5, detailed description thereof will be omitted here. Also, the imaging bodies 211, 212, 213, 214, and 215 correspond to the imaging body 21 described above.

[0187] In the second embodiment, the imaging lenses 20a1, 20a2, ..., 20a5 are disposed at equal intervals with the distance between each adjacent imaging lens being a distance d. Also, the imaging lenses 20a1 and 20b1, the imaging lenses 20a2 and 20b2, the imaging lenses 20a3 and 20b3, the imaging lenses 20a4 and 20b4, and the imaging lenses 20a5 and 20b5 are disposed in the housing 10b such that their heights from, for example, the bottom surface of the housing 10b are the same.

[0188] Furthermore, the imaging lenses 20a5 and 50b5 of the imaging body 215 arranged at the bottom among the imaging bodies 211, 212, 213, 214, and 215 are arranged at a height h from the bottom surface. For example, the imaging lenses 20a1 to 20a5 are arranged such that, starting from the bottommost imaging lens 20a5, the imaging lenses 20a4, 20a3, 20a2, and 20a1 are spaced apart by a distance d from the bottom surface side to the top surface side of the housing 10a, with the lens centers aligned with the center line in the long side direction of the housing 10a.

[0189] The shutter button 30 is a button for instructing imaging by each of the imaging lenses 20a1, 20a2, ..., 20a5 and each of the imaging lenses 20b1, 20b2, ..., 20b5 in response to an operation. When the shutter button 30 is operated, imaging is synchronously performed by each of the imaging lenses 20a1, 20a2, ..., 20a5 and each of the imaging lenses 20b1, 20b2, ..., 20b5.

[0190] 30, the housing 10b of the imaging device 1b includes an imaging section area 2b in which the imaging bodies 211-215 are arranged, and an operation section area 3b in which the shutter button 30 is arranged. The operation section area 3b is provided with a grip section 31 for a user to hold the imaging device 1b, and a fixing section 32 for fixing the imaging device 1b to a tripod or the like is provided on the bottom surface of the grip section 31.

[0191] In addition, although the imaging device 1b has been described here as including five imaging bodies 211 to 215, this is not limited to this example. In other words, the imaging device 1b only needs to include a plurality of imaging bodies 21, and may include six or more imaging bodies 21, or may include two or more and four or less imaging bodies 21.

[0192] Fig. 31 is a diagram showing an example of an imaging range that can be imaged by each of the imaging bodies 211, 212, ..., 215 applicable to the second embodiment. Each of the imaging bodies 211, 212, ..., 215 has a similar imaging range. In Fig. 31, the imaging ranges of the imaging bodies 211, 212, ..., 215 are shown as a representative by the imaging range of the imaging body 211.

[0193] 31, the Z axis is defined as the direction in which imaging lenses 20a1, 20a2, ..., 20a5 are aligned, and the X axis is defined as the direction of the optical axes of imaging lenses 20a1 and 20b1. Also, the Y axis is defined as a direction that is included in a plane that intersects with the Z axis at a right angle and that intersects with the X axis at a right angle.

[0194] The imaging body 211, by combining the imaging lenses 20a1 and 20b1, has an imaging range of the entire celestial sphere centered on the center of the imaging body 211. That is, as described above, the imaging lenses 20a1 and 20b1 each have an angle of view of 180° or more, preferably greater than 180°, and more preferably 185° or more. Therefore, by combining the imaging lenses 20a1 and 20b1, the imaging range A on the XY plane and the imaging range B on the XZ plane can each be 360°, and by combining these, an imaging range of the entire celestial sphere is realized.

[0195] Moreover, the imaging bodies 211, 212, ..., 215 are arranged at intervals of a distance d in the Z-axis direction. Therefore, each set of hemispherical images captured by each of the imaging bodies 211, 212, ..., 215 with the entire celestial sphere as an imaging range has a viewpoint that differs by a distance d in the Z-axis direction.

[0196] At this time, in the embodiment, imaging at each of the imaging lenses 20a1-20a5 and each of the imaging lenses 20b1-20b5 is performed synchronously in response to the operation of the shutter button 30. Therefore, by using the imaging device 1b according to the second embodiment, it is possible to obtain five sets of hemispherical images captured at the same timing with different viewpoints at a distance d in the Z-axis direction for each of the first and second surfaces of the housing 10b.

[0197] The five celestial sphere images generated from the five sets of hemispherical images captured at the same timing with different viewpoints at a distance d in the Z-axis direction are images aligned on the same epipolar line extending in the Z-axis direction.

[0198] [Overview of image processing according to the second embodiment] Next, image processing according to the second embodiment will be described. Fig. 32 is a functional block diagram of an example for describing the function of an information processing device 100b as an input device for inputting annotations according to the second embodiment. Note that the configuration of the information processing device 100b as hardware can be applied to the configuration of the information processing device 100a described using Fig. 6 as it is, so detailed description here will be omitted. Also, in Fig. 32, parts common to those in Fig. 7 described above are given the same reference numerals, and detailed description will be omitted.

[0199] 32, like the above-described information processing device 100a, the information processing device 100b includes an image acquisition unit 110, an image processing unit 111, an auxiliary information generation unit 112, a UI unit 113, a communication unit 114, and an output unit 115. Further, the information processing device 100b includes a 3D information generation unit 120.

[0200] Among these, the image acquiring unit 110 acquires each hemispherical image captured by each of the imaging lenses 20a1-20a5 and each of the imaging lenses 20b1-20b5 of the imaging device 1b. The image processing unit 111 generates five omnidirectional images having different viewpoints in the Z-axis direction corresponding to each of the imaging bodies 211-215 by the processing described using Figs. 8-12 and formulas (1)-(9) from each hemispherical image acquired by the image acquiring unit 110.

[0201] In addition, since the imaging device 1b according to the second embodiment is assumed to be installed so that the center line connecting the centers of the imaging lenses 20a1-20a5 is parallel to the vertical direction and imaging is performed, the tilt correction process for each hemispherical image can be omitted. Without being limited to this, the imaging device 1b may be provided with the acceleration sensor 2008 as described above, the vertical direction may be detected based on the detection result of the acceleration sensor 2008, and the tilt of the imaging device 1b with respect to the vertical direction may be obtained to perform tilt correction.

[0202] The UI unit 113 displays each of the screens 500 described with reference to Fig. 18, Fig. 20 to Fig. 23, and Fig. 26 to Fig. 29 using one of the five celestial sphere images corresponding to each of the imaging bodies 211 to 215, for example, a celestial sphere image generated from a set of hemispherical images captured by the imaging body 211. The generation of annotation data by the auxiliary information generation unit 112 is also performed based on the celestial sphere image generated from the set of hemispherical images captured by the imaging body 211.

[0203] The 3D information generator 120 generates three-dimensional information using five omnidirectional images generated by the image processor 111, the five images having different viewpoints in the Z-axis direction at a distance d each.

[0204] [3D information generation process applicable to the second embodiment] Next, a three-dimensional information generation process that is executed by the 3D information generation unit 120 and that can be applied to the second embodiment will be described.

[0205] Fig. 33 is a diagram showing an example of an image captured from five different viewpoints by the imaging device 1b applicable to the second embodiment and synthesized for each of the imaging bodies 211, 212, ..., 215. Fig. 33(a) shows an example of a subject 60, and Figs. 33(b), 33(c), 33(d), 33(e), and 33(f) show examples of omnidirectional images 3001, 3002, 3003, 3004, and 3005 synthesized from captured images of the same subject 60 captured from five different viewpoints. As exemplified in Figs. 33(b) to 33(f), each of the omnidirectional images 3001 to 3005 includes an image of the subject 60 that is shifted slightly depending on the distance d between each of the imaging bodies 211 to 215.

[0206] In addition, in Fig. 33, imaging device 1b is shown capturing an image of subject 60 present on the first surface (front surface) side of imaging device 1b, but this is for the sake of explanation, and imaging device 1b can actually capture an image of subject 60 surrounding imaging device 1b. In this case, omnidirectional images 3001 to 3005 are images based on, for example, equirectangular projection, and the left and right sides each represent the same position, and the top and bottom sides each represent one point. That is, for the sake of explanation, omnidirectional images 3001 to 3005 shown in Fig. 33 are images that are converted and partially cut out based on equirectangular projection.

[0207] The projection method of the spherical images 3001 to 3005 is not limited to equirectangular projection. For example, the spherical images 3001 to 3005 may be images using cylindrical projection when it is not necessary to have a large angle of view in the Z-axis direction.

[0208] 34 is a flowchart of an example showing a process of creating a 3D reconstruction model applicable to the second embodiment. Each process of this flowchart is executed by the information processing device 100b. In addition, it is assumed that the imaging device 1b already has 10 hemispherical images captured by the imaging bodies 211 to 215 stored in its built-in memory.

[0209] In step S10, the image acquisition unit 110 acquires each hemispherical image captured by each of the imaging bodies 211 to 215 from the imaging device 1b. The image processing unit 111 combines each of the acquired hemispherical images for each of the imaging bodies 211 to 215 to generate five omnidirectional images 3001 to 3005 captured from multiple viewpoints as shown in FIG.

[0210] In the next step S11, the 3D information generation unit 120 selects one of the omnidirectional images 3001 to 3005 created in step S10 as a reference omnidirectional image (hereinafter referred to as omnidirectional image 3001). The 3D information generation unit 120 calculates the parallax of the other omnidirectional images 3002 to 3005 with respect to the selected reference omnidirectional image for all pixels of the images.

[0211] Here, the principle of the parallax calculation method applicable to the second embodiment will be described. The basic principle of performing parallax calculation using captured images captured by an image sensor such as the image sensor 200a is a method using triangulation. Triangulation will be described with reference to FIG. 35. In FIG. 35, the cameras 400a and 400b include lenses 401a and 401b and image sensors 402a and 402b, respectively. Using triangulation, a distance D from a line connecting the cameras 400a and 400b to the target 403 is calculated from imaging position information in the captured images captured by the image sensors 402a and 402b.

[0212] In FIG. 35, value f indicates the focal length of each of lenses 401a and 401b. The length of the line connecting the optical axis centers of lenses 401a and 401b is defined as base line length B. In the example of FIG. 30, distance d between each of imaging bodies 211 to 215 corresponds to base line length B. The difference between imaging positions i1 and i2 of object 403 on imaging elements 402a and 402b is parallax q. Since D:f=B:q holds due to the similarity relationship of triangles, distance D can be calculated by equation (10).

[0213]

number

[0214] In formula (10), since the focal length f and the base line length B are known, the task of the process is to calculate the parallax q. Since the parallax q is the difference between the imaging position i1 and the imaging position i2, the fundamental task of the parallax calculation is to detect the correspondence relationship between the imaging positions in the images captured by the imaging elements 402a and 402b. In general, this matching process for finding the corresponding positions between a plurality of images is realized by searching for each parallax on the epipolar line based on the epipolar constraint.

[0215] The disparity search process can be realized using various calculation methods. For example, a block matching process using a normalized cross-correlation coefficient (NCC) shown in formula (11) can be applied. In addition, a high-density disparity calculation process using semi-global matching (SGM) can also be applied. The method to be used for calculating the disparity can be appropriately selected depending on the application of the 3D reconstruction model to be finally generated. In formula (11), the value p represents the pixel position, and the value q represents the disparity.

[0216]

number

[0217] Based on these cost functions, the correspondence between each pixel on the epipolar line is calculated, and the calculation result that is considered to be the most similar is selected. In NCC, i.e., equation (11), the numerical value C(p,q) NCC In SGM, the pixel location with the minimum cost is regarded as the corresponding point.

[0218] As an example, a case where disparity is calculated using the NCC of formula (11) will be outlined. The block matching method obtains pixel values ​​of an area cut out as a block of M pixels x N pixels centered on an arbitrary reference pixel in a reference image, and pixel values ​​of an area cut out as a block of M pixels x N pixels centered on an arbitrary target pixel in a target image. Based on the obtained pixel values, the similarity between the area including the reference pixel and the area including the target pixel is calculated. The similarity is compared while moving the block of M pixels x N pixels in the image to be searched, and the target pixel in the block at the position where the similarity is highest is set as the corresponding pixel corresponding to the reference pixel.

[0219] In equation (11), the value I(i,j) represents the pixel value of a pixel in a pixel block in the reference image, and the value T(i,j) represents the pixel value of a pixel in a pixel block in the target image. The calculation of equation (11) is performed while moving the pixel block in the target image, which corresponds to the pixel block of M pixels by N pixels in the reference image, pixel by pixel, to obtain the numerical value C(p,q). NCC A search is made for the pixel position where

[0220] Even when the imaging device 1b according to the second embodiment is used, the parallax is basically calculated using the above-mentioned principle of triangulation. Here, the imaging device 1b includes five imaging bodies 211-215 and can capture five spherical images 3001-3005 at one time. That is, the imaging device 1b according to the second embodiment can capture three or more captured images simultaneously. Therefore, the above-mentioned principle of triangulation is applied in an expanded manner to the second embodiment.

[0221] For example, as shown in equation (12), the total sum of disparities q of the costs for the cameras separated by the baseline length B can be used as the cost to detect corresponding points in the images captured by the cameras.

[0222]

number

[0223] As an example, the first, second, and third cameras are arranged on the epipolar line in the order of the first camera, the second camera, and the third camera. In this case, cost calculation is performed by the above-mentioned NCC, SGM, or the like for each of the pair of the first camera and the second camera, the pair of the first camera and the third camera, and the pair of the second camera and the third camera. The final distance D to the target can be calculated by summing up the costs calculated by each of these pairs of cameras and finding the minimum value of the sum.

[0224] The matching process described here can also be applied to the matching process in the automatic imaging position estimation function described with reference to FIG. 15-1.

[0225] However, as a method for calculating parallax in the embodiment, a stereo image measuring method using EPI (Epipolar Plane Image) can be applied. For example, the omnidirectional images 3001-3005 based on the captured images captured by the imaging bodies 211-215 in the imaging device 1b can be regarded as the same images captured by a camera moving at a constant speed, and an EPI can be created. By using the EPI, it is possible to easily search for corresponding points between the omnidirectional images 3001-3005, as compared with, for example, the above-mentioned method using triangulation.

[0226] As an example, in the imaging device 1b, for example, the imaging body 211 is used as a reference, and the distances d between the imaging body 211 and each of the imaging bodies 212 to 215 are 1- 2, d 1-3 , d 1-4 and d 1-5 Based on the calculation result, the horizontal and vertical axes (x, y) of the spherical images 3001 to 3005 and the distance D (=0, distance d 1-2 ,…,d 1-5 A three-dimensional spatial image is created by taking the yD plane as the cross-sectional image of this spatial image.

[0227] In the EPI created as described above, points on an object present in each of the original spherical images 3001 to 3005 are represented as a single straight line. The inclination of this line changes depending on the distance from the image capture device 1b to the point on the object. Therefore, by detecting the line included in the EPI, it is possible to determine corresponding points between each of the spherical images 3001 to 3005. In addition, the distance from the image capture device 1b to the object corresponding to the line can be calculated based on the inclination of the line.

[0228] The principle of EPI will be described with reference to Fig. 36 and Fig. 37. Fig. 36(a) shows a set of a plurality of images 4201, 4202, ..., each of which is a cylindrical image. Fig. 36(b) shows an outline of an EPI 422 cut out from the set of images 4201, 4202, ..., by a plane 421. In the example of Fig. 36(a), the shooting position axis of each image 4201, 4202, ... is taken in the depth direction, and the set of images 4201, 4202, ... are superimposed and converted into three-dimensional data as shown in Fig. 36(a). The set of images 4201, 4202, ..., converted into three-dimensional data, is cut out by a plane 421 parallel to the depth direction, resulting in the EPI 422 shown in Fig. 36(b).

[0229] In other words, the EPI 422 is an image in which lines with the same X coordinate are extracted from each of the images 4201, 4202, . . . and the extracted lines are arranged side by side with the X coordinates of the images 4201, 4202, .

[0230] Fig. 37 is a diagram showing the principle of EPI applicable to the second embodiment. Fig. 37(a) is a schematic diagram of Fig. 36(b) described above. In Fig. 37(a), the horizontal axis u is the depth direction in which the images 4201, 4202, ... are superimposed, and represents parallax, and the vertical axis v corresponds to the vertical axis of each of the images 4201, 4202, .... EPI 422 means an image in which captured images are superimposed in the direction of the base line length B.

[0231] This change in base length B is represented by distance ΔX in Fig. 37(b). Note that in Fig. 37(b), positions C1 and C2 correspond to the optical centers of lenses 401a and 401b, respectively, in Fig. 8. Positions u1 and u2 are positions based on positions C1 and C2, respectively, and correspond to imaging positions i1 and i2, respectively, in Fig. 8.

[0232] When the images 4201, 4202, ... are arranged in the direction of the baseline length B in this way, the positions of corresponding points on the images 4201, 4202, ... are expressed as a straight line or curve with a slope m on the EPI 422. This slope m is the disparity q used when calculating the distance D. The slope m is smaller as the distance D is closer, and is larger as the distance D is farther. This straight line or curve with a slope m that varies depending on the distance D is called a feature point locus.

[0233] The gradient m is expressed by the following formula (13). In formula (13), the value Δu is the difference between positions u1 and u2, which are the imaging points in FIG. 37(b), and can be calculated by formula (14). From the gradient m, the distance D is calculated by formula (15). Here, in formulas (13) to (15), the value v indicates the moving speed of the camera, and the value f indicates the frame rate of the camera. That is, formulas (13) to (15) are calculation formulas when each panoramic image is captured at a frame rate f while the camera is moved at a constant speed v.

[0234]

number

[0235]

number

[0236]

number

[0237] In addition, when a panoramic image is used as an image constituting an EPI, the slope m is a value based on a curve. A description will be given with reference to Figs. 38 and 39. In Fig. 38, spheres 4111, 4112, and 4113 each have the structure of an imaging body 21, and show panoramic images captured by cameras #0, #ref, and #(n-1) arranged on a straight line. The distance (baseline length) between cameras #0 and #ref is distance d2, and the distance between cameras #ref and #(n-1) is distance d1. Hereinafter, spheres 4111, 4112, and 4113 will be referred to as panoramic images 4111, 4112, and 4113, respectively.

[0238] The imaging position of the target point P on the omnidirectional image 4111 is at an angle φ n-1 Similarly, the imaging positions of the target point P on the omnidirectional images 4112 and 4113 are angled at an angle φ ref and a position having an angle of φ0.

[0239] FIG. 39 shows these angles φ0, φ ref and φ n-1 39 is plotted on the vertical axis and the positions of each camera #0, #ref, and #(n-1) on the horizontal axis. As can be seen from Fig. 39, the feature point trajectories shown as the imaging positions in each of the spherical images 4111, 4112, and 4113 and the positions of each camera #0, #ref, and #(n-1) are not straight lines but are curves 413 that are approximated based on straight lines 4121 and 4122 that connect each point.

[0240] When calculating the parallax q of the entire circumference using the omnidirectional images 3001 to 3005, a method of directly searching for corresponding points on the curve 413 from the omnidirectional images 3001 to 3005 as described above may be used, or a method of converting the omnidirectional images 3001 to 3005 into a projection system such as a pinhole and searching for corresponding points based on each converted image may be used.

[0241] In the example of Fig. 38, of the celestial sphere images 4111, 4112, and 4113, the celestial sphere image 4112 is set as a reference image (ref), the celestial sphere image 4111 is set as the (n-1)th target image, and the celestial sphere image 4113 is set as the 0th target image. Based on the celestial sphere image 4112, which is the reference image, the corresponding points of the celestial sphere images 4111 and 4113 are determined by a parallax q n-1 and q0. This disparity q n-1 and q0 can be obtained using various known methods, such as the above-mentioned equation (11).

[0242] By using EPI to create a 3D reconstruction model, it becomes possible to process a large number of panoramic images in a unified manner. In addition, by using the gradient m, the calculation is not limited to point correspondence, making the processing more robust.

[0243] Returning to the description of the flowchart in FIG. 34, after the disparity calculation in step S11, the 3D information generation unit 120 shifts the process to step S12. In step S12, the 3D information generation unit 120 performs a correction process on the disparity information indicating the disparity calculated in step S11. As the correction process on the disparity information, correction based on the Manhattan-world hypothesis, line segment correction, or the like can be applied. In the next step S13, the 3D information generation unit 120 converts the disparity information corrected in step S12 into three-dimensional point cloud information. In the next step S14, the 3D information generation unit 120 performs a smoothing process, a meshing process, or the like as necessary on the three-dimensional point cloud information into which the disparity information has been converted in step S13. Through the process up to step S14, a three-dimensional restoration model based on each of the omnidirectional images 3001 to 3005 is generated.

[0244] The above-mentioned processes in steps S11 to S14 can be executed using open source distributed Structure-from-Motion (SfM) software, Multi-View Stereo (MVS) software, etc. The input program operated on the information processing device 100b includes, for example, the SfM software, the MVS software, etc.

[0245] As described with reference to Fig. 31, the imaging device 1b according to the second embodiment has the imaging bodies 211-215 arranged on the Z axis. Therefore, the distance from the imaging device 1b to each subject is calculated preferentially in the radial direction on a plane 40 shown in Fig. 40, which is perpendicular to the direction in which the imaging lenses 20a1-20a5 are aligned. The word "preferential" here refers to the ability to generate a 3D restoration model for the angle of view.

[0246] That is, in each direction on the plane 40 (radial direction in FIG. 40), the angle of view of each of the imaging bodies 211-215 can include the entire circumference of 360°, and distance calculation can be performed for the entire circumference. On the other hand, in the Z-axis direction, the overlapping portion of the angle of view (imaging range) of each of the imaging bodies 211-215 becomes large. Therefore, in the Z-axis direction, parallax is small in the omnidirectional images 3001-3005 based on the captured images captured by each of the imaging bodies 211-215 near the angle of view of 180°. Therefore, it is difficult to calculate the distance over the entire circumference of 360° in the direction of the plane including the Z axis.

[0247] Here, as illustrated in Fig. 41, a large space including large buildings 50, 50, ... is considered as a target for creating a 3D restoration model. In Fig. 41, the X-axis, Y-axis, and Z-axis coincide with the X-axis, Y-axis, and Z-axis shown in Fig. 31. In this case, modeling at the full angle of view (360°) is in the direction of a plane 40 represented by the XY axes. Therefore, it is preferable to align and arrange a plurality of imaging bodies 211-215 in the Z-axis direction perpendicular to this plane 40.

[0248] [Configuration of signal processing in the imaging device according to the second embodiment] Next, the configuration of the image pickup device 1b according to the second embodiment with respect to signal processing will be described. Fig. 42 is a block diagram showing an example of the configuration of the image pickup device 1b according to the embodiment. In Fig. 42, the same reference numerals are given to the parts corresponding to those in Fig. 30 described above, and detailed description thereof will be omitted.

[0249] 42, imaging device 1b includes imaging elements 200a1, 200a2, ..., 200a5, imaging elements 200b1, 200b2, ..., 200b5, driving units 210a1, 210a2, ..., 210a5, driving units 210b1, 210b2, ..., 210b5, buffer memories 211a1, 211a2, ..., 211a5, and buffer memories 211b1, 211b2, ..., 211b5.

[0250] Of these, imaging elements 200a1, 200a2, ..., 200a5, driving units 210a1, 210a2, ..., 210a5, and buffer memories 211a1, 211a2, ..., 211a5 are components corresponding to imaging lenses 20a1, 20a2, ..., 20a5, respectively, and are included in imaging bodies 211, 212, ..., 215. Note that, in order to avoid complexity, only imaging body 211 is shown among imaging bodies 211 to 215 in Fig. 42.

[0251] Similarly, the imaging elements 200b1, 200b2, ..., 200b5, the driving units 210b1, 210b2, ..., 210b5, and the buffer memories 211b1, 211b2, ..., 211b5 are components corresponding to the imaging lenses 20b1, 20b2, ..., 20b5, respectively, and are included in the imaging bodies 211, 212, ..., 215, respectively.

[0252] The imaging device 1b further includes a control unit 220, a memory 221, and a switch (SW) 222. The switch 222 corresponds to the shutter button 30 shown in Fig. 30. For example, the switch 222 is in a closed state, which corresponds to a state in which the shutter button 30 is operated.

[0253] A description will now be given of the imaging body 211. The imaging body 211 includes an imaging element 200a1, a driving section 210a1, and a buffer memory 211a1, and an imaging element 200b1, a driving section 210b1, and a buffer memory 211b1.

[0254] The driving unit 210a1 and the imaging element 200a1, as well as the driving unit 210b1 and the imaging element 200b1, are equivalent to the driving unit 210a and the imaging element 200a, as well as the driving unit 210b and the imaging element 200b described using FIG. 4, and therefore detailed description thereof will be omitted here.

[0255] The buffer memory 211a1 is a memory capable of storing at least one frame of captured image The captured image output from the driving section 210a1 is temporarily stored in the buffer memory 211a1.

[0256] It should be noted that the other imaging bodies 212 to 215 have the same functions as the imaging body 211, and therefore their explanation will be omitted here.

[0257] The control unit 220 controls the overall operation of the imaging device 1b. When the control unit 220 detects a transition of the switch 222 from an open state to a closed state, the control unit 220 outputs a trigger signal. The trigger signal is simultaneously supplied to each of the driving units 210a1, 210a2, ..., 210a5 and each of the driving units 210b1, 210b2, ..., 210b5.

[0258] The memory 221 reads out each captured image from each of the buffer memories 211a1, 211a2, ..., 211a5 and each of the buffer memories 211b1, 211b2, ..., 211b5 under the control of the control unit 220 in response to the output of a trigger signal, and stores each of the read captured images. Each captured image stored in the memory 221 can be read out by the information processing device 100b connected to the imaging device 1b.

[0259] The battery 2020 is a secondary battery such as a lithium ion secondary battery, and serves as a power supply unit that supplies power to each unit in the imaging device 1b that requires power supply. The battery 2020 includes a charge / discharge control circuit that controls charging and discharging of the secondary battery.

[0260] Fig. 43 is a block diagram showing an example of the configuration of the control unit 220 and the memory 221 in the imaging device 1b according to the second embodiment. In Fig. 43, the same reference numerals are used for the parts common to Fig. 4 described above, and detailed description thereof will be omitted.

[0261] 43, the control unit 220 includes a CPU 2000, a ROM 2001, a trigger I / F 2004, a switch (SW) circuit 2005, a data I / F 2006, and a communication I / F 2007, and these units are communicatively connected to a bus 2010. The memory 221 includes a RAM 2003 and a memory controller 2002, and the memory controller 2002 is further connected to the bus 2010. Power is supplied from a battery 2020 to each of the CPU 2000, the ROM 2001, the memory controller 2002, the RAM 2003, the trigger I / F 2004, the switch circuit 2005, the data I / F 2006, the communication I / F 2007, and the bus 2010.

[0262] The memory controller 2002, in accordance with instructions from the CPU 2000, controls the storage and reading of data into and from the RAM 2003. The memory controller 2002, in accordance with instructions from the CPU 2000, also controls the reading of captured images (hemispherical images) from each of the buffer memories 211a1, 211a2, ..., 211a5 and each of the buffer memories 211b1, 211b2, ..., 211b5.

[0263] The switch circuit 2005 detects the transition of the switch 222 between the closed state and the open state, and passes the detection result to the CPU 2000. When the CPU 2000 receives the detection result from the switch circuit 2005 that the switch 222 has transitioned from the open state to the closed state, it outputs a trigger signal. The trigger signal is output via a trigger I / F 2004, and is branched to be supplied to each of the drive units 210a1, 210a2, ..., 210a5 and each of the drive units 210b1, 210b2, ..., 210b5.

[0264] Although the CPU 2000 has been described as outputting a trigger signal in response to the detection result by the switch circuit 2005, this is not limited to this example. For example, the CPU 2000 may output a trigger signal in response to a signal supplied via the data I / F 2006 or the communication I / F 2007. Furthermore, the trigger I / F 2004 may generate a trigger signal in response to the detection result of the switch circuit 2005 and supply it to each of the drive units 210a1, 210a2, ..., 210a5 and each of the drive units 210b1, 210b2, ..., 210b5.

[0265] In this configuration, when the control unit 220 detects a transition of the switch 222 from an open state to a closed state, the control unit 220 generates and outputs a trigger signal. The trigger signal is supplied to each of the driving units 210a1, 210a2, ..., 210a5 and each of the driving units 210b1, 210b2, ..., 210b5 at the same timing. Each of the driving units 210a1, 210a2, ..., 210a5 and each of the driving units 210b1, 210b2, ..., 210b5 reads out electric charges from each of the image pickup elements 200a1, 200a2, ..., 200a5 and each of the image pickup elements 200b1, 200b2, ..., 200b5 in synchronization with the supplied trigger signal.

[0266] Each driving unit 210a1, 210a2, ..., 210a5 and each driving unit 210b1, 210b2, ..., 210b5 converts the charges read out from each imaging element 200a1, 200a2, ..., 200a5 and each imaging element 200b1, 200b2, ..., 200b5 into an image, and stores the converted image in each buffer memory 211a1, 211a2, ..., 211a5 and each buffer memory 211b1, 211b2, ..., 211b5.

[0267] The control unit 220 instructs the memory 221 to read the captured images from the buffer memories 211a1, 211a2, ..., 211a5 and the buffer memories 211b1, 211b2, ..., 211b5 at a predetermined timing after the output of the trigger signal. In the memory 221, the memory controller 2002 reads the captured images from the buffer memories 211a1, 211a2, ..., 211a5 and the buffer memories 211b1, 211b2, ..., 211b5 in response to this instruction, and stores the read captured images in a predetermined area of ​​the RAM 2003.

[0268] When the information processing device 100b is connected to the imaging device 1b via, for example, the data I / F 2006, the information processing device 100b requests reading of each captured image (hemispherical image) stored in the RAM 2003 via the data I / F 2006. In response to this request, the CPU 2000 instructs the memory controller 2002 to read each captured image from the RAM 2003. In response to this instruction, the memory controller 2002 reads each captured image from the RAM 2003 and transmits each read captured image to the information processing device 100b via the data I / F 2006. The information processing device 100b executes the process according to the flowchart of FIG. 34 based on each captured image transmitted from the imaging device 1b.

[0269] Fig. 43 shows an example of the configuration of the control unit 220 and the memory 221 in the imaging device 1b according to the second embodiment. In Fig. 43, the same reference numerals are used for the parts common to Fig. 4 described above, and detailed description thereof will be omitted.

[0270] 43, the control unit 220 includes a CPU 2000, a ROM 2001, a trigger I / F 2004, a switch (SW) circuit 2005, a data I / F 2006, and a communication I / F 2007, and these units are communicatively connected to a bus 2010. The memory 221 includes a RAM 2003 and a memory controller 2002, and the memory controller 2002 is further connected to the bus 2010. Power is supplied from a battery 2020 to each of the CPU 2000, the ROM 2001, the memory controller 2002, the RAM 2003, the trigger I / F 2004, the switch circuit 2005, the data I / F 2006, the communication I / F 2007, and the bus 2010.

[0271] The memory controller 2002, in accordance with instructions from the CPU 2000, controls the storage and reading of data into and from the RAM 2003. The memory controller 2002, in accordance with instructions from the CPU 2000, also controls the reading of captured images (hemispherical images) from each of the buffer memories 211a1, 211a2, ..., 211a5 and each of the buffer memories 211b1, 211b2, ..., 211b5.

[0272] The switch circuit 2005 detects the transition of the switch 222 between the closed state and the open state, and passes the detection result to the CPU 2000. When the CPU 2000 receives the detection result from the switch circuit 2005 that the switch 222 has transitioned from the open state to the closed state, it outputs a trigger signal. The trigger signal is output via the trigger I / F 2004, branches, and is sent to each of the driving units 210a1, 210a2, ..., 210a5 and each of the driving units 210b. 1 , 210b2, ..., 210b5, respectively.

[0273] Although the CPU 2000 has been described as outputting a trigger signal in response to the detection result by the switch circuit 2005, this is not limited to this example. For example, the CPU 2000 may output a trigger signal in response to a signal supplied via the data I / F 2006 or the communication I / F 2007. Furthermore, the trigger I / F 2004 may generate a trigger signal in response to the detection result of the switch circuit 2005 and supply it to each of the drive units 210a1, 210a2, ..., 210a5 and each of the drive units 210b1, 210b2, ..., 210b5.

[0274] In this configuration, when the control unit 220 detects a transition of the switch 222 from an open state to a closed state, the control unit 220 generates and outputs a trigger signal. The trigger signal is supplied to the driving units 210a1, 210a2, ..., 210a5 and the driving units 210b1, 210b2, ..., 210b5 at the same timing. The driving units 210a1, 210a2, ..., 210a5 and the driving units 210b1, 210b2, ..., 210b5 synchronize with the supplied trigger signal to control the image pickup elements 200a1, 200a2, ..., 200a5 and the image pickup elements 200b1, 200b 2 , ..., read out the charge from 200b5.

[0275] Each driving unit 210a1, 210a2, ..., 210a5 and each driving unit 210b1, 210b2, ..., 210b5 converts the charges read out from each imaging element 200a1, 200a2, ..., 200a5 and each imaging element 200b1, 200b2, ..., 200b5 into an image, and stores the converted image in each buffer memory 211a1, 211a2, ..., 211a5 and each buffer memory 211b1, 211b2, ..., 211b5.

[0276] The control unit 220 instructs the memory 221 to read the captured images from the buffer memories 211a1, 211a2, ..., 211a5 and the buffer memories 211b1, 211b2, ..., 211b5 at a predetermined timing after the output of the trigger signal. In the memory 221, the memory controller 2002 reads the captured images from the buffer memories 211a1, 211a2, ..., 211a5 and the buffer memories 211b1, 211b2, ..., 211b5 in response to this instruction, and stores the read captured images in a predetermined area of ​​the RAM 2003.

[0277] When the information processing device 100b is connected to the imaging device 1b via, for example, the data I / F 2006, the information processing device 100b requests reading of each captured image (hemispherical image) stored in the RAM 2003 via the data I / F 2006. In response to this request, the CPU 2000 instructs the memory controller 2002 to read each captured image from the RAM 2003. In response to this instruction, the memory controller 2002 reads each captured image from the RAM 2003 and transmits each read captured image to the information processing device 100b via the data I / F 2006. The information processing device 100b executes the process according to the flowchart of FIG. 34 based on each captured image transmitted from the imaging device 1b.

[0278] In the imaging device 1b according to the second embodiment, the battery 2020 and the circuit section 2030 are provided inside the housing 10b, similarly to the imaging device 1a according to the first embodiment described with reference to FIG. 5. At least the battery 2020 out of the battery 2020 and the circuit section 2030 is fixed inside the housing 10b by a fixing means such as an adhesive or a screw. The circuit section 2030 includes at least each element of the control section 220 and the memory 221 described above. The control section 220 and the memory 221 are configured on, for example, one or more circuit boards. In the imaging device 1b, the battery 2020 and the circuit section 2030 are arranged on an extension of the row in which the imaging lenses 20a1, 20a2, ..., 20a5 are arranged in an aligned manner.

[0279] Note that, although it has been described here that the battery 2020 and the circuit section 2030 are arranged in that position, this is not limited to this example, and for example, if the circuit section 2030 is sufficiently small, it is sufficient that at least the battery 2020 is arranged in that position.

[0280] By arranging the battery 2020 and the circuit section 2030 in this manner, the imaging lenses 20a1, 20a2, ..., 20a5 (and the imaging lenses 20b1, 20b2, ..., 20b 5 ) is arranged, the width of the surfaces (front and back) can be narrowed. This prevents a part of the housing 10b of the imaging device 1b from being included in each captured image captured by each of the imaging bodies 211, 212, ..., 215, making it possible to calculate the parallax with higher accuracy. For the same reason, it is preferable to make the width of the housing 10b of the imaging device 1b as narrow as possible. The same applies to the imaging device 1a according to the first embodiment.

[0281] [Annotation input method according to the second embodiment] Next, an annotation input method according to the second embodiment will be described. As described above, in the information processing device 100b, the UI unit 113 displays a screen for inputting an annotation by using a celestial sphere image based on each hemispherical image captured by one of the five imaging bodies 211 to 215 (for example, the imaging body 211) of the imaging device 1b. That is, the UI unit 113 cuts out an image of a partial area of ​​the celestial sphere image, and displays the cut image on the screen 500 as, for example, an image 5010 shown in FIG.

[0282] At this time, the UI unit 113 acquires the three-dimensional point cloud information generated according to the flowchart of Fig. 34 from the 3D information generation unit 120. The UI unit 113 can switch the display on the screen 500 between the image 5010 and the three-dimensional point cloud information in response to, for example, an operation on a menu button 5050. The UI unit 113 displays each point included in the three-dimensional point cloud information in a different color according to, for example, the distance.

[0283] Also in the second embodiment, the annotation input process is performed according to the above-mentioned flowchart of Fig. 16. Here, the position designation in step S103 in the flowchart of Fig. 16 is performed by three-dimensional coordinates based on three-dimensional point group information.

[0284] Fig. 44 is a diagram for explaining position specification according to the second embodiment. As shown in Fig. 44(a), objects 7001, 7002, and 7003 having a three-dimensional structure are arranged in a three-dimensional space represented by mutually orthogonal X-axis, Y-axis, and Z-axis, and a case will be considered in which the objects are imaged by an imaging device 1b. Three-dimensional point cloud information generated based on each celestial sphere image acquired by imaging by the imaging device 1b includes, for example, three-dimensional information at each point on the surface of each of the objects 7001 to 7003 facing the imaging position from the imaging position.

[0285] Fig. 44(b) shows an example of an image 5010 obtained by cutting out a part of the omnidirectional image captured by the imaging body 211 of the imaging device 1b in the state of Fig. 44(a). The UI unit 13 displays the image 5010 shown in Fig. 44(b) on the screen 500. It is assumed that the user has specified positions of points on objects 7001 and 7002, as shown by markers 602a and 602b, for example, in the image 5010 shown in Fig. 44(b).

[0286] Here, in the second embodiment, it is possible to obtain three-dimensional point cloud information of a space captured by the imaging device 1b, including the objects 7001 to 7003. Therefore, position designation is performed for positions indicated by the three-dimensional coordinates of the objects 7001 and 7002, as shown as markers 602a' and 602b' in FIG.

[0287] Furthermore, by specifying a position using three-dimensional coordinates based on three-dimensional point cloud information, it is possible to calculate the length of the line or the area of ​​the polygon when the position is specified by a line or polygon, as described in the modified example of the first embodiment above.

[0288] Fig. 45 is a diagram showing an example of position specification by lines according to the second embodiment. In Fig. 45, objects 7004 and 7005 having a three-dimensional structure are arranged in a three-dimensional space represented by mutually orthogonal X-axis, Y-axis, and Z-axis, similar to Fig. 44(a). In addition, a crack 610a is observed on the cylindrical surface of the cylindrical object 7004 in a direction going around the cylindrical surface. A crack 610b is observed across the sides of the rectangular parallelepiped (rectangular prism) object 7005.

[0289] In this way, it is difficult to calculate the length of the cracks 610a and 610b having depth from a two-dimensional captured image. For example, the length of the crack 610 included in the image 5010 of FIG. 26 described above can be measured only if the wall surface on which the crack 610 exists is parallel to the surface of the image 5010. In the second embodiment, since the three-dimensional point cloud information of these objects 7004 and 7005 is acquired, the length of the cracks 610a and 610b having depth can be easily calculated. The same applies to the calculation of the area and perimeter of a polygon.

[0290] Although the above-described embodiments are preferred examples of the present invention, the present invention is not limited to these, and various modifications can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0291] 1a, 1b Imaging device 4 Buildings 6 Server 20a, 20a1, 20a2, 20a3, 20a4, 20a5, 20b, 20b1, 20b2, 20b3, 20b4, 20b5 imaging lenses 21,211,212,213,214,215 Imaging body 30 Shutter Button 100a, 100b Information processing device 110 Image acquisition unit 111 Image processing section 112 Additional Information Generation Unit 113 UI section 114 Communications Department 115 Output section 120 3D information generation section 200a, 200b, 200a1, 200a2, 200a5, 200b1, 200b2, 200b5 image sensor 210a, 210b, 210a1, 210a2, 210a5, 210b1, 210b2, 210b5 drive unit 211a1, 211a2, 211a5, 211b1, 211b2, 211b5 buffer memory 3001, 3002, 3003, 3004, 3005, 510 Panoramic images 500 screens 511 areas 600 Annotation input screen Tabs 601a, 601b, 601c 602,602' Marker 610,610a,610b Crack 2002 Memory Controller 2003 RAM 2004 Trigger I / F 2005 Switch Circuit 5010,5010a,5010b,5010c Images 5011 Floor plan image 5030a,5030b,5030c,5030d Tags 50401, 50402, 50403, 5040 10 ,5040 11 ,5040 12 Pin Marker 6010 Annotation input area 6010a, 6010b, 6010c, 6010d, 6010e Input area [Prior art documents] [Patent documents]

[0292] [Patent Document 1] JP 2018-013878 A

Claims

1. A receiving means for receiving an operation input; A display means for generating a screen to be displayed on the display unit; having the display unit is capable of generating a first screen that switches between and displays a plurality of omnidirectional images captured at different imaging positions, and a second screen that displays auxiliary information associated with a predetermined position in any one of the plurality of omnidirectional images, the accepting means accepts, on the second screen, an operation input for designating link information associated with the supplementary information; the display means generates the first screen displaying the spherical image to which the auxiliary information is associated in accordance with the specified link information. An information processing system comprising:

2. The second screen displays an image acquired from the spherical image or an image acquired from an external source.

2. The information processing system according to claim 1 .

3. The spherical image is associated with a plurality of pieces of auxiliary information, the accepting means accepts an operation input for designating one or more pieces of the auxiliary information from the plurality of pieces of auxiliary information; the second screen displays the one or more pieces of specified auxiliary information; 3. The information processing system according to claim 1 or 2.

4. When receiving an operation input specifying the additional information, the display means displays a tag indicating the additional information on the spherical image, or displays a list of the additional information.

4. The information processing system according to claim 3.

5. The operation input for specifying the one or more pieces of additional information is an operation input for specifying a search condition.

5. The information processing system according to claim 3 or 4.

6. Further comprising an output means, The output means outputs data summarizing the plurality of pieces of auxiliary information.

6. The information processing system according to claim 1, wherein the information processing system is a data processing system.

7. The data includes the link information, the plurality of pieces of auxiliary information included in the data are displayed on the second screen, the accepting means accepts, on the second screen, an operation input for designating the link information included in the data; 7. The information processing system according to claim 6.

8. The data includes a time-series image in which a plurality of images taken at different times are arranged in order of their capture time.

8. The information processing system according to claim 6 or 7.

9. An information processing device including at least the output means; A server connected to the information processing device; Equipped with The supplementary information is stored in the server, The output means acquires the additional information from the server and outputs the data.

9. The information processing system according to claim 6, wherein the first and second inputs are connected to the first and second inputs.

10. The data output from the output means is transmitted to the server.

10. The information processing system according to claim 9.

11. A receiving means for receiving an operation input; A display means for generating a screen to be displayed on the display unit; having the display means is capable of generating a first screen that switches between and displays a plurality of omnidirectional images having different imaging positions, and a second screen that displays an image associated with a predetermined position in any one of the plurality of omnidirectional images, the accepting means accepts an operation input for designating the image on the second screen, The display means generates the first screen that displays the spherical image with which the specified image is associated. An information processing system comprising:

12. The second screen displays a plurality of the images; the accepting means accepts an operation input for designating one of the plurality of images; 12. The information processing system according to claim 11.

13. The image is an image acquired from the omnidirectional image or an image acquired from outside.

13. The information processing system according to claim 11 or 12.

14. The spherical image is a three-dimensional image having three-dimensional information.

14. The information processing system according to claim 1, wherein the information processing system further comprises:

15. A computer comprising: generating a first screen that switches between and displays a plurality of spherical images captured at different positions; generating a second screen displaying auxiliary information associated with a predetermined position in any one of the plurality of omnidirectional images; receiving an operation input specifying link information associated with the supplementary information on the second screen; generating the first screen displaying the spherical image to which the auxiliary information is associated in accordance with the specified link information; A program that executes the following.

16. A computer comprising: generating a first screen that switches between and displays a plurality of spherical images captured at different positions; generating a second screen that displays an image associated with a predetermined position in any one of the plurality of omnidirectional images; receiving an operation input for designating the image on the second screen; generating the first screen displaying the spherical image with which the specified image is associated; A program that executes the following.

Images