Image processing method, program, image processing device, and image processing system

Abstract

To provide a virtual tour with improved accuracy in estimating the photographic position of a photographic image.SOLUTION: An image processing method, which can be executed by an image processing device 50 that performs processing on a photographic image captured in all directions within a predetermined base, includes: a still image acquisition step of acquiring a plurality of still images captured at different photographic positions within the base; a moving image acquisition step of acquiring a moving image captured while moving from a first point to a second point within the base; a position estimation step of estimating relative photographic positions of the plurality of still images based on the acquired moving image; and an image processing step of generating a processed image (for example, a tour image) including the plurality of still images associated with each other based on the estimated photographic positions.SELECTED DRAWING: Figure 15

Description

Technical Field

[0001] The present disclosure relates to an image processing method, a program, an image processing apparatus, and an image processing system.

Background Art

[0002] There is known a system that distributes image data captured using a photographing device capable of photographing in all directions and allows the situation at a remote site to be viewed at another site. Since a panoramic image obtained by photographing a predetermined site in all directions can be viewed by a viewer in an arbitrary direction, it is possible to convey immersive information. Such a system is used, for example, in fields such as online interior view of properties in the real estate industry.

[0003] Furthermore, there is a virtual tour service that connects captured images taken at a plurality of photographing positions of a real estate property, as if walking through the inside of the property. For creating such a virtual tour, it is important to connect the captured images taken at a plurality of photographing positions in the correct positional relationship. For creating such a virtual tour, a technique for estimating the position of a captured image using acceleration information measured by an IMU (inertial measurement unit) is already known (for example, Patent Document 1 and Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional method of estimating position based on acceleration using an IMU, there was room for improvement in terms of the accuracy of estimating the positions of a plurality of captured images for creating a virtual tour.

Means for Solving the Problems

[0006] In order to solve the above-described problems, the invention according to claim 1 is an image processing method executed by an image processing apparatus that performs processing on captured images taken omnidirectionally within a predetermined base point, the method including: a still image acquisition step of acquiring a plurality of still images captured at different shooting positions within the base point; a moving image acquisition step of acquiring a moving image captured while moving from a first point to a second point within the base point; an estimation step of estimating relative shooting positions of the plurality of still images based on the acquired moving image; and an image processing step of generating a processed image including the plurality of still images associated based on the estimated shooting positions.

Advantages of the Invention

[0007] According to the present invention, there is an effect that a virtual tour with improved estimation accuracy of the shooting positions of captured images can be provided.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the invention will be described with reference to the drawings. In the description of the drawings, the same reference numerals are assigned to the same elements, and duplicate descriptions are omitted.

[0010] ●Embodiment● ●Outline of the Image Processing System First, with reference to FIG. 1, the outline of the configuration of the image processing system according to the embodiment will be described. FIG. 1 is a diagram showing an example of the overall configuration of the image processing system. The image processing system 1 shown in FIG. 1 is a system that performs image processing on a photographed image for allowing a viewer to view online the internal space of a structure such as a real estate property.

[0011] As shown in FIG. 1, the image processing system 1 includes a photographing device 10, an image processing device 50, and a communication terminal 90. The photographing device 10, the image processing device 50, and the communication terminal 90 constituting the image processing system 1 can communicate via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), or the like. Note that the communication network 100 may include not only wired communication but also networks by wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] The image processing device 50 is a server computer that performs image processing on a photographed image of the interior space of a structure such as a real estate property at a predetermined base. The image processing device 50, for example, acquires a photographed image taken by the photographing device 10 and generates a tour image for providing a virtual tour to the user using the acquired photographed image. Here, the virtual tour is content that allows the user to view as if actually viewing the interior of a structure such as a real estate property on site. Also, the tour image is a viewing image generated using a plurality of photographed images taken by the photographing device 10 and allows the user to virtually move inside the base shown in the photographed images by user operation.

[0013] Note that the image processing device 50 may be configured by a single server computer or may be configured by a plurality of server computers. Also, although the image processing device 50 is described as being a server computer existing in a cloud environment, it may be a server existing in an on-premises environment.

[0014] The imaging device 10 is a special digital camera (omnidirectional imaging device) capable of capturing images of the interior space of structures such as real estate properties at the imaging site in all directions to obtain a panoramic (360°) image. The imaging device 10 is used, for example, by real estate agents who manage or sell real estate properties. Note that the imaging device 10 may be a wide-angle camera or a stereo camera capable of obtaining a wide-angle image having an angle of view equal to or greater than a predetermined value. A wide-angle image is generally an image captured using a wide-angle lens, and is an image captured with a lens capable of capturing a range wider than that perceived by the human eye. That is, the imaging device 10 is an imaging means capable of obtaining an image (panoramic image, wide-angle image) captured using a lens having a focal length shorter than a predetermined value. A wide-angle image generally means an image captured with a lens having a focal length of 35 mm or less in terms of 35 mm film conversion.

[0015] The communication terminal 90 is a computer such as a smartphone that displays the image processed by the image processing device 50 for a viewer to view. The communication terminal 90 is used, for example, by the same real estate agent as the imaging device 10. The communication terminal 90 is installed with, for example, a dedicated application for giving an imaging instruction to the imaging device 10 and viewing the image provided from the image processing device 50. Also, the communication terminal 90 may be configured to give an imaging instruction and view an image by accessing a dedicated website using a web browser without using a dedicated application, for example. Also, the imaging instruction and image viewing may be performed by different communication terminals 90.

[0016] Note that the communication terminal 90 is not limited to a smartphone, and may be, for example, a PC, a tablet terminal, a wearable terminal, an HMD (head mount display), or an IWB (Interactive White Board: a whiteboard having an electronic blackboard function capable of mutual communication).

[0017] ○Overview of the imaging device○ Here, with reference to FIGS. 2 to 11, the outline of the imaging device 10 that constitutes the image processing system 1 will be described. FIG. 2 is a diagram showing an example of a panoramic image captured by the imaging device. The image shown in FIG. 2 is a panoramic image of a room of a real estate property, which is an example of the internal space of a structure, captured by the imaging device 10. Since the panoramic image can capture the interior of the room in all directions, it is suitable for viewing real estate properties. The form of the panoramic image varies, but it is often generated by the Equirectangular projection method described later. The image generated by this Equirectangular projection method has the advantages that the outer shape of the image is rectangular, making it efficient and easy to store image data, and there is less distortion near the equator and vertical lines are not distorted, so it looks relatively natural.

[0018] ○ Method for generating panoramic image Next, with reference to FIGS. 3 to 9, the method for generating a panoramic image will be described. First, with reference to FIGS. 3 and 4, the outline of the process from the image captured by the imaging device 10 to the generation of the panoramic image will be described. FIG. 3(A) shows a hemispherical image (front side) captured by the imaging device, FIG. 3(B) shows a hemispherical image (rear side) captured by the imaging device, and FIG. 3(C) is a diagram showing an image represented by the Equirectangular projection method (hereinafter referred to as "Equirectangular projection image"). FIG. 4(A) is a conceptual diagram showing the state of covering the sphere with the Equirectangular projection image, and FIG. 4(B) is a diagram showing the panoramic image.

[0019] The imaging device 10 is provided with imaging elements on the front side (front) and the back side (rear) respectively. These imaging elements (image sensors) are used in combination with optical members such as lenses that can capture hemispherical images (viewing angle of 180° or more). The imaging device 10 can obtain two hemispherical images by imaging the subject around the user with the two imaging elements respectively.

[0020] As shown in FIGS. 3(A) and 3(B), the images obtained by the imaging device 10 of the imaging device are curved hemispherical images (front side and rear side). Then, the imaging device 10 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) inverted by 180 degrees to create an equidistant cylindrical projection image EC as shown in FIG. 3(C).

[0021] Then, the imaging device 10 uses OpenGL ES (Open Graphics Library for Embedded Systems) to paste the equidistant cylindrical projection image EC so as to cover the spherical surface as shown in FIG. 4(A), and create a full-sphere image (full-sphere panorama image) CE as shown in FIG. 4(B). In this way, the full-sphere image CE is represented as an image in which the equidistant cylindrical projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Also, the full-sphere image CE may be a still image or a moving image. Furthermore, the conversion method is not limited to OpenGL ES, and any method that can convert from a hemispherical image to an equidistant cylindrical projection method may be used, for example, an operation by a CPU or an operation by OpenCL may be used.

[0022] As described above, since the full-sphere image CE is an image pasted so as to cover the spherical surface, it will feel uncomfortable to humans. Therefore, the imaging device 10 can display a part of a predetermined region T (hereinafter referred to as a "predetermined region image") of the full-sphere image CE as a planar image with less curvature, so as to give a non-uncomfortable display to humans. This will be described with reference to FIGS. 5 and 6.

[0023] FIG. 5 is a diagram showing the position of a virtual camera and a predetermined region when the omnidirectional image is a three-dimensional spherical object. The virtual camera IC corresponds to the position of the viewpoint of a user who views the omnidirectional image CE displayed as a three-dimensional spherical object. FIG. 5 represents the omnidirectional image CE as a three-dimensional spherical object CS. When the thus-generated omnidirectional image CE is the spherical object CS, as shown in FIG. 5, the virtual camera IC is located inside the omnidirectional image CE. A predetermined region T in the omnidirectional image CE is a shooting region of the virtual camera IC, and is specified by predetermined region information indicating the shooting direction and the angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE. Further, the zoom of the predetermined region T can also be expressed by bringing the virtual camera IC closer to or away from the omnidirectional image CE. The predetermined region image Q is an image of the predetermined region T in the omnidirectional image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE.

[0024] Then, the predetermined region image Q is displayed on a predetermined display as an image of the shooting region of the virtual camera IC. Hereinafter, the description will be made using the shooting direction (ea, aa) and the angle of view (α) of the virtual camera IC. Note that the predetermined region T may be indicated by the imaging region (X, Y, Z) of the virtual camera IC which is the predetermined region T instead of the angle of view α and the distance f.

[0025] Next, with reference to FIG. 6, the relationship between the predetermined region information and the image of the predetermined region T will be described. FIG. 6 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in FIG. 6, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the posture of the virtual camera IC is changed so that the fixation point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP(x, y) of the predetermined region T which is the shooting region of the virtual camera IC. As shown in FIG. 6, when the diagonal angle of view of the predetermined region T represented by the angle of view α of the virtual camera IC is α, the center point CP(x, y) becomes the parameter ((x, y)) of the predetermined region information. The predetermined region image Q is an image of the predetermined region T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between an arbitrary vertex of the predetermined region T and the center point CP(x, y) (2L is the diagonal). And in FIG. 6, generally, the trigonometric function represented by the following (Equation 1) holds.

[0026] [Number]

[0027] Next, with reference to FIG. 7, the state during shooting by the imaging device 10 will be described. FIG. 7 is a diagram showing an example of the state during shooting by the imaging device. In order to shoot so as to overlook the entire room of a real estate property or the like, it is preferable to install the imaging device 10 at a position close to the height of a human eye. Therefore, as shown in FIG. 7, it is common for the imaging device 10 to perform shooting while fixing the imaging device 10 with a support member 20 such as a monopod or a tripod. As described above, the imaging device 10 is an omnidirectional imaging device capable of acquiring light rays in all directions of the entire periphery, and it can also be said that it acquires an image (omnidirectional image CE) on the unit sphere around the imaging device 10. When the shooting direction of the imaging device 10 is determined, the coordinates of the omnidirectional image are determined. For example, in FIG. 7, point A is at a distance of (d, -h) from the center point C of the imaging device 10. At this time, if the angle formed by the line segment AC and the horizontal direction is θ, the angle θ can be expressed by the following (Equation 2).

[0028] [Number]

[0029] Assuming that point A is at the depression angle θ, the distance d between point A and point B can be expressed by the following (Equation 3) using the installation height h of the imaging device 10.

[0030] [Number]

[0031] Here, a process of converting the position information on the omnidirectional image into coordinates on a planar image converted from the omnidirectional image will be schematically described. FIG. 8 is a diagram for explaining an example of an omnidirectional image. Note that FIG. 8(A) is a diagram showing the hemispherical image shown in FIG. 3(A) with lines connecting the locations where the incident angles in the horizontal and vertical directions with respect to the optical axis are equal. Hereinafter, the incident angle in the horizontal direction with respect to the optical axis is referred to as "θ", and the incident angle in the vertical direction with respect to the optical axis is referred to as "φ".

[0032] Also, FIG. 9(A) is a diagram for explaining an example of an image processed by the orthographic cylindrical projection method. Specifically, the image shown in FIG. 8 is associated with a pre-generated LUT (Look Up Table) or the like, processed by the orthographic cylindrical projection method, and the respective images shown in FIGS. 8(A) and 8(B) after processing are combined, then the planar image shown in FIG. 9(A) corresponding to the omnidirectional image is generated by the imaging device 10. The orthographic cylindrical projection image EC shown in FIG. 3(C) is an example of the planar image shown in FIG. 9(A).

[0033] As shown in Fig. 9(A), in the image processed by the orthographic cylindrical projection method, the latitude (θ) and longitude (φ) are orthogonal. In the example shown in Fig. 9(A), by setting the center of the image as (0, 0) and expressing the latitude direction as -90 to +90 and the longitude direction as -180 to +180, any position in the full-sphere image can be indicated. For example, the coordinates of the upper left of the image are (-180, -90). Note that the coordinates of the full-sphere image may be represented in a format using 360 degrees as shown in Fig. 9(A), or may be represented in radian notation or in terms of the number of pixels like a real image. Also, the coordinates of the full-sphere image may be converted and represented as two-dimensional coordinates (x, y) as shown in Fig. 9(B).

[0034] Note that the compositing process for the planar image shown in Fig. 9(A) or Fig. 9(B) is not limited to simply continuously arranging the hemispherical images shown in Fig. 8(A) and Fig. 8(B). For example, when the horizontal center of the full-sphere image is not θ = 180°, in the compositing process, the imaging device 10 first preprocesses the hemispherical image shown in Fig. 3(C) and arranges it at the center of the full-sphere image. Next, the imaging device 10 divides the preprocessed image of the hemispherical image shown in Fig. 3(B) into sizes that can be arranged on the left and right parts of the generated image, and may generate the orthographic cylindrical projection image EC shown in Fig. 3(C) by compositing the hemispherical images.

[0035] Also, in the planar image shown in Fig. 9(A), the locations corresponding to the poles (PL1 or PL2) of the hemispherical images (full-sphere images) shown in Fig. 8(A) and Fig. 8(B) become line segments CT1 or CT2. This is because, as shown in Fig. 4(A) and Fig. 4(B), the full-sphere image (for example, the full-sphere image CE) is created by pasting the planar image (orthographic cylindrical projection image EC) shown in Fig. 9(A) onto the spherical surface using OpenGL ES.

[0036] ○ Example of an imaging device applicable to the image processing system Next, with reference to FIGS. 10 and 11, an example of the imaging device 10 applicable to the image processing system 1 according to the embodiment will be described. FIG. 10 is a schematic diagram for explaining an example of the imaging device applicable to the image processing system. FIG. 10(A) shows a special imaging device including a plurality of imaging elements capable of generating an omnidirectional image by the generation method as described above. The special imaging device uses a wide-angle lens or a fish-eye lens with a wide angle of view, and can obtain an image in which the entire direction is photographed by combining the outputs of the plurality of imaging elements. FIG. 10(B) shows a general imaging device which is a so-called normal camera. The general imaging device is, for example, a mobile terminal such as a normal digital camera or a smartphone equipped with a camera. The photographer holds the general imaging device in hand and rotates it while taking pictures. The general imaging device can obtain an omnidirectional image by synthesizing the obtained images. Both the special imaging device and the general imaging device generate a final captured image by connecting a plurality of captured results by image processing (stitching process). Note that it is preferable that the optical centers of the imaging device 10 for obtaining a plurality of captured results are the same.

[0037] FIG. 11 is a diagram for explaining an example of a captured image captured by a general imaging device. FIG. 11 shows a captured image when a photographer holds and rotates a general imaging device as shown in FIG. 10(B). Since the general imaging device has a small angle of view (generally 100 degrees or less), as shown in FIG. 11, there is a problem that the upper and lower poles cannot be captured. Further, depending on the rotation by the photographer himself / herself, the position of the optical center of the imaging device is shifted, resulting in parallax during imaging, and unnatural errors such as steps are likely to occur in the stitching process. That is, as the imaging device 10 in the present embodiment, either a special imaging device or a general imaging device as shown in FIG. 10 can be applied, but preferably, a special imaging device as shown in FIG. 10(A) is preferably applied. The image processing system 1 can use a seamless and natural high-quality omnidirectional image as an image to be used for a virtual tour that requires quality such as an advertisement by using a special imaging device (omnidirectional imaging device) as shown in FIG. 10(A). In the following description, the imaging device 10 will be described as a special imaging device (omnidirectional imaging device).

[0038] ○Outline of Processing of Image Processing Apparatus○ Next, the outline of the processing executed by the image processing apparatus 50 will be described with reference to FIG. 12. FIG. 12 is a schematic diagram for explaining an example of the processing executed by the image processing apparatus. FIG. 12 shows the relationship between the shooting positions of the captured images captured by the imaging device 10 and the paths associating the plurality of captured images.

[0039] The image processing apparatus 50 estimates the shooting position (FIG. 12(A)) of the captured image acquired from the imaging device 10 by using a method such as visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion). The estimation result of the shooting position is expressed as the relative position of the shooting positions of each of the plurality of captured images, as shown in FIG. 12(B). Further, the estimation result of the shooting position is such that each shooting position is expressed in one coordinate system.

[0040] In addition to estimating the shooting position, the image processing apparatus 50 can also restore the path traveled during the shooting of the captured image, and can also restore in what order and through what path each shooting position was shot. Based on the estimated shooting position and the path traveled during shooting, the image processing apparatus 50 generates a path (tour path). The tour path indicates the connection relationship of a plurality of captured images on the tour image. As shown in FIG. 12(C), by connecting the captured images taken at each estimated shooting position and enabling the connected captured images to move back and forth between each other, a virtual tour can be created using the captured images.

[0041] ●Hardware Configuration Next, with reference to FIGS. 13 and 14, the hardware configuration of each device or terminal constituting the image processing system according to the embodiment will be described. Note that the hardware configuration shown in FIGS. 13 and 14 may have components added or deleted as necessary.

[0042] ○Hardware Configuration of the Shooting Device○ First, with reference to FIG. 13, the hardware configuration of the shooting device 10 will be described. FIG. 13 is a diagram showing an example of the hardware configuration of the shooting device. Hereinafter, the shooting device 10 is an omnidirectional (all-round) shooting device using two image sensors, but the number of image sensors may be two or more. Also, it is not necessarily a device dedicated to omnidirectional shooting, and by attaching an after-market omnidirectional shooting unit to an ordinary digital camera, smartphone, etc., it may be made to have substantially the same function as the shooting device 10.

[0043] As shown in FIG. 13, the imaging device 10 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, an audio processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F (Interface) 116, a short-distance communication circuit 117, an antenna 117a of the short-distance communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0044] Among these, the imaging unit 101 includes wide-angle lenses 102a and 102b (hereinafter referred to as lens 102 when there is no need to distinguish) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each lens. The imaging elements 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors that convert the optical image by the lenses 102a and 102b into image data of an electrical signal and output it, a timing generation circuit that generates a horizontal or vertical synchronization signal, a pixel clock, etc. of this image sensor, and a register group in which various commands or parameters necessary for the operation of this imaging element are set, etc.

[0045] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the audio processing unit 109 are connected to the CPU 111 via the bus 110. Further, the ROM 112, SRAM 113, DRAM 114, operation unit 115, input / output I / F 116, short-range communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, network I / F 121, etc. are also connected to the bus 110.

[0046] The image processing unit 104 takes in the image data output from the imaging elements 103a and 103b through the parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes these pieces of image data to create data of an orthographic cylindrical projection image as shown in FIG. 3(C).

[0047] The imaging control unit 105 generally uses the I2C bus to set commands, etc. in the register groups of the imaging elements 103a and 103b, with the imaging control unit 105 as the master device and the imaging elements 103a and 103b as slave devices. The necessary commands, etc. are received from the CPU 111. Also, the imaging control unit 105 takes in status data, etc. of the register groups of the imaging elements 103a and 103b using the same I2C bus and sends it to the CPU 111.

[0048] Also, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Depending on the imaging device 10, it may have a preview display function or a function corresponding to video display using a display (for example, the display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the output of the image data from the imaging elements 103a and 103b is continuously performed at a predetermined frame rate (frames / minute).

[0049] In addition, as will be described later, the imaging control unit 105 also functions as synchronization control means for synchronizing the output timings of the image data of the imaging elements 103a and 103b in cooperation with the CPU 111. Note that in this embodiment, the imaging apparatus 10 is not provided with a display unit (display), but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 captures the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0050] The CPU 111 controls the overall operation of the imaging apparatus 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data during processing, and the like. In particular, the DRAM 114 stores image data during processing in the image processing unit 104 and data of the processed orthographic cylindrical projection image.

[0051] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, and a touch panel having both display and operation functions. The user inputs various shooting modes, shooting conditions, and the like by operating the operation unit 115.

[0052] The input / output I / F 116 is a general term for interface circuits (such as USB I / F) with external media such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the orthographic cylindrical projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116 or transmitted to an external terminal (apparatus) via the input / output I / F 116 as necessary.

[0053] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided in the imaging device 10 by short-range wireless communication technologies such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit the data of the orthographic cylindrical projection image to the external terminal (device).

[0054] The electronic compass 118 calculates the orientation of the imaging device 10 from the earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) conforming to Exif and is used for image processing such as image correction of the captured image. Note that the related information also includes each data such as the shooting date and time of the image and the data capacity of the image data. Also, the gyro sensor 119 is a sensor that detects changes in angles (Roll angle, Pitch angle, Yaw angle) accompanying the movement of the imaging device 10. The change in angle is an example of related information (metadata) conforming to Exif and is used for image processing such as image correction of the captured image. Furthermore, the acceleration sensor 120 is a sensor that detects accelerations in three axial directions. The imaging device 10 calculates the posture (angle with respect to the gravity direction) of its own device (imaging device 10) based on the accelerations detected by the acceleration sensor 120. By providing the acceleration sensor 120, the imaging device 10 improves the accuracy of image correction. The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like.

[0055] ○Hardware Configuration of Image Processing Device○ First, the hardware configuration of the image processing apparatus 50 will be described with reference to FIG. 14. FIG. 14 is a diagram showing an example of the hardware configuration of the image processing apparatus. Each hardware component of the image processing apparatus 50 is denoted by a reference numeral in the 500s. The image processing apparatus 50 is constructed by a computer and includes, as shown in FIG. 14, a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0056] Among these, the CPU 501 controls the overall operation of the image processing apparatus 50. The ROM 502 stores programs used for driving the CPU 501 such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to and from the HD 504 in accordance with the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images. Note that the display 506 may be a touch panel display having an input means. The external device connection I / F 508 is an interface for connecting various external devices. External devices in this case are, for example, a USB memory or the like. The network I / F 509 is an interface for performing data communication using the communication network 100. The bus line 510 is an address bus or a data bus or the like for electrically connecting the components such as the CPU 501 shown in FIG. 14.

[0057] Also, the keyboard 511 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, or moving a cursor, etc. Note that the input means may be not only the keyboard 511 and the pointing device 512, but also a touch panel or a voice input device, etc. The DVD-RW drive 514 controls reading or writing of various data with respect to the DVD-RW 513 as an example of a removable recording medium. Note that the removable recording medium is not limited to the DVD-RW, and may be a DVD-R or a Blu-ray (registered trademark) Disc (Blu-ray disc), etc. The media I / F 516 controls reading or writing (storage) of data with respect to the recording medium 515 such as a flash memory.

[0058] ○ Hardware Configuration of Communication Terminal ○ FIG. 14 also shows an example of the hardware configuration of the communication terminal 90. Each hardware configuration of the communication terminal 90 is indicated by a symbol in the 900 series in parentheses. The communication terminal 90 is constructed by a computer and has the same configuration as the image processing apparatus 50 as shown in FIG. 14, and thus the description of each hardware configuration is omitted. In addition to the same configuration as the image processing apparatus 50, the communication terminal 90 includes a short-range communication circuit 917 and an antenna 917a of the short-range communication circuit 917. The short-range communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0059] Note that each of the above programs may be in an installable or executable format file and may be recorded on a computer-readable recording medium and distributed. Examples of the recording medium include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, USB memory, etc. Also, the recording medium can be provided as a program product, either domestically or abroad. For example, the image processing apparatus 50 realizes the image processing method according to the present invention when the program according to the present invention is executed.

[0060] ●Functional Configuration Subsequently, with reference to FIG. 15, the functional configuration of the image processing system according to the embodiment will be described. FIG. 15 is a diagram showing an example of the functional configuration of the image processing system. Note that FIG. 15 shows those of the devices or terminals shown in FIG. 1 that are related to the processing or operations described later.

[0061] ○Functional Configuration of the Imaging Device○ First, with reference to FIG. 15, the functional configuration of the imaging device 10 will be described. The imaging device 10 includes a transmission / reception unit 11, an operation reception unit 12, a shooting control unit 13, a moving image shooting unit 14, a still image shooting unit 15, a movement detection unit 16, a communication unit 17, and a storage / readout unit 19. Each of these units is a function or means realized by any of the components shown in FIG. 13 operating according to an instruction from the CPU 111 based on a program for the imaging device developed from the SRAM 113 onto the DRAM 114. Also, the imaging device 10 has a storage unit 1000 constructed by the ROM 112, SRAM 113, and DRAM 114 shown in FIG. 13. The storage unit 1000 stores the GUID (Globally Unique Identifier) of the own device.

[0062] The transmission / reception unit 11 is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. Further, the transmission / reception unit 11 performs data communication via the communication network 100 with other devices or terminals using the network I / F 121.

[0063] The operation reception unit 12 is mainly realized by the processing of the CPU 111 with respect to the operation unit 115, and receives various selections or inputs from the user who is the photographer.

[0064] The shooting control unit 13 is mainly realized by the processing of the CPU 111 with respect to the imaging unit 101, the image processing unit 104, and the imaging control unit 105, captures a subject such as a landscape, and acquires captured image data. The shooting control unit 13 performs shooting by switching, for example, between video shooting by the video shooting unit 14 and still image shooting by the still image shooting unit 15 in a time-division manner.

[0065] The video shooting unit 14 is mainly realized by the processing of the CPU 111 with respect to the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and performs video shooting by the imaging device 10. The video shooting unit 14 performs shooting of a moving image, for example, while moving inside a structure such as a real estate property which is a predetermined base point. The video shooting unit 14 performs video shooting in low-resolution continuous frames while the photographer holding the imaging device 10 is moving, and stores the captured image data in the storage unit 1000. The video shooting unit 14 performs video shooting, for example, when the photographer holding the imaging device 10 is moving from a first point to a second point inside a real estate property which is a predetermined base point.

[0066] The still image capturing unit 15 is mainly realized by the processing of the imaging unit 101, captures a subject such as a landscape, and performs still image capturing by the imaging device 10. The still image capturing unit 15 performs capturing of a plurality of still images captured at different shooting positions within a structure such as a real estate department, which is a predetermined base point, for example. The still image capturing unit 15 performs capturing of a still image (photo) with a higher resolution than the moving image captured by the moving image capturing unit 14, for example, and stores the captured image data in the storage unit 1000. The still image captured by the still image capturing unit 15 may be an image of one frame, or may be an HDR (High Dynamic Range) image synthesized from a plurality of images.

[0067] Here, the still image captured by the still image capturing unit 15 is preferably, for example, a high resolution of 4K resolution or higher. On the other hand, the moving image captured by the moving image capturing unit 14 is an image used for position estimation, and since it is only necessary that the subject shown in the moving image can be identified, it is an image with a lower resolution than the still image. The moving image may be, for example, a resolution of about 480p or lower. By capturing a low-resolution moving image, the imaging device 10 can reduce the overall data volume in tour shooting.

[0068] The movement detection unit 16 is mainly realized by the processing of the CPU 111 with respect to the gyro sensor 119 and the acceleration sensor 120, and detects the movement state of the imaging device 10. The movement detection unit 16 detects, for example, whether the photographer holding the imaging device 10 is moving (in a moving state) or stationary (in a stationary state) during shooting by the imaging device 10.

[0069] The communication unit 17 is mainly realized by the processing of the CPU 111 with respect to the input / output I / F 116 or the short-range communication circuit 117, and transmits and receives various data or information to and from other devices or terminals. The communication unit 17 performs data communication via various cables and the like with the communication terminal 90 using the input / output I / F 116, for example. In addition, the communication unit 17 performs data communication with the communication terminal 90 using the short-range communication circuit 117 by short-range wireless communication technology.

[0070] The memory / reading unit 19 is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 or reads out various data (or information) from the storage unit 1000. Also, the image data captured by the video shooting unit 14 and the still image shooting unit 15 is stored in the storage unit 1000. The shooting time of the captured image is associated with the image data stored in the storage unit 1000 as metadata.

[0071] ○Functional Configuration of the Image Processing Apparatus○ First, with reference to FIG. 15, the functional configuration of the image processing apparatus 50 will be described. The image processing apparatus 50 includes a transmission / reception unit 51, a reception unit 52, a determination unit 53, an application determination unit 54, a shape estimation unit 55, a position estimation unit 56, a detection unit 57, a path generation unit 58, an image processing unit 60, and a memory / reading unit 59. Each of these units is a function or means realized by any of the components shown in FIG. 14 operating according to instructions from the CPU 501 in accordance with the image processing apparatus program expanded from the HD 504 onto the RAM 503. Also, the image processing apparatus 50 has a storage unit 5000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in FIG. 14.

[0072] The transmission / reception unit 51 is mainly realized by the processing of the CPU 501 with respect to the network I / F 509, and transmits and receives various data or information to and from other devices or terminals via the communication network 100. The transmission / reception unit 51 receives (acquires), for example, a moving image captured by the imaging device 10 from the imaging device 10 or the communication terminal 90. Also, the transmission / reception unit 51 receives (acquires), for example, a still image captured by the imaging device 10 from the imaging device 10 or the communication terminal 90.

[0073] The reception unit 52 is mainly realized by the processing of the CPU 501 with respect to the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user. The determination unit 53 is realized by the processing of the CPU 501 and makes various determinations.

[0074] The use determination unit 54 is realized by the processing of the CPU 501, and determines the use of the space shown in the captured image based on the captured image in which the internal space of the structure is shown in all directions. In other words, the use is the classification, genre, or purpose of use, etc. of the space actually captured by the imaging device 10. When a real estate property is captured as a base point by the imaging device 10, the use of the room, which is the space shown in the captured image, is, for example, a toilet, a bathroom, or an entrance, etc.

[0075] The shape estimation unit 55 is realized by the processing of the CPU 501, and estimates the shape of the space shown in the captured image based on the captured image in which the internal space of the structure is shown in all directions.

[0076] The position estimation unit 56 is realized by the processing of the CPU 501, and estimates the relative imaging position of the still image acquired from the imaging device 10 based on the moving image acquired from the imaging device 10. The position estimation unit 56 estimates, for example, the relative position of the still image in which the still image was captured from the moving image, which is a low-resolution continuous frame, using a method such as visual SLAM or SfM. These methods such as visual SLAM or SfM can calculate the positions of feature points on the image, the position of the camera, and parameters from a plurality of images. On the other hand, the calculated position is relative, and a reference position is required to obtain an absolute position.

[0077] The detection unit 57 is realized by the processing of the CPU 501, and detects the imaging points of the moving image belonging to the space shape estimated by the shape estimation unit 55. The imaging points of the moving image are, for example, coordinate values in which the moving path at the time of capturing the moving image is restored.

[0078] The path generation unit 58 is realized by the processing of the CPU 501, and generates a path indicating the connection relationship of a plurality of still images based on the imaging position estimated by the position estimation unit 56. The path generation unit 58 generates, for example, a path that associates the nearest still images among the plurality of still images based on the estimated imaging position.

[0079] The image processing unit 60 is realized by the processing of the CPU 501, and performs image processing for generating a tour image for a virtual tour based on the shooting position estimated by the position estimation unit 56. For example, the image processing unit 60 generates a tour image, which is a processed image including a plurality of still images associated based on the estimated shooting position. The image processing unit 60 determines the position where the still images in the tour image should be arranged from the estimated shooting position of the still images, the shooting time of the still images, and the path (tour path) generated by the path generation unit 58.

[0080] The storage / reading unit 59 is mainly realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000. Further, the storage unit 5000 stores the tour image created by the image processing unit 60.

[0081] ○Functional Configuration of Communication Terminal○ Next, the functional configuration of the communication terminal 90 will be described with reference to FIG. 15. The communication terminal 90 includes a transmission / reception unit 91, a reception unit 92, a display control unit 93, a communication unit 94, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in FIG. 15 operating according to an instruction from the CPU 901 in accordance with a communication terminal program expanded from the HD 904 onto the RAM 903. Further, the communication terminal 90 has a storage unit 9000 constructed by the ROM 902, the RAM 903, and the HD 904 shown in FIG. 15.

[0082] The transmission / reception unit 91 is mainly realized by the processing of the CPU 901 with respect to the network I / F 909, and transmits and receives various data or information to and from other devices or terminals via the communication network 100.

[0083] The reception unit 92 is mainly realized by the processing of the CPU 901 with respect to the keyboard 911 or the pointing device 912, and receives various selections or inputs from the user.

[0084] The display control unit 93 is mainly realized by the processing of the CPU 901, and causes the display 906 to display various images, characters, etc. The display control unit 93 accesses the image processing apparatus 50, for example, using a web browser or a dedicated application, and causes the display 906 to display an image corresponding to the data distributed from the image processing apparatus 50.

[0085] The communication unit 94 is mainly realized by the processing of the CPU 901 with respect to the external device connection I / F 908 or the short-range communication circuit 917, and transmits and receives various data or information to and from other devices or terminals. The communication unit 94 performs data communication via various cables, for example, with the imaging device 10 using the external device connection I / F 908. The communication unit 94 also performs data communication using the short-range communication circuit 917 with the imaging device 10 by short-range wireless communication technology.

[0086] The storage / reading unit 99 is mainly realized by the processing of the CPU 901, and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000.

[0087] ●Processing or operation of the embodiment ○Tour shooting processing○ Subsequently, with reference to FIGS. 16 to 29, the processing or operation of the image processing system according to the embodiment will be described. In the following description, as an example of the space inside a structure that is a predetermined base, an example of a room of a real estate property is shown. First, with reference to FIGS. 16 to 20, the acquisition process of the captured image used for generating the tour image by the image processing apparatus 50 will be described. FIG. 16 is a flowchart showing an example of the tour shooting process by the imaging device.

[0088] First, in response to a predetermined request from the communication terminal 90, the imaging device 10 starts a tour shooting process (step S11). Here, tour shooting means shooting within a predetermined base for providing a virtual tour. Specifically, the communication terminal 90 executes the tour shooting function by starting and running a dedicated installed application or accessing a predetermined website using a web browser. Then, the communication unit 94 of the communication terminal 90 transmits a tour shooting request to the imaging device 10. The imaging device 10 starts the tour shooting process in response to the tour shooting request received by the communication unit 17.

[0089] Next, the video shooting unit 14 of the imaging device 10 starts shooting a video while holding the imaging device 10 and moving within the base (step S12). Specifically, when the tour shooting is started, the shooting control unit 13 of the imaging device 10 requests the video shooting unit 14 to start shooting a video. Then, the video shooting unit 14 starts shooting a moving image in response to the request from the shooting control unit 13.

[0090] Here, with reference to FIGS. 17 and 18, the shooting method by the imaging device 10 will be described. FIG. 17 is a diagram showing an example of a fixing method of the imaging device. The imaging device 10 is preferably fixed to a support member 20 such as a monopod 20a as shown in FIG. 17(A) or a tripod 20b as shown in FIG. 17(B) so that the photographer is not captured. Also, the imaging device 10 is fixed at a height desired during image viewing. The imaging device 10 is preferably arranged at the height of a person's line of sight, for example, to show a real estate property naturally.

[0091] FIG. 18 is a diagram for explaining the differences between hand-held shooting and fixed shooting. FIG. 18 shows the physical movement differences when the photographer holds the imaging device 10 to take a picture (hand-held shooting) and when the imaging device 10 is fixed with a monopod 20a and then shot (fixed shooting). As shown in FIG. 17, when shooting while moving, the simplest method is to move the imaging device 10 while it is attached to a support member 20 such as a monopod 20a. However, in the cases of hand-held shooting and fixed shooting, there are differences in physical movement. Specifically, for example, by attaching a relatively long and large monopod 20a, the physical moment becomes large and the movement of the imaging device itself is attenuated.

[0092] In Patent Document 1, the shooting position is estimated by position estimation using a sensor such as an IMU and PDR. However, when the monopod 20a is attached, as shown in FIG. 18, since the variations in acceleration and angular velocity are attenuated, there is a problem that sufficient accuracy cannot be obtained with the shooting position estimation method described in Patent Document 1. Therefore, even when the imaging device 10 is fixed to a support member 20 such as a monopod 20a so that the photographer is not captured, the image processing system 1 adopts an image processing method described later so as not to affect the processing accuracy.

[0093] Returning to FIG. 16, when the imaging device 10 detects a switching of the shooting mode (YES in step S13), the process proceeds to step S14. In this case, the imaging device 10 switches from the video shooting mode to the still image shooting mode. The imaging device 10 switches from the video shooting mode to the still image shooting mode, for example, when the shutter button of the operation unit 115 of the imaging device 10 is pressed. The imaging device 10 detects the switching of the shooting mode. Further, the imaging device 10 may be configured to switch from the video shooting mode to the still image shooting mode when it is detected by the movement detection unit 16 that the device has stopped (stationary state). On the other hand, when the imaging device 10 does not detect a switching of the shooting mode (NO in step S13), the imaging unit 14 continues the video shooting in step S12.

[0094] Next, in response to detecting the switching of the shooting mode in step S13, the moving image shooting unit 14 of the imaging device 10 stops shooting the moving image (step S14). Then, the still image shooting unit 15 takes a still image in response to the user pressing the shutter button of the operation unit 115 (step S15).

[0095] Next, in the same manner as the process of step S13, when the imaging device 10 detects the switching of the shooting mode (YES in step S16), the process proceeds to step S18. In this case, the imaging device 10 switches from the still image shooting mode to the moving image shooting mode. For example, when the movement restart is detected by the movement detection unit 16, the imaging device 10 switches from the still image shooting mode to the moving image shooting mode. Further, the imaging device 10 may be configured to automatically switch from the still image shooting mode to the moving image shooting mode after the still image shooting unit 15 takes a still image.

[0096] In response to detecting the switching of the shooting mode in step S16, the moving image shooting unit 14 resumes shooting the moving image (step S17). Similar to step S12, the moving image shooting unit 14 holds the imaging device 10 and continues shooting while moving within the base. Then, when the tour shooting is completed (YES in step S18), the imaging device 10 ends the process. The end of the tour shooting is determined, for example, by a predetermined operation on the imaging device 10 or the communication terminal 90 by the photographer. On the other hand, when the tour shooting has not ended (NO in step S18), the imaging device 10 repeats the process from step S13 and continues the tour shooting.

[0097] On the other hand, in step S16, when the imaging device 10 does not detect the switching of the shooting mode (NO in step S16), the process proceeds to step S19. When the tour shooting is completed (YES in step S19) in the same manner as step S18, the imaging device 10 ends the process. On the other hand, when the tour shooting has not ended (NO in step S19), the imaging device 10 repeats the process from step S19 and continues the still image shooting by the still image shooting unit 15.

[0098] Here, with reference to FIGS. 19 and 20, video shooting and still image shooting in the imaging device 10 will be described. FIG. 19 is a diagram for explaining the switching between video shooting and still image shooting by the imaging device. As described above, the imaging device 10 is used for two purposes: video shooting and still image shooting. Among them, video shooting is performed for the purpose of position estimation, and still image shooting is performed for the purpose of acquiring an image for browsing. Also, video shooting is performed during movement, and still image shooting is performed while stationary.

[0099] Here, the shooting specifications required for video shooting and still image shooting differ according to their respective shooting purposes. In the case of video shooting, continuous images (continuous frames) with a high frame rate are acquired for use in position estimation. On the other hand, a high resolution and color layer of the captured image are not necessary, and a low resolution and grayscale are sufficient. Also, in the case of still image shooting, acquisition of continuous frames is not necessary. On the other hand, shooting with high resolution, color information (RGB), and a high dynamic range is required. Furthermore, video shooting is performed during movement, and still image shooting is performed while stationary.

[0100] FIG. 20 is a diagram showing an example of the switching timing between video shooting and still image shooting. FIG. 20 shows a time chart of the switching between video shooting and still image shooting in the process shown in FIG. 16.

[0101] The imaging device 10 switches between video shooting and still image shooting in a time-division manner. As described above, video shooting is performed to acquire an image for position estimation, so it is necessary to perform continuous frame shooting. On the other hand, still image shooting is performed to acquire an image for browsing a virtual tour, so continuous frame shooting is not necessary, and a configuration in which one image is acquired may be sufficient, but an image obtained by high dynamic range synthesis using a plurality of frame images may also be acquired.

[0102] Still image shooting is an item in which the position of the photographer should be reflected, and it is assumed that shooting is performed by some explicit action of the photographer. For example, still image shooting is performed by switching from video shooting when the shutter button of the operation unit 115 of the imaging device 10 is pressed. In this case, the shutter pressed by the operator may be the shutter button on the main body of the imaging device 10, or may be a shutter based on the operation of a remotely operable application or controller. For example, when shooting a real estate property, it is preferable that the photographer does not appear in the captured image. Therefore, when performing an operation on the shutter button of the main body of the imaging device 10, it is preferable that the imaging device 10 captures and stores a still image after a predetermined time has elapsed after the shutter button is pressed. Thereby, after pressing the shutter button, the photographer can move outside the shooting range so as not to appear in the captured image. Also, when performing remote operation, it is preferable that the photographer performs the operation from outside the shooting range to take the shot.

[0103] Also, as a method of switching from video shooting to still image shooting, when the imaging device 10 fixed to the support member 20 is placed at a desired position or when the photographer himself / herself stops, the movement detection unit 16 detects the stationary state, and thus it may be configured to switch to still image shooting, or it may be configured to automatically switch to still image shooting when a predetermined time has elapsed since the start of video shooting. In the case of these methods, since a shutter operation by the operator is not required, there is an advantage that tour shooting can be performed smoothly.

[0104] On the other hand, since video shooting is an item that is shot regardless of the intention of the photographer, it is preferable that shooting is automatically performed. For example, video shooting is performed by automatically switching from still image shooting after still image shooting is performed. Also, in order to suppress the data amount of the captured image, video shooting may be configured to be switched from still image shooting when the movement restart is detected by the movement detection unit 16.

[0105] In this way, by switching between video shooting for capturing low-resolution moving images for position estimation and still image shooting for capturing high-resolution still images for viewing in a virtual tour in a time series, the imaging device 10 can efficiently acquire the captured images for generating tour images. In the above description, a configuration in which video shooting is performed at the start of tour shooting has been described. However, the imaging device 10 may be configured to first perform still image shooting at the start of tour shooting.

[0106] The imaging device 10 uploads the captured moving images and still images to the image processing device 50 as needed via the communication terminal 90. The image processing device 50 temporarily stores the moving images and still images transmitted from the imaging device 10 in the storage unit 5000 for generating tour images. Note that the imaging device 10 may be configured to directly upload the moving images and still images to the image processing device 50 without going through the communication terminal 90.

[0107] ○Tour Image Generation Process○ Subsequently, with reference to FIGS. 21 to 29, the generation process of tour images using the captured images captured by the imaging device 10 will be described.

[0108] FIG. 21 is a diagram for explaining an example of a tour path in a virtual tour. The tour path is preferably generated to be the same as the adjacency relationship of rooms that are the internal spaces of actual real estate properties. For example, when room A and room B are adjacent and can be accessed through a door, the still images captured in room A and room B should be connected by a path. On the other hand, when room A and room C are adjacent but cannot be accessed through a door, the still images captured in room A and room C should not be connected by a path. Therefore, when automatically generating a virtual tour, it is necessary to automatically determine which captured images should be connected by a path using the shooting positions of the still images and the paths between them.

[0109] As shown in FIG. 21, the tour path is basically generated by connecting the still images in the order in which they were taken. For example, in the case of the example shown in FIG. 21(A), the paths are connected in the order of shooting to generate a path of "Room A ⇔ Room B ⇔ Room C". On the other hand, in the case of the example shown in FIG. 21(B), since Room B and Room C are not physically connected even though the shooting order is continuous, in the actual space, it is necessary to return to Room A, which was the original room, and then enter Room C. In such a case, simply connecting the paths in the shooting order will connect rooms that cannot actually be accessed back and forth with a path, resulting in a tour different from reality. Therefore, as shown in FIG. 21(B), considering the structure of the actual space, a path connecting Room B and Room C is generated by a path passing through Room A again.

[0110] In this way, the image processing apparatus 50 generates a path by an algorithm that takes into account which room each shooting point on the moving path at the time of shooting is located in, so that the paths can be correctly connected in the generated tour image. Hereinafter, the algorithm for automatically generating a path in the image processing apparatus 50 will be described in detail.

[0111] FIG. 22 is a flowchart showing an example of the generation process of a tour image by an image processing apparatus. The image processing apparatus 50 executes the generation process of the tour image using the moving image and the still image, which are the captured images captured by the imaging apparatus 10. Note that the image processing apparatus 50 will be described as having acquired in advance the moving image and the still image captured by the imaging apparatus 10. At this time, the transmission / reception unit 51 of the image processing apparatus 50 receives (acquires) the moving image and the still image transmitted from the imaging apparatus 10. Also, in the example shown in FIGS. 23 to 27, it is assumed that the still images are images in which Room A, Room B, and Room C are respectively captured at the shooting positions ( "1 to 3" in each drawing) in Room A, Room B, and Room C.

[0112] First, the usage determination unit 54 of the image processing apparatus 50 determines the usage of the room shown in the acquired moving image (step S31). Specifically, the usage determination unit 54 determines the usage (genre) of the room shown in the moving image using parameters such as general machine learning or feature amounts of the moving image. Here, the determined usage of the room is, for example, a toilet, a bathroom, or an entrance hall.

[0113] Next, the shape estimation unit 55 estimates the shape of the room shown in the acquired moving image (step S32). Specifically, the shape estimation unit 55 detects the straight lines of the subjects shown in the moving image, obtains the vanishing points of the detected straight lines, and estimates the structure of the room from the boundaries such as the floor, walls, or ceiling, a method that uses the three-dimensional information acquired from the imaging device 10, or a method that uses general machine learning, etc., to estimate the shape of the room shown in the moving image, performs three-dimensional restoration, and obtains the bird's-eye view of the room shape.

[0114] Also, the shape estimation unit 55 estimates the size of the room shape whose structure was estimated in step S32 (step S33). Specifically, the shape estimation unit 55 adjusts the size of the estimated room shape to match the coordinate system of each point of the moving image on the movement path. As shown in Fig. 23(A), when the room shape estimated in step S32 is restored from a single image, the absolute size is unknown. As shown in Fig. 23(B), in order to arrange each room shape with the correct size on the movement path during video shooting, the shape estimation unit 55 determines the size of each room shape in the coordinate system of each point of the moving image on the movement path during video shooting. The shape estimation unit 55 estimates the size of each room shape using the average value of the shooting height during video shooting and the size such that the gap between adjacent room shapes is 0. In this way, by estimating the shape and size of the room shown in the moving image by the shape estimation unit 55, the position estimation unit 56 estimates the relative shooting positions of the moving image and the plurality of still images shot in time division.

[0115] Next, the detection unit 57 determines in which of the room shapes estimated in step S32 and step S33 each point of the moving image on the movement path is located (step S34). The detection unit 57 detects from the coordinate values in which room among the room shapes arranged at each point on the movement path in step S33 the point is located inside. Thereby, the detection unit 57 detects in which room each point on the movement path was located. The image processing apparatus 50 uniquely determines in which room each point is located based on the coordinate values on the movement path at the time of shooting of the restored moving image, for example, as shown in FIG. 24(B). In the example shown in FIG. 24(B), it is determined that the points on the path R1 are located inside room A, the points on the path R2 are located inside room B, and the points on the path R3 are located inside room C.

[0116] Also, when the detection unit 57 detects that the points of the moving image on the movement path do not exist within a single room shape (NO in step S34), the process proceeds to step S35. When the points on the movement path do not belong to a single room, that is, when the points on the movement path are located within a plurality of room shapes (room A and room B in the example of FIG. 25), as in the section S1 shown in FIG. 25(A), it cannot be uniquely determined in which room the point was located at the time of shooting. Also, when the points on the movement path are not located within any room shape, as in the section S2 shown in FIG. 25(A), it cannot be uniquely determined in which room the point was located at the time of shooting. Such a situation occurs due to errors in the restoration of the movement path, errors in the estimation of the room shape, and errors in the size of the room shape.

[0117] Therefore, when the points of the moving image on the movement path do not exist within a single room shape, the detection unit 57 determines the room in which the point closest to the target point on the movement path is located as the room in which the target point exists in order to uniquely determine in which room shape the target point exists (step S35). The detection unit 57 calculates the nearest points on the movement path for the points in the sections S1 and S2 of FIG. 25(A), and determines the rooms in which the nearest points on the movement path are located, as in the sections M1 and M2 of FIG. 25(B), as the rooms in which the target points are located.

[0118] On the other hand, in step S34, when the detection unit 57 detects that each point of the moving image on the movement path exists within a single room shape (YES in step S34), the process proceeds to step S36.

[0119] Next, the path generation unit 58 generates a path connecting the room where the point of the moving image on the movement path exists and the room where the next point exists (step S36). Since it is uniquely determined for each point on the movement path which room shape each point on the movement path is located in by the processes of steps S34 and S35, for each point on the movement path, a path is connected with the room where one point exists and the room where the next point exists as the next room. In the example of FIG. 26, the path generation unit 58 connects the room B entered next after the room A with a path (N1). Also, regarding the movement of once returning from the room B to the room A and then entering the room C, since there are sections (N2 and N3) of the room A between the shooting position 2 and the shooting position 3, the path generation unit 58 can correctly connect the path from the room A to the room C based on the estimation results in steps S32 and S33. Finally, the path generation unit 58 can generate a path of the route "R1⇔R2⇔R1⇔R3" as shown in FIG. 26. That is, a path of "room A⇔room B" and a path of "room A⇔room C" are generated. That is, by generating a path connecting each room, the path generation unit 58 can associate the nearest still image among the plurality of still images in which each room is photographed.

[0120] Next, when the determination unit 53 determines that the use of the next room determined in step S31 is a toilet or a bathroom (YES in step S37), the process proceeds to step S38. On the other hand, when the determination unit 53 determines that the use of the next room determined in step S31 is not a toilet or a bathroom (NO in step S37), the process proceeds to step S39.

[0121] Rooms leading to the toilet or bathroom utilize the fact that in most houses, there is only one room, i.e., the return room, to prevent incorrect paths from being connected. For example, as shown in Fig. 27(A), when there are many overlapping regions due to the error of the above-mentioned process, a path connecting room A to room B through the toilet or bathroom may be generated. In the example of Fig. 27(A), based on the determination according to the points on the nearest movement path by step S35, the room where the points in the overlapping section J1 exist is determined to be room B, and a path connecting the toilet or bathroom is generated. However, in a real house, there are almost no cases where the toilet or bathroom leads to two different rooms.

[0122] Therefore, when the use of the next room is the toilet or bathroom, the determination unit 53 sets the current room as the room adjacent to the next room (step S38). For example, when the use of the room where the next point on a certain point on the movement path is located is the toilet or bathroom, the determination unit 53 stores the room where the certain point exists as the room adjacent to the toilet or bathroom.

[0123] Next, when the use of the room immediately before the one determined in step S31 is the toilet or bathroom (YES in step S39), the determination unit 53 shifts the process to step S40. On the other hand, when the use of the room immediately before the one determined in step S31 is not the toilet or bathroom (NO in step S39), the determination unit 53 shifts the process to step S41. When the room immediately before a certain point on the movement path is the toilet or bathroom, that point means leaving the toilet or bathroom. For example, the points in section J1 of Fig. 27(A) indicate the points when leaving the toilet or bathroom.

[0124] When the use of the previous room is a toilet or a bathroom, the path generation unit 58 generates a path to the room adjacent to the previous room (step S40). When the room where the point immediately before a certain point on the movement path exists is a toilet or a bathroom, the path generation unit 58 connects the room where the certain point exists and the adjacent room set in step S38 with a path. Thereby, the room where one exits from the toilet or the bathroom can be made the same as the room where one was before entering the toilet or the bathroom set in step S38. As shown in FIG. 27(B), since the room A where one was immediately before entering the toilet or the bathroom is set in step S38, the path generation unit 58 connects, with a path, not the nearest room B but the room A where one was immediately before entering the toilet or the bathroom as a point on the movement path at the time of exiting (section J2). Thereby, when the image processing apparatus 50 exits from the toilet or the bathroom, it can be forced to return to the room where it entered, and it is possible to prevent the generation of an incorrect path as shown in FIG. 27(A).

[0125] Next, when all the processes on the moving image captured by the imaging device 10 are completed (YES in step S41), the image processing apparatus 50 shifts the process to step S42. On the other hand, the image processing apparatus 50 repeats the process from step S31 until all the processes on the moving image captured by the imaging device 10 are completed (NO in step S41).

[0126] Next, the path generation unit 58 sets the start point of the generated path based on the use of the room determined in step S31 (step S42). The path generation unit 58 sets, for example, the position corresponding to the room whose determined use is an entrance as the start point of the path.

[0127] Then, based on the above-described processing results, the image processing unit 60 generates a tour image, which is a processed image including a plurality of still images captured by the imaging device 10 (step S42). Specifically, as shown in FIG. 28, the image processing unit 60 generates a tour image in which a plurality of still images are associated by associating the shooting positions estimated by the position estimation unit 56 on the shape estimation result by the shape estimation unit 55. Further, as shown in FIG. 29(A), in addition to the configuration shown in FIG. 28, the image processing unit 60 superimposes the path generated by the path generation unit 58 on a plurality of still images associated by the relative shooting positions. Furthermore, as shown in FIG. 29(B), the image processing unit 60 can indicate the start point (S) of the path set in step S42 on the tour image.

[0128] In this way, the image processing apparatus 50 estimates the shooting positions of high-resolution still images based on the movement path in the moving image captured by the imaging device 10. Further, the image processing apparatus 50 generates a path indicating in what order the shooting was performed based on the estimated shooting positions, the shape of the room shown in the captured images, and the movement path of the imaging device 10.

[0129] The tour image generated by the image processing apparatus 50 can be viewed by the viewer using the communication terminal 90. When the generation of the tour image is completed, the image processing apparatus 50 notifies the communication terminal 90, which is the requester of the tour shooting. The display control unit 93 of the communication terminal 90 causes the tour image provided from the image processing apparatus 50 to be displayed on the display 906. The viewer can view a virtual tour of a real estate property or the like using the tour image provided from the image processing apparatus 50 using the communication terminal 90.

[0130] ● Effects of the Embodiment As described above, the image processing system 1 can improve the estimation accuracy of the shooting position of a captured image by accurately associating a high-resolution still image used for creating a virtual tour with the shooting positions of a plurality of still images estimated using the moving image captured by the imaging device 10. Then, the image processing system 1 can automatically create a virtual tour (high-quality virtual tour) with an appropriate path.

[0131] In addition, the image processing system 1 can efficiently acquire captured images for creating a virtual tour by switching between high-resolution still image shooting for acquiring viewing images and low-resolution moving image shooting for position estimation using the imaging device 10 in a time-division manner.

[0132] Furthermore, the image processing system 1 performs shooting using an omnidirectional shooting device that shoots the interior of a real estate property or the like, which is a predetermined base, all-round at once, and creates a natural virtual tour (high-quality virtual tour) by using natural images without errors due to parallax or the like during shooting for the virtual tour. Also, the image processing system 1 can create a virtual tour with improved position estimation accuracy even when shooting is performed by fixing the imaging device 10 to a support member 20 such as a monopod so that the photographer does not appear in the captured image.

[0133] ●Summary● As described above, the image processing method according to an embodiment of the present invention is an image processing method executed by an image processing apparatus 50 that performs processing on a captured image captured omnidirectionally within a predetermined base point. The image processing method includes a still image acquisition step of acquiring a plurality of still images captured at different shooting positions within the base point, a moving image acquisition step of acquiring a moving image captured while moving from a first point to a second point within the base point, a position estimation step of estimating the relative shooting positions of the plurality of still images based on the acquired moving image, and an image processing step of generating a processed image (for example, a tour image) including the plurality of still images associated based on the estimated shooting positions. Thereby, the image processing method can provide a virtual tour with improved estimation accuracy of the shooting positions of the captured images.

[0134] Further, the image processing method according to an embodiment of the present invention further includes a shape estimation step of estimating the shape of the space (for example, a room) within the base point shown in the moving image, and a path generation step of generating a path for associating the plurality of still images based on the shooting positions estimated by the position estimation step and the shape estimated by the shape estimation step. The position estimation step estimates the relative shooting positions of the plurality of still images based on the shape estimated by the shape estimation step, and the image processing step generates a processed image (for example, a tour image) including the paths corresponding to the plurality of still images. Thereby, the image processing method can accurately associate the plurality of still images with the shooting positions of each still image based on the shooting positions of the still images estimated using the moving image, the shape of the space shown in the captured image, and the generated path. Then, the image processing method can automatically create a virtual tour (high-quality virtual tour) with high position accuracy and an appropriate path (route).

[0135] Also, in the image processing method according to an embodiment of the present invention, the moving image is an image with a lower resolution than the still image. Further, in the image processing method, the still image and the moving image are images captured by an omnidirectional imaging device. Thereby, the image processing method can efficiently create a virtual tour (high-quality virtual tour) with an appropriate path.

[0136] Furthermore, an image processing system according to an embodiment of the present invention is an image processing system 1 including an image processing device 50 and an imaging device 10. The imaging device 10 captures a still image and a moving image by switching them in a time-division manner. Also, the imaging device 10 switches the capture of the still image and the moving image according to the moving state within the base. Thereby, the image processing system 1 can efficiently acquire a captured image for creating a virtual tour.

[0137] ●Supplementary● Each function of the above-described embodiment can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in the present embodiment refers to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and an ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (System on a chip), GPU (Graphics Processing Unit), and a device such as a conventional circuit module designed to execute each function described above.

[0138] Although the image processing method, program, image processing apparatus, and image processing system according to an embodiment of the present invention have been described so far, the present invention is not limited to the above-described embodiment, and can be changed within the scope that those skilled in the art can conceive, such as addition, change, or deletion of other embodiments. As long as the functions and effects of the present invention are achieved in any aspect, they are included in the scope of the present invention.

Explanation of Reference Numerals

[0139] 1 Image processing system 10 Photographing device 11 Transmission / reception unit 12 Operation reception unit 13 Photographing control unit (an example of photographing control means) 14 Moving image photographing unit 15 Still image photographing unit 16 Movement detection unit 20 Support member 50 Image processing apparatus 51 Transmission / reception unit (an example of still image acquisition means, an example of moving image acquisition means) 54 Usage determination unit 55 Shape estimation unit 56 Position estimation unit (an example of position estimation means) 57 Detection unit 58 Path generation unit (an example of path generation means) 60 Image processing unit (an example of image processing means) 90 Communication terminal 91 Transmission / reception unit 92 Reception unit 93 Display control unit (an example of display control means)

Claims

1. An image processing method executed by an image processing apparatus that performs processing on a captured image captured omnidirectionally within a predetermined base, comprising: A still image acquisition step of acquiring a plurality of still images captured at different shooting positions within the base; A moving image acquisition step of acquiring a moving image captured while moving from a first point to a second point within the base; A position estimation step of estimating the relative shooting positions of the plurality of still images based on the acquired moving image; An image processing step of generating a processed image including the plurality of still images associated based on the estimated shooting positions; An image processing method for executing the above.

2. The image processing method according to Claim 1, further comprising: Executing a path generation step of generating a path for associating the plurality of still images; The image processing step generates the processed image including the path corresponding to the plurality of still images.

3. The image processing method according to Claim 2, further comprising: Executing a shape estimation step of estimating the shape of the space within the base shown in the moving image; The position estimation step estimates the relative shooting positions of the plurality of still images based on the shape estimated by the shape estimation step; The path generation step generates the path based on the shooting position estimated by the position estimation step and the shape estimated by the shape estimation step.

4. The image processing method according to Claim 3, further comprising: Executing a detection step of detecting the shape to which the point on the moving path where the moving image is captured belongs; When a point where the moving image belonging to a plurality of the shapes is captured is detected, the path generation step generates the path so that the point belongs to the nearest shape.

5. The image processing method according to Claim 4, further comprising: Executing a use determination step of determining the use of the space; The path generation step generates the path based on the determined use.

6. The image processing method according to Claim 5, wherein the path generation step sets a start point of the generated path based on the determined use.

7. The image processing method according to any one of Claims 1 to 6, wherein the moving image is an image having a lower resolution than the still image.

8. The image processing method according to any one of claims 1 to 7, wherein the still image and the moving image are images captured by an omnidirectional imaging device.

9. A program for causing a computer to execute the image processing method according to any one of claims 1 to 8.

10. An image processing apparatus that performs processing on a captured image captured omnidirectionally within a predetermined base, Still image acquisition means for acquiring a plurality of still images captured at different imaging positions within the base; Moving image acquisition means for acquiring a moving image captured while moving from a first point to a second point within the base; Position estimation means for estimating the relative imaging positions of the plurality of still images based on the acquired moving image; Image processing means for generating a processed image including the plurality of still images associated based on the estimated imaging positions; An image processing apparatus comprising:

11. An image processing system comprising the image processing apparatus according to claim 10 and an imaging device, wherein the imaging device An image processing system comprising imaging control means for switching between the still image and the moving image and capturing them in a time-division manner.

12. The image processing system according to claim 11, wherein the imaging control means switches between capturing the still image and the moving image according to the moving state within the base.

13. The image processing system according to any one of claims 10 to 12, further comprising a communication terminal capable of communicating with the image processing apparatus, wherein the communication terminal An image processing system comprising display control means for displaying the processed image generated by the image processing apparatus.

14. The image processing system according to claim 13, wherein the image processing apparatus Comprises path generation means for generating a path for associating the plurality of still images based on the imaging positions estimated by the position estimation means, The image processing means generates the processed image including the path corresponding to the plurality of still images.

15. The image processing system according to claim 14, wherein the display control means displays the processed image in which the corresponding path is superimposed on the plurality of still images.

Images