Image processing method, program, image processing device, and image processing system

Abstract

The objective is to provide a virtual tour with improved accuracy in estimating the shooting location of captured images. [Solution] An image processing method performed by an image processing device 50 that processes images taken in all directions within a predetermined location, comprising: a still image acquisition step of acquiring multiple still images taken at different shooting positions within the location; a moving image acquisition step of acquiring moving images taken while moving from a first point to a second point within the location; a position estimation step of estimating the relative shooting positions of the multiple still images based on the acquired moving images; and an image processing step of generating a processed image (e.g., a tour image) including multiple still images associated based on the estimated shooting positions.

Description

Technical Field

[0001] The present disclosure relates to an image processing method, a program, an image processing apparatus, and an image processing system.

Background Art

[0002] There is known a system that distributes image data captured using a capturing device capable of capturing in all directions and enables viewing the situation at a remote site at another site. A panoramic image captured by capturing a predetermined site in all directions can allow a viewer to view in an arbitrary direction, and thus can convey information with a sense of presence. Such a system is used, for example, in fields such as online interior view of properties in the real estate industry.

[0003] Furthermore, there is a virtual tour service that connects captured images taken at a plurality of shooting positions of a real estate property, as if walking through the inside of the property. For creating such a virtual tour, it is important to connect the captured images taken at a plurality of shooting positions in a correct positional relationship. For creating such a virtual tour, a technique for estimating the position of a captured image using acceleration information measured by an IMU (inertial measurement unit) is already known (for example, Patent Document 1 and Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, conventional methods of position estimation using acceleration with an IMU had room for improvement in terms of the accuracy of position estimation for multiple captured images used to create a virtual tour. [Means for solving the problem]

[0006] To solve the above-mentioned problems, the invention according to claim 1 is an image processing method performed by an image processing device that processes captured images taken in all directions within a predetermined location, the method comprising: a still image acquisition step of acquiring a plurality of still images taken at different shooting positions within the location; a moving image acquisition step of acquiring moving images taken while moving from a first point to a second point within the location; an estimation step of estimating the relative shooting positions of the plurality of still images based on the acquired moving images; and an image processing step of generating a processed image including the plurality of still images associated based on the estimated shooting positions. [Effects of the Invention]

[0007] The present invention has the effect of providing a virtual tour with improved accuracy in estimating the shooting location of captured images. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall configuration of an image processing system. [Figure 2] This figure shows an example of a 360-degree spherical image captured by a camera. [Figure 3] (A) is a hemispherical image (front) taken with the imaging device, (B) is a hemispherical image (back) taken with the imaging device, and (C) is an image represented using equirectangular projection. [Figure 4] (A) A conceptual diagram showing the state of covering the sphere with an equirectangular projection image, and (B) A diagram showing a full-sphere image. [Figure 5] This diagram shows the positions of a virtual camera and a predetermined region when a 360-degree spherical image is treated as a three-dimensional sphere. [Figure 6] This figure shows the relationship between information from a predetermined region and an image of a predetermined region T. [Figure 7] This figure shows an example of the conditions during imaging using the imaging device. [Figure 8] This is a diagram illustrating an example of a 360-degree image. [Figure 9] This figure illustrates an example of a planar image converted from a 360-degree image. [Figure 10] (A)(B) This is a schematic diagram illustrating an example of an imaging device applicable to an image processing system. [Figure 11] This is a diagram illustrating an example of an image captured by a general radiographic device. [Figure 12] (A) to (C) are schematic diagrams illustrating an example of processing performed by an image processing device. [Figure 13] This figure shows an example of the hardware configuration of an imaging device. [Figure 14] This figure shows an example of the hardware configuration of an image processing device and a communication terminal. [Figure 15] This figure shows an example of the functional configuration of an image processing system. [Figure 16] This flowchart shows an example of tour photography processing using a camera. [Figure 17] (A)(B) This figure shows an example of a method for fixing the imaging device. [Figure 18] This diagram illustrates the differences between handheld and fixed-position shooting. [Figure 19] This diagram illustrates the switching between video recording and still image capture using a recording device. [Figure 20] This figure shows an example of the timing for switching between video recording and still image shooting. [Figure 21] (A)(B) This is a diagram illustrating an example of a tour path in a virtual tour. [Figure 22] This flowchart shows an example of the process of generating tour images using an image processing device. [Figure 23] (A)(B) Schematic diagram for explaining an example of the estimation process of the size of the room shape. [Figure 24] (A)(B) Schematic diagram for explaining an example of the determination process of the room where the point on the movement path is located. [Figure 25] (A)(B) Schematic diagram for explaining an example of the determination process of the room where the point on the movement path is located. [Figure 26] Schematic diagram for explaining an example of the path generated by the path generation unit. [Figure 27] (A)(B) Schematic diagram for explaining an example of the path generated when the use of the room is a toilet or a bathroom. [Figure 28] Schematic diagram for explaining an example of the tour image generated by the image processing device. [Figure 29] (A)(B) Schematic diagram for explaining an example of the tour image generated by the image processing device.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the invention will be described with reference to the drawings. In the description of the drawings, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted.

[0010] ●Embodiment● ●Overview of the Image Processing System First, the outline of the configuration of the image processing system according to the embodiment will be described using FIG. 1. FIG. 1 is a diagram showing an example of the overall configuration of the image processing system. The image processing system 1 shown in FIG. 1 is a system that performs image processing on a captured image for allowing a viewer to view the internal space of a structure such as a real estate property online.

[0011] As shown in Figure 1, the image processing system 1 includes a shooting device 10, an image processing device 50, and a communication terminal 90. The shooting device 10, the image processing device 50, and the communication terminal 90 that constitute the image processing system 1 can communicate via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), etc. In addition to wired communication, the communication network 100 may also include wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] The image processing device 50 is a server computer that performs image processing on captured images of the interior space of a structure such as a real estate property, which is a predetermined location. For example, the image processing device 50 acquires captured images taken by the camera 10 and uses the acquired images to generate tour images for providing a virtual tour to the user. Here, a virtual tour is content that allows the user to view the interior of a structure such as a real estate property as if they were actually touring the site. The tour images are generated using multiple captured images taken by the camera 10 and are viewing images that allow the user to virtually move around within the location shown in the captured images through user operation.

[0013] The image processing device 50 may consist of a single server computer or multiple server computers. Furthermore, although the image processing device 50 is described as a server computer located in a cloud environment, it may also be a server located in an on-premises environment.

[0014] The imaging device 10 is a special digital camera (360° spherical imaging device) capable of capturing a 360° spherical image by photographing the interior space of a structure such as a real estate property at the shooting location in all directions. The imaging device 10 is used, for example, by a real estate agent who manages or sells real estate properties. The imaging device 10 may also be a wide-angle camera or stereo camera capable of acquiring wide-angle images with a field of view greater than a predetermined value. A wide-angle image is generally an image taken using a wide-angle lens, an image taken with a lens that can capture a wider area than what the human eye can perceive. In other words, the imaging device 10 is an imaging means capable of acquiring images (360° spherical images, wide-angle images) taken using a lens with a focal length shorter than a predetermined value. A wide-angle image generally refers to an image taken with a lens with a focal length of 35mm or less in 35mm film equivalent.

[0015] The communication terminal 90 is a computer such as a smartphone that displays images processed by the image processing device 50 and allows viewers to view them. The communication terminal 90 is used, for example, by the same real estate agent as the camera 10. The communication terminal 90 has a dedicated application installed for, for example, issuing shooting instructions to the camera 10 and viewing images provided by the image processing device 50. Alternatively, the communication terminal 90 may be configured to issue shooting instructions and view images by accessing a dedicated website using a web browser, for example, without using a dedicated application. Furthermore, the shooting instructions and image viewing may be performed by different communication terminals 90.

[0016] The communication terminal 90 is not limited to smartphones; for example, it may be a PC, tablet, wearable device, HMD (head-mounted display), or IWB (Interactive White Board: an electronic whiteboard with mutual communication capabilities).

[0017] ○Overview of the imaging device○ Here, we will explain the general outline of the imaging device 10 that constitutes the image processing system 1 using Figures 2 to 11. Figure 2 is a diagram showing an example of a 360-degree spherical image captured by the imaging device. The image shown in Figure 2 is a 360-degree spherical image of a room in a real estate property, which is an example of the interior space of a structure, captured by the imaging device 10. 360-degree spherical images are suitable for viewing real estate properties because they can capture the interior of a room in all directions. While there are various forms of 360-degree spherical images, they are often generated using the equirectangular projection method, which will be described later. The advantages of images generated by this equirectangular projection method are that the outline of the image is rectangular, making image data storage efficient and easy, and that there is little distortion near the equator, resulting in a relatively natural appearance because vertical lines are not distorted.

[0018] ○ Method for generating a 360-degree image Next, the method for generating a 360-degree image will be explained using Figures 3 to 9. First, Figures 3 and 4 will be used to outline the process from the image captured by the imaging device 10 to the generation of a 360-degree image. Figure 3(A) shows the hemispherical image (front side) captured by the imaging device, Figure 3(B) shows the hemispherical image (back side) captured by the imaging device, and Figure 3(C) shows the image represented by equirectangular projection (hereinafter referred to as "equistratic projection image"). Figure 4(A) is a conceptual diagram showing the state in which the sphere is covered by the equirectangular projection image, and Figure 4(B) shows the 360-degree image.

[0019] The imaging device 10 has image sensors on both the front and rear sides. These image sensors are used in conjunction with optical components such as lenses that can capture hemispherical images (angle of view of 180° or more). The imaging device 10 can obtain two hemispherical images by capturing subjects around the user with each of the two image sensors.

[0020] As shown in Figures 3(A) and 3(B), the images obtained by the image sensor of the imaging device 10 are curved hemispherical images (front and rear). The imaging device 10 then combines the hemispherical image (front) and the hemispherical image (rear) that has been inverted 180 degrees to create an equirectangular projection image EC as shown in Figure 3(C).

[0021] The imaging device 10 then uses OpenGL ES (Open Graphics Library for Embedded Systems) to overlay the equirectangular projection image EC so that it covers the sphere, as shown in Figure 4(A), and creates a full-sphere image (full-sphere panoramic image) CE as shown in Figure 4(B). In this way, the full-sphere image CE is represented as an image where the equirectangular projection image EC is facing the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensional) and 3D (3-Dimensional) data. The full-sphere image CE may be a still image or a video. Furthermore, the conversion method is not limited to OpenGL ES; any method capable of converting from a hemispherical image to an equirectangular projection method is acceptable, for example, it may be a CPU calculation or an OpenCL calculation.

[0022] As described above, the 360-degree spherical image CE is an image pasted to cover a sphere, which can cause discomfort to the human eye. Therefore, the imaging device 10 can display a predetermined area T of the 360-degree spherical image CE (hereinafter referred to as the "predetermined area image") as a flat image with less curvature, thereby providing a display that does not cause discomfort to the human eye. This will be explained using Figures 5 and 6.

[0023] Figure 5 shows the positions of a virtual camera and a predetermined region when the 360-degree image is treated as a three-dimensional solid sphere. The virtual camera IC corresponds to the viewpoint of the user viewing the 360-degree image CE, which is displayed as a three-dimensional solid sphere. Figure 5 represents the 360-degree image CE as a three-dimensional solid sphere CS. If the generated 360-degree image CE is a solid sphere CS, then, as shown in Figure 5, the virtual camera IC is located inside the 360-degree image CE. The predetermined region T in the 360-degree image CE is the shooting area of ​​the virtual camera IC and is identified by predetermined region information indicating the shooting direction and field of view of the virtual camera IC in the three-dimensional virtual space containing the 360-degree image CE. Furthermore, zooming in on the predetermined region T can also be represented by moving the virtual camera IC closer to or further away from the 360-degree image CE. The predetermined region image Q is an image of the predetermined region T in the 360-degree image CE. Therefore, the predetermined region T can be identified by the field of view α and the distance f from the virtual camera IC to the 360-degree image CE.

[0024] The predetermined region image Q is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The following explanation will use the shooting direction (ea, aa) and field of view (α) of the virtual camera IC. Note that the predetermined region T may be represented not by the field of view α and distance f, but by the imaging area (X, Y, Z) of the virtual camera IC which is the predetermined region T.

[0025] Next, we will explain the relationship between the predetermined region information and the image of the predetermined region T using Figure 6. Figure 6 is a diagram showing the relationship between the predetermined region information and the image of the predetermined region T. As shown in Figure 6, "ea" is the elevation angle, "aa" is the azimuth angle, and "α" is the field of view (Angle). That is, the orientation of the virtual camera IC is changed so that the point of fixation of the virtual camera IC, indicated by the shooting direction (ea,aa), becomes the center point CP(x,y) of the predetermined region T, which is the shooting area of ​​the virtual camera IC. As shown in Figure 6, the center point CP(x,y) when the diagonal field of view of the predetermined region T, represented by the field of view α of the virtual camera IC, is α becomes the parameter ((x,y)) of the predetermined region information. The predetermined region image Q is the image of the predetermined region T in the 360-degree spherical image CE. f is the distance from the virtual camera IC to the center point CP(x,y). L is the distance between any vertex of the predetermined region T and the center point CP(x,y) (2L is the diagonal). In Figure 6, the trigonometric functions generally hold true, as shown in (Equation 1) below.

[0026]

number

[0027] Next, Figure 7 will be used to explain the state of the imaging device 10 during imaging. Figure 7 is a diagram showing an example of the state of imaging by the imaging device. In order to capture an entire room of a real estate property, it is preferable to install the imaging device 10 at a height close to the height of human eyes. For this reason, as shown in Figure 7, the imaging device 10 is generally fixed with a support member 20 such as a monopod or tripod for imaging. As described above, the imaging device 10 is a 360-degree imaging device capable of acquiring light rays from all directions around it, and it can also be said that it acquires an image (360-degree image CE) on a unit sphere around the imaging device 10. When the imaging direction is determined, the coordinates of the 360-degree image are determined. For example, in Figure 7, point A is at a distance of (d, -h) from the center point C of the imaging device 10, and if the angle between line segment AC and the horizontal direction is θ, then the angle θ can be expressed by the following (Equation 2).

[0028]

number

[0029] Assuming that point A is at a depression angle θ, the distance d between point A and point B can be expressed by the following equation (Equation 3), using the installation height h of the imaging device 10.

[0030]

number

[0031] Here, we will briefly explain the process of converting positional information on a 360-degree image to coordinates on a planar image converted from the 360-degree image. Figure 8 is a diagram illustrating an example of a 360-degree image. Figure 8(A) shows the hemispherical image shown in Figure 3(A) with lines connecting points where the angles of incidence in the horizontal and vertical directions relative to the optical axis are equal. Hereafter, the angle of incidence in the horizontal direction relative to the optical axis will be referred to as "θ", and the angle of incidence in the vertical direction relative to the optical axis will be referred to as "φ".

[0032] Figure 9(A) illustrates an example of an image processed using equirectangular projection. Specifically, the image shown in Figure 8 is matched using a pre-generated LUT (Look Up Table), processed using equirectangular projection, and the processed images shown in Figures 8(A) and 8(B) are combined to generate the planar image shown in Figure 9(A), which corresponds to the 360-degree spherical image, by the imaging device 10. The equirectangular projection image EC shown in Figure 3(C) is an example of the planar image shown in Figure 9(A).

[0033] As shown in Figure 9(A), in an image processed using equirectangular projection, latitude (θ) and longitude (φ) are orthogonal. In the example shown in Figure 9(A), the center of the image is (0,0), and the latitude direction is represented as -90 to +90, and the longitude direction as -180 to +180, allowing any position in the spherical image to be indicated. For example, the coordinates of the top left of the image are (-180,-90). The coordinates of the spherical image may be expressed in a 360-degree format as shown in Figure 9(A), or in radians or in pixels like a real image. Alternatively, the coordinates of the spherical image may be converted to two-dimensional coordinates (x,y) as shown in Figure 9(B).

[0034] Furthermore, the image compositing process shown in Figure 9(A) or Figure 9(B) is not limited to simply arranging the hemispherical images shown in Figures 8(A) and 8(B) in sequence. For example, if the horizontal center of the spherical image is not θ=180°, in the compositing process, the imaging device 10 first preprocesses the hemispherical image shown in Figure 3(C) and places it at the center of the spherical image. Next, the imaging device 10 divides the image obtained by preprocessing the hemispherical image shown in Figure 3(B) into left and right portions, and then combines the hemispherical images to generate the equirectangular projection image EC shown in Figure 3(C).

[0035] Furthermore, in the planar image shown in Figure 9(A), the points corresponding to the poles (PL1 or PL2) of the hemispherical images (global images) shown in Figures 8(A) and 8(B) are line segments CT1 or CT2. This is because, as shown in Figures 4(A) and 4(B), the global image (for example, the global image CE) is created by using OpenGL ES to superimpose the planar image (equirectangular projection image EC) shown in Figure 9(A) onto a sphere.

[0036] ○Examples of imaging devices applicable to image processing systems Next, an example of an imaging device 10 applicable to the image processing system 1 according to the embodiment will be described using Figures 10 and 11. Figure 10 is a schematic diagram illustrating an example of an imaging device applicable to the image processing system. Figure 10(A) shows a special imaging device equipped with multiple image sensors that can generate a 360-degree spherical image by the generation method described above. The special imaging device uses a wide-angle lens or a fisheye lens with a wide field of view and can acquire an image that captures all directions by combining the outputs of multiple image sensors. Figure 10(B) shows a general imaging device, which is a so-called ordinary camera. The general imaging device is, for example, a regular digital camera or a mobile terminal such as a smartphone equipped with a camera. The photographer takes pictures while holding the general imaging device in their hand and rotating it. The general imaging device can acquire an image in all directions by combining the obtained images. Both the special imaging device and the general imaging device generate the final captured image by stitching together multiple shooting results through image processing. It is preferable that the optical centers of the imaging device 10 for obtaining multiple shooting results are the same.

[0037] Figure 11 is a diagram illustrating an example of an image captured by a general imaging device. Figure 11 shows an image captured when a photographer holds a general imaging device, as shown in Figure 10(B), in their hand and rotates it while taking a picture. Because general imaging devices have a small field of view (generally less than 100 degrees), there is a problem that the upper and lower poles cannot be captured, as shown in Figure 11. In addition, rotation by the photographer themselves can cause parallax during shooting due to a shift in the position of the optical center of the imaging device, and unnatural errors such as steps are likely to occur in the stitching process. In other words, in this embodiment, either a special imaging device or a general imaging device, as shown in Figure 10, can be applied as the imaging device 10, but it is preferable to apply a special imaging device as shown in Figure 10(A). By using a special imaging device (spherical imaging device) as shown in Figure 10(A), the image processing system 1 can use flawless, natural, high-quality spherical images for use in virtual tours where quality such as advertising is required. In the following explanation, the imaging device 10 will be described as a special imaging device (spherical imaging device).

[0038] ○Overview of the processing performed by the image processing device○ Next, an overview of the processing performed by the image processing device 50 will be explained using Figure 12. Figure 12 is a schematic diagram illustrating an example of the processing performed by the image processing device. Figure 12 shows the relationship between the shooting position of the captured image taken by the shooting device 10 and the path that links multiple captured images.

[0039] The image processing device 50 estimates the shooting position (Figure 12(A)) of the captured image acquired from the imaging device 10 using methods such as visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion). The estimated shooting position is expressed as the relative position of each shooting position of multiple captured images, as shown in Figure 12(B). Furthermore, the estimated shooting position is expressed in a single coordinate system for each shooting position.

[0040] In addition to estimating the shooting location, the image processing device 50 can also reconstruct the path taken when the captured image was taken, and can reconstruct the order and path taken to reach each shooting location. Based on the estimated shooting location and the path taken when the image was taken, the image processing device 50 generates a path (tour path). A tour path indicates the connection relationship between multiple captured images on a tour image. As shown in Figure 12(C), by connecting the captured images taken at each estimated shooting location and allowing users to move between the connected captured images, a virtual tour using the captured images can be created.

[0041] ● Hardware configuration Next, the hardware configuration of each device or terminal constituting the image processing system according to the embodiment will be described using Figures 13 and 14. Note that the hardware configuration shown in Figures 13 and 14 may have components added or removed as needed.

[0042] ○Hardware configuration of the imaging device○ First, the hardware configuration of the imaging device 10 will be explained using Figure 13. Figure 13 is a diagram showing an example of the hardware configuration of the imaging device. In the following, the imaging device 10 will be described as a 360-degree (omnidirectional) imaging device using two image sensors, but there may be more than two image sensors. Furthermore, it is not necessarily required to be a device dedicated to omnidirectional imaging; an omnidirectional imaging unit can be attached to a regular digital camera or smartphone, etc., to provide essentially the same functionality as the imaging device 10.

[0043] As shown in Figure 13, the imaging device 10 consists of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operating unit 115, an input / output interface 116, a short-range communication circuit 117, an antenna 117a for the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network interface 121.

[0044] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lens 102 when distinction is not necessary) each having a field of view of 180° or more for forming a hemispherical image, and two image sensors 103a and 103b provided corresponding to each lens. The image sensors 103a and 103b have an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the lenses 102a and 102b into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers for setting various commands or parameters necessary for the operation of the image sensors.

[0045] The image sensors 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the image sensors 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to the ROM 112, SRAM 113, DRAM 114, operation unit 115, input / output I / F 116, short-range communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, and network I / F 121, etc.

[0046] The image processing unit 104 receives image data output from image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then combines these image data to create equirectangular projection image data as shown in Figure 3(C).

[0047] The imaging control unit 105 generally uses the I2C bus to set commands and other information in the registers of the image sensors 103a and 103b, with the imaging control unit 105 acting as the master device and the image sensors 103a and 103b as slave devices. It receives the necessary commands and other information from the CPU 111. The imaging control unit 105 also uses the I2C bus to acquire status data and other information from the registers of the image sensors 103a and 103b and send it to the CPU 111.

[0048] Furthermore, the imaging control unit 105 instructs the image sensors 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Depending on the imaging device 10, there may also be functions to display a preview or video on a display (for example, the display of an external terminal such as a smartphone that communicates with the imaging device 10 via a short-range communication circuit 117). In this case, the image data output from the image sensors 103a and 103b is performed continuously at a predetermined frame rate (frames / minute).

[0049] Furthermore, as will be described later, the imaging control unit 105 also functions as a synchronization control means that works in cooperation with the CPU 111 to synchronize the output timing of image data from the image sensors 103a and 103b. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

[0050] The CPU 111 controls the overall operation of the imaging device 10 and performs necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111 and data in progress. In particular, the DRAM 114 stores image data in progress of processing by the image processing unit 104 and data of completed equirectangular projection images.

[0051] The control unit 115 is a collective term for various operation buttons, a power switch, a shutter button, and a touch panel that serves both display and operation functions. The user inputs various shooting modes, shooting conditions, etc., by operating the control unit 115.

[0052] The input / output interface (I / F) 116 is a general term for interface circuits (such as USB I / F) to external media such as SD cards or personal computers. The I / F 116 can be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded to external media via the I / F 116, or transmitted to an external terminal (device) via the I / F 116 as needed.

[0053] The short-range communication circuit 117 communicates with an external terminal (device) via an antenna 117a provided on the imaging device 10 using a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi. The short-range communication circuit 117 can transmit equirectangular projection image data to the external terminal (device).

[0054] The electronic compass 118 calculates the orientation of the camera 10 from the Earth's magnetic field and outputs orientation information. This orientation information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of captured images. The related information also includes the date and time the image was taken and the data size of the image data. The gyro sensor 119 is a sensor that detects changes in angle (Roll angle, Pitch angle, Yaw angle) associated with the movement of the camera 10. The changes in angle are an example of related information (metadata) according to Exif and are used for image processing such as image correction of captured images. Furthermore, the acceleration sensor 120 is a sensor that detects acceleration in three axes. The camera 10 calculates its orientation (angle relative to the direction of gravity) based on the acceleration detected by the acceleration sensor 120. By providing the acceleration sensor 120, the accuracy of image correction of the camera 10 is improved. The network I / F 121 is an interface for data communication using a communication network 100 such as the Internet via a router or the like.

[0055] ○Hardware configuration of the image processing device○ First, the hardware configuration of the image processing device 50 will be explained using Figure 14. Figure 14 is a diagram showing an example of the hardware configuration of the image processing device. Each hardware component of the image processing device 50 is indicated by a code in the 500 series. The image processing device 50 is built by a computer and, as shown in Figure 14, includes a CPU 501, ROM 502, RAM (Random Access Memory) 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0056] Of these, the CPU 501 controls the operation of the entire image processing device 50. The ROM 502 stores programs used to drive the CPU 501, such as the IPL (Initial Program Loader). The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The display 506 may be a touch panel display equipped with input means. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory stick. The network I / F 509 is an interface for data communication using the communication network 100. The bus line 510 is an address bus or data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 14.

[0057] Furthermore, the keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, or moving a cursor, etc. Note that the input means may not be limited to the keyboard 511 and the pointing device 512, but may also be a touch panel or an audio input device, etc. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that the removable recording medium is not limited to DVD-RW, but may also be DVD-R or Blu-ray® Disc, etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0058] ○Hardware configuration of communication terminals○ Figure 14 also shows an example of the hardware configuration of the communication terminal 90. Each hardware component of the communication terminal 90 is indicated by a 900-series code in parentheses. The communication terminal 90 is built by a computer and has a configuration similar to that of the image processing device 50, as shown in Figure 14, so the explanation of each hardware component is omitted. In addition to the same configuration as the image processing device 50, the communication terminal 90 is equipped with a short-range communication circuit 917 and an antenna 917a for the short-range communication circuit 917. The short-range communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0059] The above programs may be distributed as installable or executable files recorded on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, and USB memory. The recording media can also be provided domestically or internationally as a program product. For example, the image processing device 50 implements the image processing method according to the present invention by executing the program according to the present invention.

[0060] ●Functional Configuration Next, the functional configuration of the image processing system according to the embodiment will be described using Figure 15. Figure 15 is a diagram showing an example of the functional configuration of the image processing system. Note that Figure 15 shows the devices or terminals shown in Figure 1 that are related to the processing or operation described later.

[0061] ○Functional Configuration of the Imaging Device○ First, the functional configuration of the imaging device 10 will be explained using Figure 15. The imaging device 10 includes a transmitting / receiving unit 11, an operation reception unit 12, an imaging control unit 13, a video recording unit 14, a still image recording unit 15, a motion detection unit 16, a communication unit 17, and a storage / reading unit 19. Each of these units is a function or means realized by the operation of any of the components shown in Figure 13 by instructions from the CPU 111 according to a program for the imaging device deployed on SRAM 113 to DRAM 114. The imaging device 10 also has a storage unit 1000 constructed from ROM 112, SRAM 113, and DRAM 114 shown in Figure 13. The storage unit 1000 stores the GUID (Globally Unique Identifier) ​​of the device.

[0062] The transmitting / receiving unit 11 is primarily implemented by the processing of the CPU 111 and communicates various data or information with other devices or terminals. Furthermore, the transmitting / receiving unit 11 uses the network I / F 121 to perform data communication with other devices or terminals via the communication network 100.

[0063] The operation reception unit 12 is mainly implemented by the processing of the CPU 111 to the operation unit 115, and receives various selections or inputs from the user who is taking the picture.

[0064] The shooting control unit 13 is mainly implemented by the processing of the imaging unit 101, image processing unit 104, and imaging control unit 105 by the CPU 111, and captures images of subjects such as landscapes and acquires captured image data. The shooting control unit 13 performs shooting by, for example, switching between video shooting by the video shooting unit 14 and still image shooting by the still image shooting unit 15 in a time-division manner.

[0065] The video recording unit 14 is mainly realized by the processing of the CPU 111 to the imaging unit 101, image processing unit 104, and imaging control unit 105, and performs video recording by the imaging device 10. The video recording unit 14, for example, records moving images while moving within a structure such as a real estate property, which is a predetermined location. The video recording unit 14 performs video recording in low-resolution continuous frames while the photographer holding the imaging device 10 is moving, and stores the captured image data in the storage unit 1000. The video recording unit 14, for example, performs video recording when the photographer holding the imaging device 10 is moving from a first point to a second point within a real estate property, which is a predetermined location.

[0066] The still image capture unit 15 is mainly implemented by the processing of the imaging unit 101, and captures images of subjects such as landscapes and takes still images using the shooting device 10. The still image capture unit 15 captures multiple still images taken at different shooting positions within a structure such as a real estate office, which is a predetermined location. The still image capture unit 15 captures still images (photographs) with a higher resolution than the moving images captured by the moving image capture unit 14, and stores the captured image data in the storage unit 1000. The still images captured by the still image capture unit 15 may be single-frame images or HDR (High Dynamic Range) images created by combining multiple images.

[0067] Here, the still images captured by the still image capture unit 15 are preferably high resolution, for example, 4K resolution or higher. On the other hand, the moving images captured by the video capture unit 14 are images used for position estimation, and only the subjects in the moving images need to be identifiable, so they are lower resolution images than still images. The moving images may have a resolution of, for example, 480p or lower. By capturing low-resolution moving images, the shooting device 10 can reduce the overall amount of data in tour shooting.

[0068] The movement detection unit 16 is mainly implemented by the CPU 111 processing the gyro sensor 119 and the acceleration sensor 120, and detects the movement state of the imaging device 10. For example, the movement detection unit 16 detects whether the person holding the imaging device 10 is moving (in a moving state) or stationary (in a stationary state) while the imaging device 10 is taking pictures.

[0069] The communication unit 17 is primarily implemented by the processing performed by the CPU 111 on the input / output interface 116 or the short-range communication circuit 117, and transmits and receives various data or information with other devices or terminals. For example, the communication unit 17 uses the input / output interface 116 to perform data communication with the communication terminal 90 via various cables, etc. The communication unit 17 also uses the short-range communication circuit 117 to perform data communication with the communication terminal 90 using short-range wireless communication technology.

[0070] The storage / reading unit 19 is primarily implemented by the CPU 111 and stores various data (or information) in the storage unit 1000 and reads various data (or information) from the storage unit 1000. The storage unit 1000 also stores image data captured by the video recording unit 14 and the still image recording unit 15. The image data stored in the storage unit 1000 has the time of capture of the captured image associated with it as metadata.

[0071] ○Functional Configuration of Image Processing Device○ First, the functional configuration of the image processing device 50 will be explained using Figure 15. The image processing device 50 includes a transmitting / receiving unit 51, a receiving unit 52, a judgment unit 53, an application determination unit 54, a shape estimation unit 55, a position estimation unit 56, a detection unit 57, a path generation unit 58, an image processing unit 60, and a storage / reading unit 59. Each of these units is a function or means realized by the operation of any of the components shown in Figure 14 by instructions from the CPU 501 according to the image processing device program deployed from HD 504 onto RAM 503. The image processing device 50 also has a storage unit 5000 constructed from ROM 502, RAM 503, and HD 504 shown in Figure 14.

[0072] The transmitting / receiving unit 51 is mainly implemented by the processing of the CPU 501 to the network I / F 509, and transmits and receives various data or information with other devices or terminals via the communication network 100. The transmitting / receiving unit 51, for example, receives (acquires) moving images captured by the camera 10 from the camera 10 or the communication terminal 90. The transmitting / receiving unit 51 also receives (acquires) still images captured by the camera 10 from the camera 10 or the communication terminal 90.

[0073] The reception unit 52 is mainly implemented by the CPU 501 processing the keyboard 511 or pointing device 512, and accepts various selections or inputs from the user. The decision unit 53 is implemented by the CPU 501 and makes various decisions.

[0074] The usage determination unit 54 is implemented by the processing of the CPU 501 and determines the usage of the space shown in the captured image based on the captured image which shows the interior space of the structure in all directions. In other words, the usage is the classification, genre, or purpose of use of the space actually captured by the shooting device 10. If a real estate property is captured as the base captured by the shooting device 10, the usage of the room, which is the space shown in the captured image, is, for example, a toilet, bathroom, or entrance.

[0075] The shape estimation unit 55 is implemented by the processing of the CPU 501 and estimates the shape of the space shown in the captured image based on the captured image, which shows the internal space of the structure in all directions.

[0076] The position estimation unit 56 is implemented by the processing of the CPU 501 and estimates the relative shooting position of still images acquired from the shooting device 10 based on the moving images acquired from the shooting device 10. The position estimation unit 56 estimates the relative position of still images from low-resolution continuous frames of moving images, for example, using methods such as visual SLAM or SfM. These visual SLAM or SfM methods can calculate the positions of feature points on images, the camera position, and parameters from multiple images. On the other hand, the calculated position is relative, and a reference position is required to obtain an absolute position.

[0077] The detection unit 57 is implemented by the processing of the CPU 501 and detects the capture points of the video image that belong to the spatial shape estimated by the shape estimation unit 55. The capture points of the video image are, for example, coordinate values ​​from which the movement path at the time of capture of the video image has been reconstructed.

[0078] The path generation unit 58 is implemented by the processing of the CPU 501 and generates paths that show the connection relationships between multiple still images based on the shooting positions estimated by the position estimation unit 56. For example, the path generation unit 58 generates paths that associate the nearest still images among the multiple still images based on the estimated shooting positions.

[0079] The image processing unit 60 is implemented by the CPU 501 and performs image processing to generate tour images for the virtual tour based on the shooting positions estimated by the position estimation unit 56. For example, the image processing unit 60 generates a tour image, which is a processed image containing multiple still images associated based on the estimated shooting positions. The image processing unit 60 determines the position where the still images should be placed in the tour image from the estimated shooting positions of the still images, the shooting times of the still images, and the path (tour path) generated by the path generation unit 58.

[0080] The memory and read unit 59 is primarily implemented by the CPU 501 and performs the storage of various data (or information) to the memory unit 5000 and the reading of various data (or information) from the memory unit 5000. The memory unit 5000 also stores tour images created by the image processing unit 60.

[0081] ○Communication terminal function configuration○ Next, the functional configuration of the communication terminal 90 will be explained using Figure 15. The communication terminal 90 has a transmitting / receiving unit 91, a receiving unit 92, a display control unit 93, a communication unit 94, and a storage / reading unit 99. Each of these units is a function or means realized by any of the components shown in Figure 15 operating according to instructions from the CPU 901 that follow the communication terminal program deployed from HD 904 onto RAM 903. The communication terminal 90 also has a storage unit 9000 constructed from ROM 902, RAM 903, and HD 904 shown in Figure 15.

[0082] The transmitting / receiving unit 91 is mainly implemented by the processing of the network interface 909 by the CPU 901, and transmits and receives various data or information with other devices or terminals via the communication network 100.

[0083] The reception unit 92 is mainly implemented by the processing of the keyboard 911 or pointing device 912 by the CPU 901, and accepts various selections or inputs from the user.

[0084] The display control unit 93 is mainly implemented by the processing of the CPU 901 and displays various images or characters on the display 906. The display control unit 93 accesses the image processing device 50 using, for example, a web browser or a dedicated application and displays images corresponding to the data distributed from the image processing device 50 on the display 906.

[0085] The communication unit 94 is primarily implemented by the processing performed by the CPU 901 on the external device connection interface 908 or the short-range communication circuit 917, and transmits and receives various data or information with other devices or terminals. For example, the communication unit 94 uses the external device connection interface 908 to perform data communication with the imaging device 10 via various cables, etc. The communication unit 94 also uses the short-range communication circuit 917 to perform data communication with the imaging device 10 using short-range wireless communication technology.

[0086] The memory and read unit 99 is mainly implemented by the CPU 901 and stores various data (or information) in the memory unit 9000 and reads various data (or information) from the memory unit 9000.

[0087] ●Processing or operation of the embodiment ○Tour photography processing○ Next, the processing or operation of the image processing system according to the embodiment will be described using Figures 16 to 29. In the following description, an example of a room in a real estate property will be shown as an example of the space inside a structure that is a predetermined base. First, the process of acquiring captured images used for generating tour images by the image processing device 50 will be described using Figures 16 to 20. Figure 16 is a flowchart showing an example of tour photography processing by the photography device.

[0088] First, the camera 10 starts the tour photography process in response to a predetermined request from the communication terminal 90 (step S11). Here, tour photography refers to taking photographs within a predetermined location in order to provide a virtual tour. Specifically, the communication terminal 90 performs the tour photography function by launching and running a dedicated application that has been installed, or by accessing a predetermined website using a web browser. Then, the communication unit 94 of the communication terminal 90 sends a tour photography request to the camera 10. The camera 10 starts the tour photography process in response to the tour photography request received by the communication unit 17.

[0089] Next, the video recording unit 14 of the camera 10 starts recording video while holding the camera 10 and moving around the base (step S12). Specifically, when the tour recording starts, the camera control unit 13 of the camera 10 requests the video recording unit 14 to start recording video. Then, the video recording unit 14 starts recording moving images in response to the request from the camera control unit 13.

[0090] Here, the method of taking images using the camera 10 will be explained with reference to Figures 17 and 18. Figure 17 is a diagram showing an example of how the camera is fixed. The camera 10 is preferably fixed to a support member 20 such as a monopod 20a as shown in Figure 17(A) or a tripod 20b as shown in Figure 17(B) so that the photographer is not visible in the image. The camera 10 is also fixed at the height desired when viewing the image. For example, the camera 10 is preferably positioned at eye level to show real estate properties in a natural way.

[0091] Figure 18 illustrates the differences between handheld and fixed shooting. Figure 18 shows the difference in physical movement when the photographer holds the shooting device 10 in their hand (handheld shooting) and when the shooting device 10 is fixed to a monopod 20a (fixed shooting). As shown in Figure 17, the simplest method for mobile shooting is to move the shooting device 10 while it is attached to a support member 20 such as a monopod 20a, but there are differences in physical movement between handheld and fixed shooting. Specifically, for example, by attaching a relatively long and large monopod 20a, the physical moment increases, and the movement of the shooting device itself is dampened.

[0092] Patent Document 1 estimates the shooting position using position estimation with sensors such as an IMU and PDR. However, when the monopod 20a is attached, as shown in Figure 18, the changes in acceleration and angular velocity are attenuated, so the method of estimating the shooting position as described in Patent Document 1 does not yield sufficient accuracy. Therefore, the image processing system 1 employs an image processing method described later so as not to affect processing accuracy even when the shooting device 10 is fixed to a support member 20 such as a monopod 20a to prevent the photographer from appearing in the image.

[0093] Returning to Figure 16, if the shooting device 10 detects a change in shooting mode (YES in step S13), it proceeds to step S14. In this case, the shooting device 10 switches from video shooting mode to still image shooting mode. For example, the shooting device 10 switches from video shooting mode to still image shooting mode when the shutter button on the operation unit 115 of the shooting device 10 is pressed. The shooting device detects the change in shooting mode. Alternatively, the shooting device 10 may be configured to switch from video shooting mode to still image shooting mode when the movement detection unit 16 detects that it has stopped (is stationary). On the other hand, if the shooting unit 14 of the shooting device 10 does not detect a change in shooting mode (NO in step S13), it continues video shooting in step S12.

[0094] Next, the video recording unit 14 of the shooting device 10 stops recording video in response to the detection of the shooting mode switch in step S13 (step S14). Then, the still image shooting unit 15 takes a still image in response to the user pressing the shutter button on the operation unit 115 (step S15).

[0095] Next, if the shooting device 10 detects a change in shooting mode (YES in step S16), similar to the process in step S13, it proceeds to step S18. In this case, the shooting device 10 switches from still image shooting mode to video shooting mode. For example, if the motion detection unit 16 detects that movement has resumed, the shooting device 10 switches from still image shooting mode to video shooting mode. Alternatively, the shooting device 10 may be configured to automatically switch from still image shooting mode to video shooting mode after still image shooting has been performed by the still image shooting unit 15.

[0096] The video recording unit 14 resumes video recording in response to the detection of the switching of the shooting mode in step S16 (step S17). The video recording unit 14 continues recording while holding the shooting device 10 and moving around the base, similar to step S12. The shooting device 10 then terminates processing when the tour recording is finished (YES in step S18). The end of the tour recording is determined, for example, by a predetermined operation performed by the photographer on the shooting device 10 or the communication terminal 90. On the other hand, if the tour recording is not finished (NO in step S18), the shooting device 10 repeats the processing from step S13 and continues the tour recording.

[0097] On the other hand, in step S16, if the shooting device 10 does not detect a change in shooting mode (NO in step S16), it proceeds to step S19. Similar to step S18, the shooting device 10 terminates processing when the tour shooting is finished (YES in step S19). On the other hand, if the tour shooting is not finished (NO in step S19), the shooting device 10 repeats the processing from step S19 and continues still image shooting by the still image shooting unit 15.

[0098] Here, we will explain video recording and still image capture in the shooting device 10 using Figures 19 and 20. Figure 19 is a diagram illustrating the switching between video recording and still image capture by the shooting device. As described above, the shooting device 10 is used for two purposes: video recording and still image capture. Of these, video recording is performed for the purpose of position estimation, and still image capture is performed for the purpose of acquiring images for viewing. Furthermore, video recording is performed when moving, and still image capture is performed when stationary.

[0099] Here, video recording and still image shooting require different shooting specifications depending on their respective purposes. For video recording, high frame rate continuous images (continuous frames) are acquired for use in position estimation, but high resolution and color layers are not required for the captured images; low resolution and grayscale are sufficient. For still image shooting, continuous frame acquisition is not required, but high resolution, color information (RGB), and high dynamic range are required. Furthermore, video recording is performed while moving, while still image shooting is performed when stationary.

[0100] Figure 20 shows an example of the timing for switching between video recording and still image capture. Figure 20 shows a time chart of the switching between video recording and still image capture in the process shown in Figure 16.

[0101] The shooting device 10 switches between video recording and still image capture in a time-division manner. As described above, video recording is performed to acquire images for position estimation, so it is necessary to capture continuous frames. On the other hand, still image capture is performed to acquire images for viewing the virtual tour, so it is not necessary to capture continuous frames, and a configuration in which a single image is acquired is sufficient, although an image may be acquired by high dynamic range synthesis using images from multiple frames.

[0102] Still image capture is an item where the photographer's position should be reflected, and the capture should be performed by some explicit action of the photographer. For example, still image capture is performed by switching from video recording when the shutter button on the operation unit 115 of the shooting device 10 is pressed. In this case, the shutter pressed by the operator may be the shutter button on the main body of the shooting device 10, or it may be a shutter based on the operation of a remotely controllable app or controller. For example, when photographing real estate properties, it is preferable that the photographer is not visible in the captured image. Therefore, when operating the shutter button on the main body of the shooting device 10, it is preferable that the shooting device 10 captures and stores a still image after a predetermined time has elapsed since the shutter button was pressed. This allows the photographer to avoid being visible in the captured image by moving outside the shooting range after pressing the shutter button. Furthermore, when performing remote operation, it is preferable that the photographer operates and takes the picture from outside the shooting range.

[0103] Furthermore, as a method for switching from video recording to still image recording, the system may be configured to switch to still image recording when the shooting device 10 fixed to the support member 20 is placed in the desired position or when the photographer himself becomes still, by detecting the stationary state with the movement detection unit 16, or it may be configured to automatically switch to still image recording after a predetermined time has elapsed since video recording started. In these methods, since the operator does not need to operate the shutter, there is an advantage in being able to perform tour photography smoothly.

[0104] On the other hand, since video recording is an item that is captured regardless of the photographer's intention, it is preferable that it be performed automatically. For example, video recording is performed automatically after still image recording has been completed, switching from still image recording. Also, in order to reduce the amount of data of the captured images, video recording may be configured to switch from still image recording when the motion detection unit 16 detects that movement has resumed.

[0105] In this way, the shooting device 10 can efficiently acquire images for generating tour images by switching in a time series between shooting video, which captures low-resolution moving images for position estimation, and shooting still images, which captures high-resolution still images for viewing in the virtual tour. In the above description, a configuration in which video shooting is performed at the start of tour shooting was described, but the shooting device 10 may also be configured to first take still images at the start of tour shooting.

[0106] The camera 10 uploads captured video and still images to the image processing device 50 via the communication terminal 90 as needed. The image processing device 50 temporarily stores the video and still images transmitted from the camera 10 in the storage unit 5000 for the purpose of generating tour images. Alternatively, the camera 10 may be configured to upload video and still images directly to the image processing device 50 without going through the communication terminal 90.

[0107] ○Tour Image Generation Processing○ Next, we will explain the process of generating tour images using the images captured by the imaging device 10, with reference to Figures 21 to 29.

[0108] Figure 21 illustrates an example of a tour path in a virtual tour. It is preferable that the tour path be generated to correspond to the adjacency relationships between rooms within an actual property. For example, if room A and room B are adjacent and accessible by a door, the still images taken in room A and room B should be connected by a path. On the other hand, if room A and room C are adjacent but inaccessible by a door, the still images taken in room A and room C should not be connected by a path. Therefore, when automatically generating a virtual tour, it is necessary to automatically determine which images should be connected by a path using the shooting locations of the still images and the paths between them.

[0109] As shown in Figure 21, the tour path is basically generated by connecting the still images in the order they were taken. For example, in the example shown in Figure 21(A), the path is connected in the order of shooting, generating a path "Room A ⇔ Room B ⇔ Room C". On the other hand, in the example shown in Figure 21(B), even though Room B and Room C were taken consecutively, they are not physically connected, so in the actual space, one must return to Room A, where they were originally, before entering Room C. In such a case, simply connecting the path in the order of shooting would connect rooms that are not actually accessible from each other, resulting in a tour that does not reflect reality. Therefore, as shown in Figure 21(B), the structure of the actual space is taken into consideration, and the path connecting Room B and Room C is generated by passing through Room A again.

[0110] In this way, the image processing device 50 generates paths in the generated tour images using an algorithm that takes into account which room each shooting point on the movement path during shooting is located in, so that the paths can be correctly connected. The algorithm for automatically generating paths in the image processing device 50 will be described in detail below.

[0111] Figure 22 is a flowchart illustrating an example of the tour image generation process by the image processing device. The image processing device 50 uses moving images and still images captured by the camera 10 to perform the tour image generation process. It is assumed that the image processing device 50 has already acquired moving images and still images captured by the camera 10. In this case, the transmitting and receiving unit 51 of the image processing device 50 receives (acquires) the moving images and still images transmitted from the camera 10. In the examples shown in Figures 23 to 27, the still images are assumed to be of rooms A, B, and C, respectively, taken at the shooting locations (1 to 3 in each drawing) within rooms A, B, and C.

[0112] First, the use determination unit 54 of the image processing device 50 determines the use of the room shown in the acquired video (step S31). Specifically, the use determination unit 54 determines the use (genre) of the room shown in the video using general machine learning or parameters such as video feature quantities. The room uses determined here are, for example, toilets, bathrooms, or entrances.

[0113] Next, the shape estimation unit 55 estimates the shape of the room as seen in the acquired video (step S32). Specifically, the shape estimation unit 55 detects straight lines of the subject in the video, finds the vanishing points of the detected straight lines, and estimates the shape of the room as seen in the video by methods such as estimating the room structure from the boundaries of the floor, walls, or ceiling, using three-dimensional information acquired from the shooting device 10, or using general machine learning, and performs three-dimensional reconstruction to obtain an overhead view of the room shape.

[0114] Furthermore, the shape estimation unit 55 estimates the size of the room shape whose structure was estimated in step S32 (step S33). Specifically, the shape estimation unit 55 adjusts the estimated size of the room shape to match the coordinate system of each point in the moving image along the movement path. As shown in Figure 23(A), the absolute size of the room shape estimated in step S32 is unknown when it is reconstructed from a single image. As shown in Figure 23(B), in order to place each room shape with the correct size on the movement path during video recording, the shape estimation unit 55 determines the size of each room shape in the coordinate system of each point in the moving image along the movement path during video recording. The shape estimation unit 55 estimates the size of each room shape using the shooting height during video recording and the average value of the size that makes the gap between adjacent room shapes zero. In this way, by estimating the shape and size of the rooms captured in the moving image by the shape estimation unit 55, the position estimation unit 56 estimates the relative shooting position of the moving image and the multiple still images captured in time division.

[0115] Next, the detection unit 57 determines which room shape, among the room shapes estimated in steps S32 and S33, each point of the moving path in the video image is located within (step S34). The detection unit 57 detects from the coordinate values ​​which room, among the room shapes arranged in step S33, each point on the moving path is located inside. In this way, the detection unit 57 detects which room each point on the moving path was located inside. The image processing device 50, for example as shown in Figure 24(B), uniquely determines which room each point is located inside based on the coordinate values ​​on the moving path at the time the restored video image was captured. In the example shown in Figure 24(B), it is determined that the point on path R1 is located inside room A, the point on path R2 is located inside room B, and the point on path R3 is located inside room C.

[0116] Furthermore, if the detection unit 57 detects that a point in the moving image on the movement path is not located within a single room shape (NO in step S34), it proceeds to step S35. As shown in section S1 in Figure 25(A), if a point on the movement path does not belong to a single room, that is, if a point on the movement path is located within multiple room shapes (room A and room B in the example of Figure 25), it is not possible to uniquely determine which room the point was located in at the time of shooting. Also, as shown in section S2 in Figure 25(A), if a point on the movement path is not located within any of the room shapes, it is not possible to uniquely determine which room the point was located in at the time of shooting. These issues arise due to errors in reconstruction of the movement path, errors in estimation of the room shape, and errors in the size of the room shape.

[0117] Therefore, if the points in the moving image on the movement path do not exist within a single room shape, the detection unit 57 determines that the room in which the point closest to the point on the movement path is located is the room in which the point exists, in order to uniquely determine which room shape the target point exists within (step S35). The detection unit 57 calculates the nearest point on the movement path for the points in sections S1 and S2 in Figure 25(A), and determines that the room in which the nearest point on the movement path is located is the room in which the target point is located, as shown in sections M1 and M2 in Figure 25(B).

[0118] On the other hand, in step S34, if the detection unit 57 detects that each point of the moving image is located within a single room shape (YES in step S34), it proceeds to step S36.

[0119] Next, the path generation unit 58 generates a path connecting the room where a point in the moving image on the movement path exists with the room where the next point exists (step S36). Since the processing in steps S34 and S35 has uniquely determined which room shape each point on the movement path is located in, the path generation unit 58 connects each point on the movement path with a path, designating the room where one point exists and the room where the next point exists as the next room. In the example in Figure 26, the path generation unit 58 connects room A to room B, which is entered after room A, with a path (N1). Also, in the case of movement from room B back to room A and then entering room C, since there is a section of room A (N2 and N3) between shooting position 2 and shooting position 3, the path generation unit 58 can correctly connect the path from room A to room C based on the estimation results in steps S32 and S33. Finally, the path generation unit 58 can generate a path "R1⇔R2⇔R1⇔R3" as shown in Figure 26. In other words, paths "Room A ⇔ Room B" and "Room A ⇔ Room C" are generated. That is, by generating paths connecting each room, the path generation unit 58 can associate the nearest still image from among the multiple still images taken of each room.

[0120] Next, if the determination unit 53 determines that the use of the next room determined in step S31 is a toilet or bathroom (YES in step S37), it proceeds to step S38. On the other hand, if the determination unit 53 determines that the use of the next room determined in step S31 is not a toilet or bathroom (NO in step S37), it proceeds to step S39.

[0121] The system prevents incorrect paths from being created by taking advantage of the fact that in most houses, there is only one room leading to a toilet or bathroom, meaning there is only one room to return to. For example, as shown in Figure 27(A), if there is a lot of overlap due to the error in the above processing, a path may be generated that connects room A to room B via the toilet or bathroom. In the example in Figure 27(A), the determination based on the nearest point on the travel path in step S35 determines that the room where the overlapping section J1 point exists is room B, and a path connecting the toilet or bathroom is generated. However, in real houses, it is rare for a toilet or bathroom to lead to two different rooms.

[0122] Therefore, the determination unit 53 sets the current room as the room adjacent to the next room if the use of the next room is a toilet or bathroom (step S38). For example, if the room located at the next point on the travel path is a toilet or bathroom, the determination unit 53 stores the room where the point is located as a room adjacent to a toilet or bathroom.

[0123] Next, the determination unit 53 determines that the purpose of the previous room determined in step S31 is a toilet or bathroom (YES in step S39), and proceeds to step S40. On the other hand, the determination unit 53 determines that the purpose of the previous room determined in step S31 is not a toilet or bathroom (NO in step S39), and proceeds to step S41. If the room immediately preceding a point on the travel path is a toilet or bathroom, that point means that the user has left the toilet or bathroom. For example, the point in section J1 in Figure 27(A) indicates the point when the user has left the toilet or bathroom.

[0124] The path generation unit 58 generates a path to the adjacent room if the previous room's purpose is a toilet or bathroom (step S40). If the room where the previous point on the movement path is located is a toilet or bathroom, the path generation unit 58 connects the room where the point is located with a path to the adjacent room set in step S38. This makes it possible to set the room the user enters after leaving the toilet or bathroom as the same room they were in before entering the toilet or bathroom, as set in step S38. As shown in Figure 27(B), since room A, the room the user was in immediately before entering the toilet or bathroom, is set in step S38, the path generation unit 58 connects the point on the movement path at the time of exit with a path to room A, the room the user was in immediately before entering the toilet or bathroom, rather than the nearest room B (section J2). This forces the image processing device 50 to return to the room the user entered when leaving the toilet or bathroom, preventing the generation of an incorrect path like the one shown in Figure 27(A).

[0125] Next, if the image processing device 50 has completed all processing on the video captured by the camera 10 (YES in step S41), it proceeds to step S42. On the other hand, the image processing device 50 repeats the processing from step S31 until all processing on the video captured by the camera 10 is completed (NO in step S41).

[0126] Next, the path generation unit 58 sets the starting point of the generated path based on the room's purpose determined in step S31 (step S42). For example, the path generation unit 58 sets the starting point of the path to the location corresponding to the room whose purpose is determined to be an entrance.

[0127] Then, based on the processing results described above, the image processing unit 60 generates a tour image, which is a processed image containing multiple still images captured by the imaging device 10 (step S42). Specifically, as shown in Figure 28, the image processing unit 60 generates a tour image in which multiple still images are associated by associating the shooting positions estimated by the position estimation unit 56 with the shape estimation results by the shape estimation unit 55. In addition, as shown in Figure 29(A), the image processing unit 60 superimposes the path generated by the path generation unit 58 onto the multiple still images associated by their relative shooting positions, in addition to the configuration shown in Figure 28. Furthermore, as shown in Figure 29(B), the image processing unit 60 can indicate the starting point (S) of the path set in step S42 on the tour image.

[0128] In this way, the image processing device 50 estimates the shooting position of a high-resolution still image based on the movement path in the moving image captured by the imaging device 10. The image processing device 50 also generates a path indicating the order in which the images were taken, based on the estimated shooting position, the shape of the room captured in the image, and the movement path of the imaging device 10.

[0129] The tour images generated by the image processing device 50 can be viewed by viewers using the communication terminal 90. When the generation of the tour images is complete, the image processing device 50 notifies the communication terminal 90, which is the source of the tour photography request. The display control unit 93 of the communication terminal 90 displays the tour images provided by the image processing device 50 on the display 906. Viewers can use the communication terminal 90 to view virtual tours of real estate properties, etc., using the tour images provided by the image processing device 50.

[0130] ● Effects of the embodiment As explained above, the image processing system 1 can improve the accuracy of estimating the shooting position of captured images by accurately correlating high-resolution still images used to create virtual tours with the estimated shooting positions of multiple still images using moving images captured by the shooting device 10. Furthermore, the image processing system 1 can automatically create virtual tours with appropriate paths (high-quality virtual tours).

[0131] Furthermore, the image processing system 1 can efficiently acquire images for creating a virtual tour by using the shooting device 10 to switch in time-division mode between high-resolution still image capture for acquiring images for viewing and low-resolution video capture for position estimation.

[0132] Furthermore, the image processing system 1 uses a 360-degree camera to capture images of the interior of a designated location, such as a real estate property, in all directions at once. By using these natural images, free from errors due to parallax during shooting, for the virtual tour, it can create a natural virtual tour (a high-quality virtual tour). In addition, even when the camera 10 is fixed to a support member 20, such as a monopod, to prevent the photographer from appearing in the captured images, the image processing system 1 can create a virtual tour with improved position estimation accuracy.

[0133] ●Summary● As described above, the image processing method according to one embodiment of the present invention is an image processing method performed by an image processing device 50 that processes captured images taken in all directions within a predetermined location, and includes: a still image acquisition step of acquiring a plurality of still images taken at different shooting positions within the location; a moving image acquisition step of acquiring moving images taken while moving from a first point to a second point within the location; a position estimation step of estimating the relative shooting positions of the plurality of still images based on the acquired moving images; and an image processing step of generating a processed image (e.g., a tour image) including a plurality of still images associated based on the estimated shooting positions. As a result, the image processing method can provide a virtual tour with improved accuracy in estimating the shooting positions of captured images.

[0134] Furthermore, an image processing method according to one embodiment of the present invention further performs a shape estimation step of estimating the shape of the space (e.g., a room) within the base as seen in the moving image, and a path generation step of generating a path that links a plurality of still images based on the shooting position estimated by the position estimation step and the shape estimated by the shape estimation step. The position estimation step estimates the relative shooting positions of the plurality of still images based on the shape estimated by the shape estimation step, and the image processing step generates a processed image (e.g., a tour image) that includes paths corresponding to the plurality of still images. As a result, the image processing method can accurately link a plurality of still images with the shooting position of each still image based on the shooting position of the still images estimated using the moving image, the shape of the space seen in the captured image, and the generated path. The image processing method can then automatically create a virtual tour (high-quality virtual tour) with high positional accuracy and appropriate paths (routes).

[0135] Furthermore, in the image processing method according to one embodiment of the present invention, the moving image is an image with a lower resolution than the still image. Also, in the image processing method, the still image and the moving image are images captured by a 360-degree spherical imaging device. As a result, the image processing method can efficiently create a virtual tour with an appropriate path (a high-quality virtual tour).

[0136] Furthermore, an image processing system according to one embodiment of the present invention is an image processing system 1 comprising an image processing device 50 and a shooting device 10, wherein the shooting device 10 captures still images and moving images by switching between them in a time-division manner. The shooting device 10 also switches between capturing still images and moving images according to the movement status within the site. As a result, the image processing system 1 can efficiently acquire captured images for creating a virtual tour.

[0137] ●Additional Information● Each function of the embodiment described above can be realized by one or more processing circuits. Here, "processing circuit" in this embodiment includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), SOCs (System on a chip), GPUs (Graphics Processing Units), and conventional circuit modules designed to execute each of the functions described above.

[0138] Although an image processing method, program, image processing apparatus, and image processing system according to one embodiment of the present invention have been described so far, the present invention is not limited to the embodiments described above, and can be modified to the extent that a person skilled in the art can imagine, such as by adding, changing, or deleting other embodiments, and any embodiment that achieves the effects and advantages of the present invention is included within the scope of the present invention. [Explanation of Symbols]

[0139] 1. Image Processing System 10. Imaging device 11 Transmitter / Receiver 12 Operation reception section 13. Image capture control unit (an example of image capture control means) 14. Video Production Department 15 Still Photography Section 16. Motion detection unit 20 Support members 50 Image Processing Devices 51 Transmitting and receiving unit (Example of still image acquisition means, Example of moving image acquisition means) 54 Application determination section 55 Shape estimation section 56 Position Estimation Unit (An Example of Position Estimation Means) 57 Detection unit 58 Path generation unit (an example of a path generation means) 60 Image Processing Unit (An example of image processing means) 90 Communication terminals 91 Transmitter / Receiver 92 Reception Department 93 Display Control Unit (Example of Display Control Means)

Claims

1. An image processing method for generating a tour image that allows viewing each of the multiple spaces of a structure in all directions, A video acquisition step involves acquiring video images while moving from a first shooting position that captures a first space in all directions within a plurality of spaces of the structure to a second shooting position that captures a second space in all directions within a plurality of spaces of the structure, A position estimation step in which the relative positions of the first shooting position and the second shooting position are estimated based on the video image acquired in the video image acquisition step, A path generation step generates a path that associates the first shooting position and the second shooting position based on the position estimated in the position estimation step, An image processing method that performs this task.

2. A shape estimation step is performed to estimate the spatial shape of the structure as seen in the video. The position estimation step estimates the relative position between the first shooting position and the second shooting position based on the shape estimated by the shape estimation step. The image processing method according to claim 1, wherein the path generation step generates the path based on the relative positions of the first shooting position and the second shooting position estimated by the position estimation step, and the shape estimated by the shape estimation step.

3. A detection step is performed to detect the space to which a point on the movement path where the video image was captured belongs. The image processing method according to claim 2, wherein the path generation step, when a location where a moving image belonging to a plurality of spaces has been captured is detected, generates the path such that the location belongs to the nearest space.

4. A usage determination step is performed to determine the use of the aforementioned space. The image processing method according to claim 3, wherein the path generation step generates the path based on the determined use.

5. The image processing method according to claim 4, wherein the path generation step sets the starting point of the generated path based on the determined use.

6. A program that causes a computer to execute an image processing method to generate tour images that allow viewing each of the multiple spaces of a structure in all directions, A video acquisition step involves acquiring video images while moving from a first shooting position that captures a first space in all directions within a plurality of spaces of the structure to a second shooting position that captures a second space in all directions within a plurality of spaces of the structure, A position estimation step in which the relative positions of the first shooting position and the second shooting position are estimated based on the video image acquired in the video image acquisition step, A path generation step generates a path that associates the first shooting position and the second shooting position based on the position estimated in the position estimation step, A program that executes the command.

7. An image processing apparatus that generates a tour image for viewing each of the multiple spaces of a structure in all directions, A motion image acquisition means that acquires motion images while moving from a first shooting position that captures a first space in all directions among multiple spaces of the structure to a second shooting position that captures a second space in all directions among multiple spaces of the structure, A position estimation means that estimates the relative position between the first shooting position and the second shooting position based on the video image acquired by the video image acquisition means, A path generation means generates a path that associates the first shooting position and the second shooting position based on the position estimated by the position estimation means, An image processing device equipped with the following features.

8. An image processing system comprising the image processing device described in claim 7, a shooting device, and a communication terminal capable of communicating with the image processing device, The aforementioned communication terminal is An image processing system comprising display control means for displaying tour images generated by the aforementioned image processing device.

9. The image processing system according to claim 8, wherein the display control means indicates the starting point of the path on the tour image generated by the image processing device.

Images