Image processing method, program, image processing device, and image processing system

Abstract

To provide a virtual tour with improved accuracy in estimating the photographic position of a photographic image.SOLUTION: An image processing method, which can be executed by an image processing device 50 that performs processing on a photographic image captured in all directions within a predetermined base, includes: a still image acquisition step of acquiring a plurality of still images captured at different photographic positions within the base; a moving image acquisition step of acquiring a moving image captured while moving from a first point to a second point within the base; a position estimation step of estimating relative photographic positions of the plurality of still images based on the acquired moving image; and an image processing step of generating a processed image (for example, a tour image) including the plurality of still images associated with each other based on the estimated photographic positions.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present disclosure relates to an image processing method, a program, an image processing device, and an image processing system. [Background technology]

[0002] A system is known that distributes image data captured using an omnidirectional camera, allowing a user to view the situation at a remote location from another location. Omnidirectional images of a specific location can be viewed by the viewer in any direction, conveying realistic information. Such systems are used, for example, in the real estate industry, for online property viewings.

[0003] Furthermore, there are virtual tour services that interconnect images taken at multiple shooting positions of a real estate property, allowing users to feel as if they are walking around and viewing the interior of the property. To create such a virtual tour, it is important to connect the images taken at multiple shooting positions in the correct positional relationship. To create such a virtual tour, there is already known technology for estimating the position of captured images using acceleration information measured by an IMU (inertial measurement unit) (for example, Patent Document 1 and Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,375,306 [Patent Document 2] U.S. Patent No. 10,530,997 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional method of estimating position based on acceleration using an IMU leaves room for improvement in terms of the accuracy of position estimation for multiple captured images used to create a virtual tour. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the invention of claim 1 is an image processing method executed by an image processing device that processes images taken in all directions within a specified base, and includes the following steps: a still image acquisition step that acquires multiple still images taken at different shooting positions within the base; a moving image acquisition step that acquires moving images taken while moving from a first point to a second point within the base; an estimation step that estimates the relative shooting positions of the multiple still images based on the acquired moving images; and an image processing step that generates a processed image including the multiple still images associated based on the estimated shooting positions. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a virtual tour with improved accuracy in estimating the shooting positions of captured images. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the overall configuration of an image processing system. [Figure 2] FIG. 1 is a diagram illustrating an example of a spherical image captured by an imaging device. [Figure 3] (A) is a hemispherical image (front) taken with a camera, (B) is a hemispherical image (back) taken with a camera, and (C) is an image represented by equirectangular projection. [Figure 4] (A) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (B) is a diagram showing a spherical image. [Figure 5] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 6] 10 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. [Figure 7] FIG. 2 is a diagram illustrating an example of a state during shooting by the imaging device. [Figure 8] FIG. 1 is a diagram illustrating an example of a spherical image. [Figure 9] FIG. 10 is a diagram illustrating an example of a planar image converted from a spherical image. [Figure 10] 1A and 1B are schematic diagrams illustrating an example of an imaging device that can be applied to an image processing system. [Figure 11] FIG. 1 is a diagram illustrating an example of a photographed image photographed by a general photographing device. [Figure 12] 1A to 1C are schematic diagrams illustrating an example of processing executed by an image processing device. [Figure 13] FIG. 2 is a diagram illustrating an example of a hardware configuration of the imaging device. [Figure 14] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing device and a communication terminal. [Figure 15] FIG. 1 illustrates an example of a functional configuration of an image processing system. [Figure 16] 10 is a flowchart illustrating an example of tour photography processing by the photography device. [Figure 17] 10A and 10B are diagrams showing an example of a method for fixing the imaging device. [Figure 18] FIG. 10 is a diagram for explaining the differences between handheld shooting and fixed shooting. [Figure 19] FIG. 2 is a diagram for explaining switching between video and still image shooting by the imaging device. [Figure 20] FIG. 10 is a diagram showing an example of timing for switching between video shooting and still image shooting. [Figure 21] 10A and 10B are diagrams illustrating an example of a tour path in a virtual tour. [Figure 22] 10 is a flowchart illustrating an example of a process for generating a tour image by the image processing device. [Figure 23] 10A and 10B are schematic diagrams illustrating an example of a process for estimating the size of a room shape. [Figure 24] 10A and 10B are schematic diagrams illustrating an example of a process for determining a room in which a point on a movement path is located. [Figure 25] 10A and 10B are schematic diagrams illustrating an example of a process for determining a room in which a point on a movement path is located. [Figure 26] 10 is a schematic diagram for explaining an example of a path generated by a path generating unit. FIG. [Figure 27] 10A and 10B are schematic diagrams illustrating an example of a path that is generated when the room is used as a toilet or a bathroom. [Figure 28] FIG. 2 is a schematic diagram illustrating an example of a tour image generated by the image processing device. [Figure 29] 1A and 1B are schematic diagrams illustrating an example of a tour image generated by an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0010] ●Embodiment● ●Outline of the image processing system First, an outline of the configuration of an image processing system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the overall configuration of an image processing system. The image processing system 1 shown in Fig. 1 is a system that performs image processing on captured images to allow viewers to view the interior space of a structure such as a real estate property online.

[0011] 1, the image processing system 1 includes an image capturing device 10, an image processing device 50, and a communication terminal 90. The image capturing device 10, the image processing device 50, and the communication terminal 90 that constitute the image processing system 1 can communicate via a communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), or the like. Note that the communication network 100 may include not only wired communication networks but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] The image processing device 50 is a server computer that performs image processing on captured images of the interior space of a structure such as a real estate property, which is a predetermined base. The image processing device 50, for example, acquires captured images captured by the image capture device 10 and generates tour images for providing a virtual tour to a user using the acquired captured images. Here, a virtual tour is content that allows a user to view the interior of a structure such as a real estate property as if they were actually on-site. The tour images are generated using multiple captured images captured by the image capture device 10 and are images for viewing that allow the user to virtually move around the base depicted in the captured images by user operation.

[0013] The image processing device 50 may be configured by one server computer or multiple server computers. Although the image processing device 50 will be described as a server computer existing in a cloud environment, it may also be a server existing in an on-premise environment.

[0014] The image capturing device 10 is a special digital camera (spherical image capturing device) capable of capturing spherical (360°) images by capturing images of the interior space of a structure such as a real estate property at a capture location in all directions. The image capturing device 10 is used, for example, by a real estate agent who manages or sells real estate properties. The image capturing device 10 may also be a wide-angle camera or a stereo camera capable of capturing wide-angle images with a field of view equal to or greater than a predetermined value. A wide-angle image is generally an image captured using a wide-angle lens, which is an image captured with a lens that can capture a wider range than what the human eye can perceive. In other words, the image capturing device 10 is a capturing means capable of capturing images (spherical images, wide-angle images) captured with a lens whose focal length is shorter than a predetermined value. A wide-angle image generally refers to an image captured with a lens whose focal length is equal to or shorter than 35 mm when converted into a 35 mm film format.

[0015] The communication terminal 90 is a computer such as a smartphone that displays images processed by the image processing device 50 and allows viewers to view them. The communication terminal 90 is used, for example, by the same real estate agent as the image capture device 10. A dedicated application is installed on the communication terminal 90 for issuing image capture instructions to the image capture device 10 and viewing images provided by the image processing device 50. The communication terminal 90 may also be configured to issue image capture instructions and view images by accessing a dedicated website using a web browser, for example, without using a dedicated application. Alternatively, the image capture instructions and image viewing may be performed by different communication terminals 90.

[0016] The communication terminal 90 is not limited to a smartphone, but may be, for example, a PC, a tablet terminal, a wearable terminal, an HMD (head mounted display), or an IWB (Interactive White Board: an electronic whiteboard with a blackboard function that allows mutual communication), etc.

[0017] ○Outline of the imaging device○ Here, an overview of the imaging device 10 constituting the image processing system 1 will be described with reference to FIGS. 2 to 11. FIG. 2 is a diagram showing an example of a spherical image captured by the imaging device. The image shown in FIG. 2 is a spherical image captured by the imaging device 10 of a room in a real estate property, which is an example of the interior space of a structure. Spherical images are suitable for viewing real estate properties because they can capture the interior of a room in all directions. Spherical images come in various forms, but are often generated using the equirectangular (equirectangular projection) projection method described below. Images generated using this equirectangular projection have the advantages of appearing relatively natural because the outer shape of the image is rectangular, making it efficient and easy to store image data, and there is little distortion near the equator and no distortion of vertical straight lines.

[0018] ○How to generate spherical images Next, a method for generating a spherical image will be described with reference to Figs. 3 to 9. First, an outline of the process for generating a spherical image from an image captured by the image capturing device 10 will be described with reference to Figs. 3 and 4. Fig. 3(A) is a diagram showing a hemispherical image (front side) captured by the image capturing device, Fig. 3(B) is a hemispherical image (rear side) captured by the image capturing device, and Fig. 3(C) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 4(A) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(B) is a diagram showing a spherical image.

[0019] The photographing device 10 is provided with image sensors on both the front side (front side) and the back side (rear side). These image sensors (image sensors) are used in conjunction with optical components such as lenses that can capture hemispherical images (angle of view of 180° or more). The photographing device 10 can obtain two hemispherical images by capturing images of subjects around the user using the two image sensors.

[0020] 3(A) and (B), the images captured by the imaging element of the image capturing device 10 are curved hemispherical images (front and rear). The image capturing device 10 then combines the hemispherical image (front) with a hemispherical image (rear) flipped 180 degrees to create an equirectangular projection image EC as shown in FIG. 3(C).

[0021] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the imaging device 10 applies an equirectangular projection image EC to cover the spherical surface as shown in FIG. 4A, and creates a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 4B. In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Furthermore, the celestial sphere image CE may be a still image or a video. Furthermore, the conversion method is not limited to OpenGL ES, and any method capable of converting a hemispherical image into an equirectangular projection may be used, for example, a CPU operation or OpenCL operation.

[0022] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, the imaging device 10 displays a predetermined region T (hereinafter referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 5 and 6.

[0023] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined region when the celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of a user viewing the celestial sphere image CE displayed as a three-dimensional sphere. FIG. 5 represents the celestial sphere image CE as a three-dimensional sphere CS. If the celestial sphere image CE generated in this manner is a three-dimensional sphere CS, as shown in FIG. 5, the virtual camera IC is located inside the celestial sphere image CE. The predetermined region T in the celestial sphere image CE is the shooting region of the virtual camera IC and is specified by predetermined region information that indicates the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE. Zooming of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0024] The predetermined area image Q is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The following description will be given using the shooting direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0025] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 6. FIG. 6 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 6, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the shooting direction (ea, aa), becomes the center point CP(x, y) of the predetermined area T, which is the shooting area of ​​the virtual camera IC. As shown in FIG. 6, when the diagonal angle of view of the predetermined area T, represented by the angle of view α of the virtual camera IC, is α, the center point CP(x, y) becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the celestial sphere image CE. f represents the distance from the virtual camera IC to the center point CP(x, y). L represents the distance between any vertex of the predetermined area T and the center point CP(x, y) (2L is the diagonal). In FIG. 6, the trigonometric function shown in the following (Equation 1) generally holds.

[0026]

number

[0027] Next, a description will be given of a state when photographing is performed by the image capturing device 10, using FIG. 7. FIG. 7 is a diagram illustrating an example of a state when photographing is performed by the image capturing device. To capture an image that allows a panoramic view of a room such as a real estate property, it is preferable to install the image capturing device 10 at a position close to the height of the human eye. Therefore, as shown in FIG. 7, the image capturing device 10 is generally fixed to a support member 20 such as a monopod or a tripod when photographing. As described above, the image capturing device 10 is a celestial sphere image capturing device capable of capturing light rays in all directions around the entire periphery, and can also be said to capture an image on a unit sphere around the image capturing device 10 (a celestial sphere image CE). Once the photographing direction of the image capturing device 10 is determined, the coordinates of the celestial sphere image are determined. For example, in FIG. 7, point A is located at a distance (d, -h) from the center point C of the image capturing device 10. If the angle between the line segment AC and the horizontal direction is θ, the angle θ can be expressed by the following (Equation 2).

[0028]

number

[0029] Assuming that point A is at a depression angle θ, the distance d between points A and B can be expressed by the following (Equation 3) using the installation height h of the image capturing device 10.

[0030]

number

[0031] Here, a process of converting position information on a celestial sphere image into coordinates on a planar image converted from the celestial sphere image will be outlined. Fig. 8 is a diagram for explaining an example of a celestial sphere image. Fig. 8(A) is a diagram showing the hemispherical image shown in Fig. 3(A) by connecting points where the horizontal and vertical incident angles with respect to the optical axis are equal. Hereinafter, the horizontal incident angle with respect to the optical axis will be referred to as "θ", and the vertical incident angle with respect to the optical axis will be referred to as "φ".

[0032] Moreover, Fig. 9(A) is a diagram illustrating an example of an image processed by equirectangular projection. Specifically, the image shown in Fig. 8 is associated with a pre-generated LUT (Look Up Table) or the like, processed by equirectangular projection, and the processed images shown in Fig. 8(A) and Fig. 8(B) are combined, whereby the planar image shown in Fig. 9(A) corresponding to the celestial sphere image is generated by the imaging device 10. The equirectangular projection image EC shown in Fig. 3(C) is an example of the planar image shown in Fig. 9(A).

[0033] As shown in FIG. 9(A), in an image processed by equirectangular projection, latitude (θ) and longitude (φ) are orthogonal to each other. In the example shown in FIG. 9(A), the center of the image is (0,0), and the latitude direction is expressed as -90 to +90 and the longitude direction is expressed as -180 to +180, so that any position in the omnidirectional image can be indicated. For example, the coordinates of the upper left corner of the image are (-180, -90). Note that the coordinates of the omnidirectional image may be expressed in a format using 360 degrees as shown in FIG. 9(A), or may be expressed in radians or in pixel numbers as in real images. Furthermore, the coordinates of the omnidirectional image may be converted into two-dimensional coordinates (x, y) as shown in FIG. 9(B) and expressed.

[0034] Note that the synthesis process for the planar image shown in Fig. 9(A) or 9(B) is not limited to the process of simply consecutively arranging the hemispherical images shown in Fig. 8(A) and 8(B). For example, if the horizontal center of the celestial sphere image is not θ=180°, in the synthesis process, the image capturing device 10 first preprocesses the hemispherical image shown in Fig. 3(C) and arranges it at the center of the celestial sphere image. Next, the image capturing device 10 may divide the hemispherical image shown in Fig. 3(B) into left and right portions of the image to be generated, each of which has a size that allows the preprocessed images to be arranged on the left and right portions, and synthesize the hemispherical images to generate the equirectangular projection image EC shown in Fig. 3(C).

[0035] Furthermore, in the planar image shown in Fig. 9(A), the point corresponding to the pole (PL1 or PL2) of the hemispherical image (spherical image) shown in Fig. 8(A) and Fig. 8(B) is the line segment CT1 or CT2. This is because, as shown in Fig. 4(A) and Fig. 4(B), the celestial sphere image (for example, the celestial sphere image CE) is created by pasting the planar image (equirectangular projection image EC) shown in Fig. 9(A) onto a spherical surface by using OpenGL ES.

[0036] Examples of imaging devices applicable to image processing systems Next, an example of an image capturing device 10 applicable to the image processing system 1 according to the embodiment will be described with reference to FIGS. 10 and 11. FIG. 10 is a schematic diagram illustrating an example of an image capturing device applicable to the image processing system. FIG. 10(A) shows a special-purpose image capturing device equipped with multiple image capturing elements capable of generating a celestial sphere image using the above-described generation method. The special-purpose image capturing device uses a wide-angle lens or a fisheye lens with a wide angle of view and can capture an omnidirectional image by combining the outputs of the multiple image capturing elements. FIG. 10(B) shows a general-purpose image capturing device, which is a so-called ordinary camera. The general-purpose image capturing device is, for example, a regular digital camera or a mobile device such as a smartphone equipped with a camera. A photographer takes pictures by holding the general-purpose image capturing device in their hand and rotating it. The general-purpose image capturing device can capture an omnidirectional image by synthesizing the obtained images. Both the special-purpose image capturing device and the general-purpose image capturing device generate a final captured image by stitching multiple captured images using image processing (stitching process). Note that it is preferable that the optical centers of the image capturing devices 10 used to capture multiple captured images are the same.

[0037] FIG. 11 is a diagram illustrating an example of an image captured by a general camera. FIG. 11 shows an image captured when a photographer takes a picture while holding a general camera as shown in FIG. 10(B) and rotating it. A general camera has a small angle of view (generally less than 100 degrees), which makes it difficult to capture the top and bottom poles, as shown in FIG. 11. Furthermore, when the photographer rotates the camera, the optical center of the camera may shift, causing parallax during shooting and making it prone to unnatural errors such as steps in the stitching process. That is, although both a special camera as shown in FIG. 10 and a general camera are applicable as the camera 10 in this embodiment, it is preferable to use a special camera as shown in FIG. 10(A). By using a special camera (spherical camera) as shown in FIG. 10(A), the image processing system 1 can use natural, high-quality spherical images without any distortion as images used in virtual tours that require high quality, such as for advertisements. In the following description, the image capturing device 10 will be described as a special image capturing device (omnidirectional image capturing device).

[0038] ○Outline of image processing device○ Next, an outline of the processing executed by the image processing device 50 will be described with reference to Fig. 12. Fig. 12 is a schematic diagram for explaining an example of the processing executed by the image processing device. Fig. 12 shows the relationship between the shooting positions of images shot by the image shooting device 10 and paths that associate multiple shot images.

[0039] The image processing device 50 estimates the shooting positions (FIG. 12(A)) of the captured images acquired from the image capturing device 10 using techniques such as visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion). The estimated shooting positions are expressed as relative positions of the respective shooting positions of the multiple captured images, as shown in FIG. 12(B). Furthermore, the estimated shooting positions are expressed in a single coordinate system for each shooting position.

[0040] In addition to estimating the shooting positions, the image processing device 50 can also restore the route traveled when the captured images were taken, and can also restore the order in which each shooting position was visited and the route taken to capture the images. The image processing device 50 generates a path (tour path) based on the estimated shooting positions and the route traveled when the images were taken. The tour path indicates the connection relationship between multiple captured images on a tour image. As shown in FIG. 12(C), a virtual tour using captured images can be created by connecting the captured images taken at each estimated shooting position and allowing users to move between the connected captured images.

[0041] ●Hardware configuration Next, the hardware configuration of each device or terminal constituting the image processing system according to the embodiment will be described with reference to Figures 13 and 14. Note that components may be added or deleted from the hardware configurations shown in Figures 13 and 14 as needed.

[0042] ○Hardware configuration of the imaging device○ First, the hardware configuration of the image capturing device 10 will be described using Fig. 13. Fig. 13 is a diagram showing an example of the hardware configuration of an image capturing device. In the following, the image capturing device 10 is assumed to be an omnidirectional (omnidirectional) image capturing device using two image capturing elements, but the number of image capturing elements may be two or more. Furthermore, the image capturing device does not necessarily have to be a device dedicated to omnidirectional image capturing; an omnidirectional image capturing unit may be attached to a regular digital camera, smartphone, or the like to have substantially the same functions as the image capturing device 10.

[0043] As shown in FIG. 13, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F (Interface) 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0044] Of these, imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) each having a field angle of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each lens. Imaging elements 103a and 103b each include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by lenses 102a and 102b into image data in the form of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set.

[0045] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0046] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data such as that shown in Figure 3(C).

[0047] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0048] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / minute).

[0049] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. Although the imaging device 10 is not provided with a display unit in this embodiment, a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0050] The CPU 111 controls the overall operation of the image capturing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0051] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0052] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0053] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0054] The electronic compass 118 calculates the orientation of the image capture device 10 from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) conforming to the Exif standard and is used for image processing such as image correction of captured images. The related information also includes data such as the capture date and time of the image and the data size of the image data. The gyro sensor 119 detects angle changes (roll angle, pitch angle, and yaw angle) associated with the movement of the image capture device 10. The angle changes are an example of related information (metadata) conforming to the Exif standard and are used for image processing such as image correction of captured images. The acceleration sensor 120 detects acceleration in three axial directions. The image capture device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (the image capture device 10) based on the acceleration detected by the acceleration sensor 120. The inclusion of the acceleration sensor 120 in the image capture device 10 improves the accuracy of image correction. The network I / F 121 is an interface for data communication using a communication network 100 such as the Internet via a router or the like.

[0055] ○Hardware configuration of image processing device○ First, the hardware configuration of the image processing device 50 will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the hardware configuration of an image processing device. Each piece of hardware configuration of the image processing device 50 is indicated by a reference numeral in the 500 series. The image processing device 50 is constructed by a computer, and as shown in Fig. 14, includes a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0056] Of these, the CPU 501 controls the overall operation of the image processing device 50. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The display 506 may also be a touch panel display equipped with input means. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory. The network I / F 509 is an interface for data communication using the communication network 100. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 14 .

[0057] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 511 and the pointing device 512, but also a touch panel, a voice input device, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. The removable recording medium is not limited to a DVD-RW, but may also be a DVD-R or a Blu-ray (registered trademark) Disc, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0058] ○Hardware configuration of communication terminal○ Fig. 14 also shows an example of the hardware configuration of the communication terminal 90. Each piece of hardware configuration of the communication terminal 90 is indicated by a reference numeral in the 900 series in parentheses. The communication terminal 90 is constructed by a computer, and as shown in Fig. 14, has the same configuration as the image processing device 50, so a description of each piece of hardware configuration will be omitted. In addition to the same configuration as the image processing device 50, the communication terminal 90 also has a short-range communication circuit 917 and an antenna 917a for the short-range communication circuit 917. The short-range communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0059] Each of the above programs may be recorded on a computer-readable recording medium as an installable or executable file and distributed. Examples of recording media include CD-Rs (Compact Disc Recordables), DVDs (Digital Versatile Disks), Blu-ray Discs, SD cards, and USB memory. The recording media may also be provided domestically or internationally as program products. For example, the image processing device 50 executes the program according to the present invention to realize the image processing method according to the present invention.

[0060] ●Function configuration Next, the functional configuration of the image processing system according to the embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of the functional configuration of the image processing system. Fig. 15 shows devices or terminals shown in Fig. 1 that are related to the processing or operation described below.

[0061] ○ Functional configuration of the imaging device ○ First, the functional configuration of the photographing device 10 will be described with reference to Fig. 15. The photographing device 10 has a transmitting / receiving unit 11, an operation receiving unit 12, a photographing control unit 13, a video photographing unit 14, a still image photographing unit 15, a movement detection unit 16, a communication unit 17, and a storage / readout unit 19. Each of these units is a function or means realized when any of the components shown in Fig. 13 operates in response to an instruction from the CPU 111 in accordance with a photographing device program loaded from the SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 13. The storage unit 1000 stores the GUID (Globally Unique Identifier) ​​of the photographing device itself.

[0062] The transmitting / receiving unit 11 is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. Furthermore, the transmitting / receiving unit 11 uses the network I / F 121 to communicate data with other devices or terminals via the communication network 100.

[0063] The operation reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user who is the photographer.

[0064] The shooting control unit 13 is mainly realized by the processing of the CPU 111 on the imaging unit 101, image processing unit 104, and imaging control unit 105, and captures an image of a subject such as a landscape and acquires captured image data. The shooting control unit 13 performs shooting by, for example, switching between video shooting by the video shooting unit 14 and still image shooting by the still image shooting unit 15 in a time-division manner.

[0065] The video shooting unit 14 is mainly realized by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and performs video shooting using the imaging device 10. The video shooting unit 14 shoots moving images while moving within a structure such as a real estate property, which is a predetermined base, for example. The video shooting unit 14 shoots video in successive frames at low resolution while the photographer holding the imaging device 10 is moving, and stores the captured image data in the storage unit 1000. The video shooting unit 14 shoots video while the photographer holding the imaging device 10 is moving from a first point to a second point within a real estate property, which is a predetermined base, for example.

[0066] The still image capturing unit 15 is mainly realized by the processing of the imaging unit 101, and captures an image of a subject such as a landscape, and takes a still image using the imaging device 10. The still image capturing unit 15 captures a plurality of still images taken at different shooting positions within a structure such as a real estate office, which is a predetermined base, for example. The still image capturing unit 15 captures still images (photographs) with a higher resolution than the video captured by the video capturing unit 14, for example, and stores the captured image data in the storage unit 1000. The still image captured by the still image capturing unit 15 may be a single frame image, or may be an HDR (High Dynamic Range) image composed of multiple images.

[0067] Here, it is preferable that the still images captured by the still image capturing unit 15 have a high resolution of, for example, 4K or higher. On the other hand, the moving images captured by the moving image capturing unit 14 are images used for position estimation, and have a lower resolution than the still images because it is sufficient for the subject appearing in the moving images to be identifiable. The moving images may have a resolution of, for example, about 480p or lower. By capturing low-resolution moving images, the imaging device 10 can reduce the overall data volume in tour shooting.

[0068] The movement detection unit 16 is mainly realized by the processing of the CPU 111 on the gyro sensor 119 and the acceleration sensor 120, and detects the movement state of the image capturing device 10. For example, the movement detection unit 16 detects whether the photographer holding the image capturing device 10 is moving (in a moving state) or standing still (in a stationary state) while the image capturing device 10 is capturing an image.

[0069] The communication unit 17 is mainly realized by the processing of the CPU 111 on the input / output I / F 116 or the short-range communication circuit 117, and transmits and receives various data or information to and from other devices or terminals. The communication unit 17 uses the input / output I / F 116 to perform data communication with the communication terminal 90 via various cables, for example. The communication unit 17 also uses the short-range communication circuit 117 to perform data communication with the communication terminal 90 using short-range wireless communication technology.

[0070] The storage / readout unit 19 is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000. The storage unit 1000 also stores image data captured by the video capture unit 14 and the still image capture unit 15. The image data stored in the storage unit 1000 is associated with the capture time of the captured image as metadata.

[0071] ○Functional configuration of image processing device○ First, the functional configuration of the image processing device 50 will be described with reference to Fig. 15. The image processing device 50 has a transmitting / receiving unit 51, a receiving unit 52, a determining unit 53, a use determining unit 54, a shape estimating unit 55, a position estimating unit 56, a detecting unit 57, a path generating unit 58, an image processing unit 60, and a storing / reading unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 14 operates in response to an instruction from the CPU 501 in accordance with a program for the image processing device loaded from the HD 504 onto the RAM 503. The image processing device 50 also has a storage unit 5000 constructed by the ROM 502, RAM 503, and HD 504 shown in Fig. 14.

[0072] The transmitter / receiver 51 is mainly realized by processing of the CPU 501 on the network I / F 509, and transmits and receives various data or information to and from other devices or terminals via the communication network 100. The transmitter / receiver 51 receives (acquires), for example, moving images captured by the camera device 10 from the camera device 10 or the communication terminal 90. The transmitter / receiver 51 also receives (acquires), for example, still images captured by the camera device 10 from the camera device 10 or the communication terminal 90.

[0073] The reception unit 52 is realized mainly by the processing of the CPU 501 on the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user. The determination unit 53 is realized by the processing of the CPU 501, and makes various determinations.

[0074] The use determination unit 54 is realized by processing of the CPU 501, and determines the use of the space shown in the captured image based on the captured image showing the interior space of the structure in all directions. In other words, the use is the classification, genre, or purpose of use of the space actually captured by the image capture device 10. If a real estate property was photographed as the base for the image capture by the image capture device 10, the use of the room that is the space shown in the captured image could be, for example, a toilet, bathroom, or entrance.

[0075] The shape estimation unit 55 is realized by processing of the CPU 501, and estimates the shape of the space shown in the captured image based on the captured image showing the interior space of the structure in all directions.

[0076] The position estimation unit 56 is realized by processing of the CPU 501, and estimates the relative shooting position of a still image acquired from the image capture device 10 based on a moving image acquired from the image capture device 10. The position estimation unit 56 estimates the relative position of a still image from a moving image, which is a low-resolution continuous frame, using a technique such as visual SLAM or SfM. This technique such as visual SLAM or SfM can calculate the positions of feature points on an image, the position and parameters of the camera, from multiple images. However, the calculated position is relative, and a reference position is required to obtain an absolute position.

[0077] The detection unit 57 is realized by processing of the CPU 501, and detects shooting points of the moving image that belong to the spatial shape estimated by the shape estimation unit 55. The shooting points of the moving image are, for example, coordinate values ​​obtained by restoring the movement path when the moving image was shot.

[0078] The path generation unit 58 is realized by processing of the CPU 501, and generates paths indicating the connection relationships between a plurality of still images based on the shooting positions estimated by the position estimation unit 56. For example, the path generation unit 58 generates paths that associate the nearest still images among the plurality of still images with each other based on the estimated shooting positions.

[0079] The image processing unit 60 is realized by processing of the CPU 501, and performs image processing to generate tour images for the virtual tour based on the shooting positions estimated by the position estimation unit 56. The image processing unit 60 generates, for example, a tour image, which is a processed image including a plurality of still images associated based on the estimated shooting positions. The image processing unit 60 determines the positions at which the still images should be placed in the tour image based on the estimated shooting positions of the still images, the shooting times of the still images, and the paths (tour paths) generated by the path generation unit 58.

[0080] The storage / readout unit 59 is mainly realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads out various data (or information) from the storage unit 5000. The storage unit 5000 also stores tour images created by the image processing unit 60.

[0081] ○Functional configuration of communication terminal○ Next, the functional configuration of the communication terminal 90 will be described with reference to Fig. 15. The communication terminal 90 has a transmitting / receiving unit 91, a receiving unit 92, a display control unit 93, a communication unit 94, and a storage / reading unit 99. Each of these units is a function or means realized when any of the components shown in Fig. 15 operates in response to an instruction from the CPU 901 in accordance with a communication terminal program loaded from the HD 904 onto the RAM 903. The communication terminal 90 also has a storage unit 9000 constructed by the ROM 902, RAM 903, and HD 904 shown in Fig. 15.

[0082] The transmitting / receiving unit 91 is mainly realized by the processing of the CPU 901 for the network I / F 909, and transmits and receives various data or information to and from other devices or terminals via the communication network 100.

[0083] The reception unit 92 is mainly realized by the processing of the CPU 901 on the keyboard 911 or the pointing device 912, and receives various selections or inputs from the user.

[0084] The display control unit 93 is mainly realized by the processing of the CPU 901, and causes various images, characters, etc. to be displayed on the display 906. The display control unit 93 accesses the image processing device 50 using, for example, a web browser or a dedicated application, and causes the display 906 to display an image corresponding to data distributed from the image processing device 50.

[0085] The communication unit 94 is mainly realized by the processing of the CPU 901 on the external device connection I / F 908 or the short-range communication circuit 917, and transmits and receives various data or information to and from other devices or terminals. The communication unit 94 uses the external device connection I / F 908, for example, to perform data communication with the image capturing device 10 via various cables, etc. The communication unit 94 also uses the short-range communication circuit 917 to perform data communication with the image capturing device 10 using short-range wireless communication technology.

[0086] The storage / readout unit 99 is mainly realized by the processing of the CPU 901 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0087] Processing or operation of the embodiment ○Tour photography processing○ Next, the processing or operation of the image processing system according to the embodiment will be described with reference to Figs. 16 to 29. In the following description, an example of a room in a real estate property will be shown as an example of the interior space of a structure that is a predetermined base. First, with reference to Figs. 16 to 20, the process of acquiring photographed images used to generate tour images by the image processing device 50 will be described. Fig. 16 is a flowchart showing an example of tour photographing processing by the photographing device.

[0088] First, the photographing device 10 starts a tour photographing process in response to a predetermined request from the communication terminal 90 (step S11). Here, tour photographing refers to photographing within a predetermined location in order to provide a virtual tour. Specifically, the communication terminal 90 executes the tour photographing function by starting and running an installed dedicated application or by accessing a predetermined website using a web browser. Then, the communication unit 94 of the communication terminal 90 transmits a tour photographing request to the photographing device 10. The photographing device 10 starts the tour photographing process in response to the tour photographing request received by the communication unit 17.

[0089] Next, the video shooting unit 14 of the photographing device 10 starts shooting video while moving around the base holding the photographing device 10 (step S12). Specifically, when tour shooting starts, the shooting control unit 13 of the photographing device 10 requests the video shooting unit 14 to start shooting video. Then, in response to the request from the shooting control unit 13, the video shooting unit 14 starts shooting moving images.

[0090] Here, a photographing method using the photographing device 10 will be described with reference to Figures 17 and 18. Figure 17 is a diagram showing an example of a method for fixing the photographing device. The photographing device 10 is preferably fixed to a support member 20 such as a monopod 20a as shown in Figure 17(A) or a tripod 20b as shown in Figure 17(B) so that the photographer is not photographed. The photographing device 10 is also fixed at a height desired for viewing images. For example, the photographing device 10 is preferably placed at a height that is within the line of sight of a person so that a real estate property appears natural.

[0091] Fig. 18 is a diagram for explaining the differences between handheld and fixed shooting. Fig. 18 shows the difference in physical movement between shooting with the camera 10 held in the photographer's hand (handheld shooting) and shooting with the camera 10 fixed to a monopod 20a (fixed shooting). As shown in Fig. 17, when shooting on the move, the simplest method is to move the camera 10 while it is attached to a support member 20 such as a monopod 20a, but there are differences in physical movement between handheld and fixed shooting. Specifically, for example, attaching a relatively long and large monopod 20a increases the physical moment, and the movement of the camera itself is attenuated.

[0092] In Patent Document 1, the shooting position is estimated by position estimation using a sensor such as an IMU and PDR. However, when a monopod 20a is attached, as shown in Fig. 18, variations in acceleration and angular velocity are attenuated, which causes a problem that the shooting position estimation method described in Patent Document 1 does not achieve sufficient accuracy. Therefore, the image processing system 1 employs an image processing method described below so as not to affect processing accuracy even when shooting with the image capture device 10 fixed to a support member 20 such as a monopod 20a so that the photographer does not appear in the image.

[0093] Returning to FIG. 16, when the photographing device 10 detects a change in the photographing mode (YES in step S13), the process proceeds to step S14. In this case, the photographing device 10 switches from the video shooting mode to the still image shooting mode. For example, when the shutter button on the operation unit 115 of the photographing device 10 is pressed, the photographing device 10 switches from the video shooting mode to the still image shooting mode. The photographing device 10 detects a change in the photographing mode. The photographing device 10 may also be configured to switch from the video shooting mode to the still image shooting mode when the movement detection unit 16 detects that the photographing device 10 has stopped (a stationary state). On the other hand, when the photographing unit 14 of the photographing device 10 does not detect a change in the photographing mode (NO in step S13), the photographing device 10 continues photographing the video in step S12.

[0094] Next, the video shooting unit 14 of the imaging device 10 stops video shooting in response to the detection of the change in shooting mode in step S13 (step S14). Then, the still image shooting unit 15 performs still image shooting in response to the user pressing the shutter button of the operation unit 115 (step S15).

[0095] Next, similarly to the process of step S13, if the photographing device 10 detects a change in the photographing mode (YES in step S16), the process proceeds to step S18. In this case, the photographing device 10 switches from the still image photographing mode to the video photographing mode. For example, when the movement detection unit 16 detects that movement has resumed, the photographing device 10 switches from the still image photographing mode to the video photographing mode. Furthermore, the photographing device 10 may be configured to automatically switch from the still image photographing mode to the video photographing mode after the still image photographing unit 15 has taken a still image.

[0096] The video shooting unit 14 resumes video shooting in response to detecting the switch in shooting mode in step S16 (step S17). As in step S12, the video shooting unit 14 continues shooting while moving around the base holding the camera device 10. Then, when the tour shooting is finished (YES in step S18), the camera device 10 ends the process. The end of the tour shooting is determined, for example, by a predetermined operation by the photographer on the camera device 10 or the communication terminal 90. On the other hand, when the tour shooting is not finished (NO in step S18), the camera device 10 repeats the process from step S13 to continue the tour shooting.

[0097] On the other hand, in step S16, if the photographing device 10 does not detect a change in the photographing mode (NO in step S16), the photographing device 10 proceeds to step S19. As in step S18, if the tour photographing has ended (YES in step S19), the photographing device 10 ends the process. On the other hand, if the tour photographing has not ended (NO in step S19), the photographing device 10 repeats the process from step S19, and continues still image photographing by the still image photographing unit 15.

[0098] Here, video shooting and still image shooting in the imaging device 10 will be described with reference to Figs. 19 and 20. Fig. 19 is a diagram for explaining switching between video shooting and still image shooting by the imaging device. As described above, the imaging device 10 is used for two purposes: video shooting and still image shooting. Of these, video shooting is performed for the purpose of position estimation, and still image shooting is performed for the purpose of obtaining images for viewing. Furthermore, video shooting is performed while moving, and still image shooting is performed while stationary.

[0099] Here, video and still image capture require different imaging specifications depending on the purpose of each capture. In the case of video capture, continuous images (continuous frames) are acquired at a high frame rate for use in position estimation, but high resolution and color layers are not required for the captured images; low resolution and grayscale are sufficient. In addition, in the case of still image capture, acquisition of continuous frames is not required, but high resolution, color information (RGB), and high dynamic range are required. Furthermore, video capture is performed while moving, while still image capture is performed while stationary.

[0100] 20 is a diagram showing an example of the timing of switching between moving image shooting and still image shooting, and shows a time chart of switching between moving image shooting and still image shooting in the processing shown in FIG.

[0101] The image capturing device 10 switches between video capture and still image capture in a time-division manner. As described above, video capture is performed to acquire images for position estimation, so it is necessary to capture consecutive frames. On the other hand, still image capture is performed to acquire images for viewing in a virtual tour, so it is not necessary to capture consecutive frames; a configuration in which a single image is acquired may suffice, but an image may also be acquired by high dynamic range synthesis using images from multiple frames.

[0102] Still image capture is an item that should reflect the photographer's position, and is assumed to be performed by some explicit action of the photographer. For example, still image capture is performed by switching from video capture when the shutter button on the operation unit 115 of the image capture device 10 is pressed. In this case, the shutter pressed by the operator may be the shutter button on the image capture device 10 itself, or may be a shutter based on operation of a remotely operable app or controller. For example, when photographing a real estate property, it is preferable that the photographer not appear in the captured image. Therefore, when operating the shutter button on the image capture device 10 itself, it is preferable that the image capture device 10 captures and stores a still image a predetermined time after the shutter button is pressed. This allows the photographer to avoid appearing in the captured image by moving outside the capture range after pressing the shutter button. Furthermore, when performing remote operation, it is preferable that the photographer perform the operation from outside the capture range.

[0103] As a method for switching from video recording to still image recording, a configuration may be adopted in which when the camera 10 fixed to the support member 20 is placed in a desired position or when the photographer himself stands still, the movement detection unit 16 detects the stationary state and switches to still image recording, or a configuration may be adopted in which the camera automatically switches to still image recording when a predetermined time has passed since video recording began. These methods have the advantage that tour recording can be performed smoothly because there is no need for the operator to operate the shutter.

[0104] On the other hand, since video shooting is an item that is shot regardless of the photographer's intention, it is preferable that the shooting be performed automatically. For example, video shooting is automatically performed by switching from still image shooting after still image shooting has been performed. Furthermore, in order to reduce the amount of data of the captured images, video shooting may be configured to be switched from still image shooting when the movement detection unit 16 detects that movement has resumed.

[0105] In this way, the image capture device 10 can efficiently acquire captured images for generating tour images by chronologically switching between video capture, which captures low-resolution video images for position estimation, and still image capture, which captures high-resolution still images for viewing in the virtual tour. Note that, although the above description has been given of a configuration in which video capture is performed at the start of tour capture, the image capture device 10 may also be configured to first capture still images at the start of tour capture.

[0106] The photographing device 10 uploads captured moving images and still images to the image processing device 50 via the communication terminal 90 as needed. The image processing device 50 temporarily stores the moving images and still images transmitted from the photographing device 10 in the storage unit 5000 for generating tour images. Note that the photographing device 10 may be configured to upload the moving images and still images directly to the image processing device 50 without going through the communication terminal 90.

[0107] ○Tour image generation processing○ Next, the process of generating tour images using images captured by the image capturing device 10 will be described with reference to FIGS.

[0108] FIG. 21 is a diagram illustrating an example of a tour path in a virtual tour. The tour path is preferably generated so as to be equivalent to the adjacency relationship between rooms, which are the internal spaces of an actual real estate property. For example, if rooms A and B are adjacent and can be accessed by a door, the still images taken in rooms A and B should be connected by a path. On the other hand, if rooms A and C are adjacent but cannot be accessed by a door, the still images taken in rooms A and C should not be connected by a path. Therefore, when automatically generating a virtual tour, it is necessary to automatically determine which captured images should be connected by a path, using the capture positions of the still images and the route between them.

[0109] As shown in FIG. 21, a tour path is basically generated by connecting still images in the order they were taken. For example, in the example shown in FIG. 21(A), paths are connected in the order they were taken, generating a path of "Room A ⇔ Room B ⇔ Room C." On the other hand, in the example shown in FIG. 21(B), even though rooms B and C are consecutively taken, they are not physically connected, so in the actual space, the user must first return to room A before entering room C. In such a case, simply connecting paths in the order they were taken would result in connecting rooms that should not be accessible, resulting in a tour that differs from reality. For this reason, as shown in FIG. 21(B), taking into account the structure of the actual space, a path connecting rooms B and C is generated that again passes through room A.

[0110] In this way, the image processing device 50 generates paths using an algorithm that takes into account the location of each shooting point on the movement route during shooting, so that paths can be correctly connected in the generated tour image. Below, the algorithm for automatically generating paths in the image processing device 50 will be described in detail.

[0111] Fig. 22 is a flowchart showing an example of a process for generating tour images by an image processing device. The image processing device 50 executes the process for generating tour images using moving images and still images that are captured by the image capture device 10. Note that the description will be given assuming that the image processing device 50 has previously acquired moving images and still images captured by the image capture device 10. At this time, the transmitter / receiver 51 of the image processing device 50 receives (acquires) moving images and still images transmitted from the image capture device 10. In addition, in the examples shown in Figs. 23 to 27, it is assumed that the still images were captured of rooms A, B, and C at the capturing positions ("1 to 3" in each drawing) within rooms A, B, and C, respectively.

[0112] First, the purpose determination unit 54 of the image processing device 50 determines the purpose of the room shown in the acquired video (step S31). Specifically, the purpose determination unit 54 determines the purpose (genre) of the room shown in the video using general machine learning or parameters such as feature quantities of the video. Here, the determined purpose of the room is, for example, a toilet, a bathroom, or an entrance.

[0113] Next, the shape estimation unit 55 estimates the shape of the room shown in the acquired video (step S32). Specifically, the shape estimation unit 55 detects straight lines of the subject shown in the video, finds the vanishing points of the detected straight lines, and estimates the shape of the room shown in the video by a method of estimating the structure of the room from boundaries of the floor, walls, ceiling, etc., a method using three-dimensional information acquired from the image capture device 10, or a method using general machine learning, and performs three-dimensional reconstruction to acquire the shape of the room as seen from above.

[0114] The shape estimation unit 55 also estimates the size of the room shape whose structure was estimated in step S32 (step S33). Specifically, the shape estimation unit 55 adjusts the size of the estimated room shape to match the coordinate system of each point in the video on the movement path. As shown in FIG. 23(A), when the room shape estimated in step S32 is restored from a single image, the absolute size is unknown. As shown in FIG. 23(B), in order to position each room shape at the correct size on the movement path during video shooting, the shape estimation unit 55 determines the size of each room shape in the coordinate system of each point in the video on the movement path during video shooting. The shape estimation unit 55 estimates the size of each room shape using the shooting height during video shooting and the average size that eliminates the gap between adjacent room shapes. In this way, by having the shape estimation unit 55 estimate the shape and size of the room depicted in the video, the position estimation unit 56 estimates the relative shooting positions of the video and the multiple still images captured in a time-division manner.

[0115] Next, the detection unit 57 determines which room shape, among the room shapes estimated in steps S32 and S33, each point on the moving image on the moving path is located in (step S34). The detection unit 57 detects, from the coordinate values, which room, among the room shapes arranged in step S33, each point on the moving path is located inside. As a result, the detection unit 57 detects which room each point on the moving path was located inside. For example, as shown in FIG. 24(B), the image processing device 50 uniquely determines which room each point is located inside based on the coordinate values ​​on the moving path when the restored moving image was captured. In the example shown in FIG. 24(B), it is determined that the points on path R1 are located inside room A, the points on path R2 are located inside room B, and the points on path R3 are located inside room C.

[0116] Furthermore, if the detection unit 57 detects that a point in the moving image on the movement path does not exist within a single room shape (NO in step S34), it proceeds to step S35. When a point on the movement path does not belong to a single room, as in the section S1 shown in FIG. 25(A), that is, when the point on the movement path is located within multiple room shapes (room A and room B in the example of FIG. 25), it is not possible to uniquely determine which room the point was located in at the time of image capture. Also, when a point on the movement path is not located within any room shape, as in the section S2 shown in FIG. 25(A), it is not possible to uniquely determine which room the point was located in at the time of image capture. This occurs due to errors in the reconstruction of the movement path, errors in the estimation of the room shape, and errors in the size of the room shape.

[0117] Therefore, when a point in the moving image on the movement path does not exist within a single room shape, the detection unit 57 determines that the room in which the point on the movement path closest to the target point is located is the room in which the target point exists, in order to uniquely determine which room shape the target point exists in (step S35). The detection unit 57 calculates the points on the movement path that are closest to the points in sections S1 and S2 in Figure 25(A), and determines that the rooms in which the points on the movement path that are closest to each other are located, such as sections M1 and M2 in Figure 25(B), are the rooms in which the target point is located.

[0118] On the other hand, in step S34, if the detection unit 57 detects that each point of the moving image on the movement path exists within a single room shape (YES in step S34), the process proceeds to step S36.

[0119] Next, the path generation unit 58 generates a path connecting the room in which a point in the video image on the movement path exists with the room in which the next point exists (step S36). Since the processing in steps S34 and S35 uniquely determines which room shape each point on the movement path is located within, the path generation unit 58 connects the room in which a certain point exists with the room in which the next point exists as the next room for each point on the movement path with a path. In the example of FIG. 26, the path generation unit 58 connects room B, which was entered after room A, with a path (N1). Furthermore, even in the movement of returning from room B to room A and then entering room C, since there is a section of room A (N2 and N3) between shooting position 2 and shooting position 3, the path generation unit 58 can correctly connect the path from room A to room C based on the estimation results in steps S32 and S33. Finally, the path generation unit 58 can generate a path of the route "R1⇔R2⇔R1⇔R3" as shown in FIG. 26. That is, a path of "room A⇔room B" and a path of "room A⇔room C" are generated. That is, by generating paths connecting the rooms, the path generation unit 58 can associate the nearest still images among the multiple still images taken of each room with each other.

[0120] Next, if the use of the next room determined in step S31 is a toilet or a bathroom (YES in step S37), the determination unit 53 shifts the process to step S38. On the other hand, if the use of the next room determined in step S31 is not a toilet or a bathroom (NO in step S37), the determination unit 53 shifts the process to step S39.

[0121] In most homes, there is only one room leading to a toilet or bathroom, i.e., there is only one room to return to, and this takes advantage of this to prevent incorrect paths from being created. For example, as shown in FIG. 27(A), if there are many overlapping areas due to errors in the above-mentioned processing, a path may be generated that connects room A to room B via the toilet or bathroom. In the example of FIG. 27(A), based on the determination made in step S35 based on the point on the nearest movement path, the room where the overlapping point in section J1 exists is determined to be room B, and a path connecting the toilet or bathroom is generated. However, in real homes, it is rare for a toilet or bathroom to lead to two different rooms.

[0122] Therefore, if the use of the next room is a toilet or a bathroom, the determination unit 53 sets the current room as a room adjacent to the next room (step S38). For example, if the use of a room where a point next to a certain point on the movement route is located is a toilet or a bathroom, the determination unit 53 stores the room where the certain point exists as a room adjacent to the toilet or the bathroom.

[0123] Next, if the use of the previous room determined in step S31 is a toilet or a bathroom (YES in step S39), the judgment unit 53 shifts the process to step S40. On the other hand, if the use of the previous room determined in step S31 is not a toilet or a bathroom (NO in step S39), the judgment unit 53 shifts the process to step S41. If the room previous to a certain point on the movement path is a toilet or a bathroom, that point means that the toilet or bathroom has been left. For example, the point in section J1 in Figure 27(A) indicates the point when the toilet or bathroom has been left.

[0124] If the previous room is used as a toilet or bathroom, the path generation unit 58 generates a path to the room adjacent to the previous room (step S40). If the room where the previous point on the movement path is located is a toilet or bathroom, the path generation unit 58 connects the room where the point is located to the adjacent room set in step S38 with a path. This makes it possible to make the room to which the user exits the toilet or bathroom the same as the room from which the user was located before entering the toilet or bathroom set in step S38. As shown in FIG. 27(B), since the room A from which the user was located immediately before entering the toilet or bathroom is set in step S38, the path generation unit 58 connects the room A from which the user was located immediately before entering the toilet or bathroom with a path (section J2) as the point on the movement path at the time of exiting the room, rather than the nearest room B. This forces the image processing device 50 to return to the room from which the user entered when exiting the toilet or bathroom, thereby preventing the generation of an incorrect path such as that shown in FIG. 27(A).

[0125] Next, when all processing for the moving images captured by the imaging device 10 is completed (YES in step S41), the image processing device 50 shifts the processing to step S42. On the other hand, the image processing device 50 repeats the processing from step S31 until all processing for the moving images captured by the imaging device 10 is completed (NO in step S41).

[0126] Next, the path generation unit 58 sets the start point of the generated path based on the use of the room determined in step S31 (step S42). For example, the path generation unit 58 sets a position corresponding to a room whose use is determined to be an entrance as the start point of the path.

[0127] Then, based on the above-described processing results, the image processing unit 60 generates a tour image, which is a processed image including a plurality of still images captured by the image capturing device 10 (step S42). Specifically, as shown in FIG. 28, the image processing unit 60 generates a tour image in which a plurality of still images are associated by associating the shooting positions estimated by the position estimation unit 56 with the shape estimation result by the shape estimation unit 55. Furthermore, as shown in FIG. 29(A), in addition to the configuration shown in FIG. 28, the image processing unit 60 superimposes a path generated by the path generation unit 58 on the plurality of still images associated by the relative shooting positions. Furthermore, as shown in FIG. 29(B), the image processing unit 60 can indicate the start point (S) of the path set in step S42 on the tour image.

[0128] In this way, the image processing device 50 estimates the shooting position of a high-resolution still image based on the movement path in the moving image captured by the image capturing device 10. Furthermore, the image processing device 50 generates a path indicating the order in which the images were captured based on the estimated shooting position, the shape of the room shown in the captured image, and the movement path of the image capturing device 10.

[0129] The tour images generated by the image processing device 50 can be viewed by a viewer using the communication terminal 90. When the generation of the tour images is complete, the image processing device 50 notifies the communication terminal 90 that requested the tour shooting. The display control unit 93 of the communication terminal 90 causes the tour images provided by the image processing device 50 to be displayed on the display 906. The viewer can use the communication terminal 90 to view a virtual tour of a real estate property or the like using the tour images provided by the image processing device 50.

[0130] Effect of the embodiment As described above, the image processing system 1 can improve the accuracy of estimating the shooting positions of captured images by accurately associating high-resolution still images used to create a virtual tour with the shooting positions of multiple still images estimated using moving images captured by the image capturing device 10. The image processing system 1 can then automatically create a virtual tour with an appropriate path (route) (a high-quality virtual tour).

[0131] In addition, the image processing system 1 can efficiently acquire captured images for creating a virtual tour by using the photographing device 10 to switch between capturing high-resolution still images to obtain images for viewing and capturing low-resolution video images for position estimation in a time-division manner.

[0132] Furthermore, the image processing system 1 uses an omnidirectional imaging device that captures the interior of a predetermined base, such as a real estate property, in all directions at once, and uses natural images free of errors due to parallax during imaging for the virtual tour, thereby making it possible to create a natural virtual tour (high-quality virtual tour). Furthermore, the image processing system 1 can create a virtual tour with improved position estimation accuracy, even when imaging is performed with the imaging device 10 fixed to a support member 20 such as a monopod so that the photographer does not appear in the captured images.

[0133] ●Summary● As described above, an image processing method according to one embodiment of the present invention is an image processing method executed by an image processing device 50 that processes images captured in all directions within a predetermined base, and includes the following steps: a still image acquisition step that acquires multiple still images captured at different shooting positions within the base; a video acquisition step that acquires video images captured while moving from a first point to a second point within the base; a position estimation step that estimates the relative shooting positions of the multiple still images based on the acquired video images; and an image processing step that generates a processed image (e.g., a tour image) including the multiple still images associated based on the estimated shooting positions. This makes it possible to provide a virtual tour with improved accuracy in estimating the shooting positions of the captured images.

[0134] Furthermore, an image processing method according to one embodiment of the present invention further includes a shape estimation step of estimating the shape of a space (e.g., a room) within the base that appears in the video image, and a path generation step of generating a path that associates multiple still images based on the shooting positions estimated by the position estimation step and the shape estimated by the shape estimation step. The position estimation step estimates the relative shooting positions of the multiple still images based on the shapes estimated by the shape estimation step, and the image processing step generates a processed image (e.g., a tour image) that includes paths corresponding to the multiple still images. This allows the image processing method to accurately associate multiple still images with the shooting positions of each still image based on the shooting positions of the still images estimated using the video image, the shape of the space depicted in the captured images, and the generated path. The image processing method can then automatically create a virtual tour (high-quality virtual tour) with high positional accuracy and an appropriate path (route).

[0135] In the image processing method according to an embodiment of the present invention, the moving images are images with a lower resolution than the still images. In the image processing method, the still images and the moving images are images captured by an omnidirectional imaging device. This allows the image processing method to efficiently create a virtual tour with an appropriate path (route) (high-quality virtual tour).

[0136] Furthermore, an image processing system according to an embodiment of the present invention is an image processing system 1 including an image processing device 50 and a photographing device 10, which photographs still images and moving images by switching between them in a time-division manner. The photographing device 10 also switches between photographing still images and moving images depending on the state of movement within a base. This allows the image processing system 1 to efficiently acquire photographed images for creating a virtual tour.

[0137] ●Additional Information● Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in the present embodiment includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices designed to perform each function described above, such as an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a graphics processing unit (GPU), and a conventional circuit module.

[0138] So far, we have described an image processing method, program, image processing device, and image processing system according to one embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and other modifications, such as additions, changes, or deletions, can be made within the scope of what a person skilled in the art can conceive, and any aspect is included in the scope of the present invention as long as it achieves the functions and effects of the present invention. [Explanation of symbols]

[0139] 1. Image processing system 10 Imaging equipment 11 Transmitter / Receiver 12 Operation reception section 13. Imaging control unit (an example of imaging control means) 14 Videography Department 15 Still Image Shooting Section 16. Motion detection unit 20 Support member 50 Image processing device 51 Transmitting / receiving unit (an example of a still image acquiring means, an example of a moving image acquiring means) 54 Application determination section 55 Shape estimation part 56 Position estimation unit (an example of a position estimation means) 57 Detection unit 58 Path generation unit (an example of a path generation means) 60 Image processing unit (an example of image processing means) 90 Communication terminal 91 Transmitter / Receiver 92 Reception Department 93 Display control unit (an example of display control means)

Claims

1. An image processing method for processing a plurality of omnidirectional images obtained by photographing each of a plurality of spaces of a structure in an omnidirectional manner, the method comprising: an image acquisition step of acquiring the plurality of omnidirectional images and a moving image captured while moving from a first space to a second space in the plurality of spaces; a position estimation step of estimating relative shooting positions of the plurality of omnidirectional images based on the acquired moving images; a path generation step of generating a path that associates the shooting positions of the plurality of omnidirectional images based on the estimated relative positions of the shooting positions of the plurality of omnidirectional images; An image processing method that performs

2. A shape estimation step is carried out to estimate the spatial shape of the structure shown in the moving image, the position estimation step estimates relative shooting positions of the plurality of omnidirectional images based on the shapes estimated in the shape estimation step; The image processing method according to claim 1 , wherein the path generating step generates the path based on the estimated relative shooting positions of the plurality of omnidirectional images and the estimated shape.

3. A detection step is carried out to detect the space to which the point on the moving path where the video image was taken belongs; The image processing method according to claim 2, wherein the path generation step generates the path when a point at which the moving image was taken that belongs to multiple spaces is detected so that the point belongs to the nearest space.

4. A use determination step is carried out to determine the use of the space, The image processing method according to claim 3 , wherein the path generating step generates the path based on the determined use.

5. An image processing method as described in Claim 4, wherein the path generation step sets the starting point of the generated path based on the determined use.

6. An image processing method described in any one of claims 1 to 5, wherein the resolution of the moving image is lower than the resolution of the omnidirectional image.

7. An image processing method according to any one of claims 1 to 6, wherein the omnidirectional image and the moving image are images captured by an omnidirectional imaging device.

8. A program for causing a computer to execute image processing for processing a plurality of omnidirectional images obtained by photographing each of a plurality of spaces of a structure in an omnidirectional manner, the image processing comprising: an image acquisition step of acquiring the plurality of omnidirectional images and a moving image captured while moving from a first space to a second space in the plurality of spaces; a position estimation step of estimating relative shooting positions of the plurality of omnidirectional images based on the acquired moving images; a path generation step of generating a path that associates the shooting positions of the plurality of omnidirectional images based on the estimated relative positions of the shooting positions of the plurality of omnidirectional images; Programs including.

9. An image processing device that processes a plurality of omnidirectional images obtained by photographing each of a plurality of spaces of a structure in an omnidirectional manner, for viewing each of the plurality of spaces in an omnidirectional manner, comprising: an image acquisition means for acquiring the plurality of omnidirectional images and a moving image captured while moving from a first space to a second space in the plurality of spaces; a position estimation means for estimating relative photographing positions of the plurality of omnidirectional images based on the acquired moving images; a path generating means for generating a path that associates the shooting positions of the plurality of omnidirectional images based on the estimated relative positions of the shooting positions of the plurality of omnidirectional images; An image processing device comprising:

10. An image processing system comprising the image processing device according to claim 9, a photographing device, and a communication terminal capable of communicating with the image processing device, The communication terminal An image processing system comprising a display control means for displaying an image processed by the image processing device.

11. The display control means displays the start point of the path on the image processed by the image processing device. The image processing system according to claim 10.

Images