IMAGE PROCESSING METHOD, PROGRAM, IMAGE PROCESSING APPARATUS, AND IMAGE PROCESSING SYSTEM

Abstract

To provide a virtual tour with improved accuracy in estimating the photographic position of a photographic image.SOLUTION: An image processing method, which can be executed by an image processing device 50 that performs processing on a photographic image captured in all directions within a predetermined base, includes: a still image acquisition step of acquiring a plurality of still images captured at different photographic positions within the base; a moving image acquisition step of acquiring a moving image captured while moving from a first point to a second point within the base; a position estimation step of estimating relative photographic positions of the plurality of still images based on the acquired moving image; and an image processing step of generating a processed image (for example, a tour image) including the plurality of still images associated with each other based on the estimated photographic positions.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present disclosure relates to an image processing method, a program, an image processing device, and an image processing system. [Background technology]

[0002] There is a known system that distributes image data captured by an omnidirectional imaging device, allowing a user to view the situation at a remote location at another location. A spherical image captured at a specific location in all directions allows a user to view the image in any direction, and therefore conveys information with a sense of realism. Such a system is used, for example, in the field of online property viewing in the real estate industry.

[0003] Furthermore, there is a virtual tour service that interconnects images taken at multiple shooting positions of a real estate property, allowing users to feel as if they are walking around and viewing the inside of the property. In order to create this virtual tour, it is important to connect the images taken at multiple shooting positions in the correct positional relationship. To create such a virtual tour, a technology is already known that estimates the position of a captured image using acceleration information measured by an IMU (inertial measurement unit) (for example, Patent Document 1 and Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Pat. No. 10,375,306 [Patent Document 2] U.S. Pat. No. 1,053,097 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional method of estimating position based on acceleration using an IMU leaves room for improvement in terms of the accuracy of estimating positions from multiple captured images used to create a virtual tour. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the invention of claim 1 is an image processing method executed by an image processing device that processes images captured in all directions within a specified base, the image processing method executing the following steps: a still image acquisition step of acquiring multiple still images captured at different shooting positions within the base; a moving image acquisition step of acquiring moving images captured while moving from a first point to a second point within the base; an estimation step of estimating relative shooting positions of the multiple still images based on the acquired moving images; and an image processing step of generating a processed image including the multiple still images associated based on the estimated shooting positions. Effect of the Invention

[0007] According to the present invention, it is possible to provide a virtual tour with improved estimation accuracy of the shooting positions of captured images. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of an overall configuration of an image processing system. [Diagram 2] FIG. 1 is a diagram illustrating an example of a spherical image captured by an imaging device. [Diagram 3] (A) is a hemispherical image (front) captured by the camera, (B) is a hemispherical image (back) captured by the camera, and (C) is an image represented by equirectangular projection. [Figure 4] (A) A conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (B) a diagram showing a spherical image. [Diagram 5] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. [Figure 6] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 7] FIG. 2 is a diagram showing an example of a state during shooting by the shooting device. [Figure 8] FIG. 1 is a diagram illustrating an example of a spherical image. [Figure 9] FIG. 11 is a diagram illustrating an example of a planar image converted from a spherical image. [Figure 10] 1A and 1B are schematic diagrams for explaining an example of an imaging device that can be applied to the image processing system. [Figure 11] FIG. 2 is a diagram for explaining an example of a photographed image photographed by a general photographing device. [Figure 12] 3A to 3C are schematic diagrams illustrating an example of processing executed by an image processing device. [Figure 13] FIG. 2 is a diagram illustrating an example of a hardware configuration of the imaging device. [Figure 14] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing device and a communication terminal. [Figure 15] FIG. 1 illustrates an example of a functional configuration of an image processing system. [Figure 16] 10 is a flowchart showing an example of tour photography processing by the photography device. [Figure 17] 13(A) and (B) are diagrams showing an example of a method for fixing the imaging device. [Figure 18] 1A and 1B are diagrams for explaining differences between handheld shooting and fixed shooting. [Figure 19] 1 is a diagram for explaining switching between video shooting and still image shooting by the imaging device; [Figure 20] FIG. 11 is a diagram showing an example of timing for switching between moving image shooting and still image shooting. [Figure 21] 1A and 1B are diagrams illustrating an example of a tour path in a virtual tour. [Figure 22] 11 is a flowchart showing an example of a tour image generation process performed by the image processing device. [Figure 23] 13A and 13B are schematic diagrams for explaining an example of a process for estimating the size of a room shape. [Figure 24] 13A and 13B are schematic diagrams illustrating an example of a process for determining a room in which a point on a movement path is located. [Diagram 25] 13A and 13B are schematic diagrams illustrating an example of a process for determining a room in which a point on a movement path is located. [Figure 26] 10 is a schematic diagram for explaining an example of a path generated by a path generating unit. FIG. [Figure 27] 13A and 13B are schematic diagrams illustrating an example of a path that is generated when the room is used as a toilet or a bathroom. [Figure 28] FIG. 2 is a schematic diagram for explaining an example of a tour image generated by the image processing device. [Figure 29] 1A and 1B are schematic diagrams illustrating an example of a tour image generated by an image processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0010] ●Embodiment● ●Outline of the image processing system First, an outline of the configuration of an image processing system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the overall configuration of an image processing system. The image processing system 1 shown in Fig. 1 is a system that performs image processing on a captured image for allowing a viewer to view the internal space of a structure such as a real estate property online.

[0011] 1, the image processing system 1 includes a photographing device 10, an image processing device 50, and a communication terminal 90. The photographing device 10, the image processing device 50, and the communication terminal 90 constituting the image processing system 1 can communicate with each other via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. Note that the communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] The image processing device 50 is a server computer that performs image processing on captured images of the interior space of a structure such as a real estate property, which is a specific base. The image processing device 50, for example, acquires captured images captured by the image capture device 10, and generates tour images for providing a user with a virtual tour using the acquired captured images. Here, the virtual tour is a content that allows a user to view the interior of a structure such as a real estate property as if they were actually viewing the interior on-site. The tour image is an image for viewing that is generated using a plurality of captured images captured by the image capture device 10, and allows the user to virtually move around the base shown in the captured images by user operation.

[0013] The image processing device 50 may be configured by one server computer or may be configured by multiple server computers. Although the image processing device 50 will be described as a server computer existing in a cloud environment, it may be a server existing in an on-premise environment.

[0014] The photographing device 10 is a special digital camera (omnidirectional photographing device) capable of photographing the interior space of a structure such as a real estate property at a photographing base in all directions to obtain a 360° omnidirectional image. The photographing device 10 is used, for example, by a real estate agent who manages or sells real estate properties. The photographing device 10 may be a wide-angle camera or a stereo camera capable of obtaining a wide-angle image having a predetermined angle of view or more. A wide-angle image is generally an image photographed using a wide-angle lens, and is an image photographed using a lens that can photograph a wider range than the human eye can sense. In other words, the photographing device 10 is a photographing means capable of obtaining an image (omnidirectional image, wide-angle image) photographed using a lens with a focal length shorter than a predetermined value. A wide-angle image generally means an image photographed using a lens with a focal length of 35 mm or less in 35 mm film equivalent.

[0015] The communication terminal 90 is a computer such as a smartphone that displays the image processed by the image processing device 50 and allows the viewer to view the image. The communication terminal 90 is used, for example, by the same real estate agent as the photographing device 10. A dedicated application for issuing photographing instructions to the photographing device 10 and viewing images provided by the image processing device 50 is installed in the communication terminal 90. The communication terminal 90 may be configured to issue photographing instructions and view images by accessing a dedicated website using a web browser without using a dedicated application. The photographing instructions and image viewing may be performed by different communication terminals 90.

[0016] In addition, the communication terminal 90 is not limited to a smartphone, but may be, for example, a PC, a tablet terminal, a wearable terminal, an HMD (head mounted display), or an IWB (Interactive White Board: an electronic whiteboard with a blackboard function that allows mutual communication), etc.

[0017] ○Outline of the imaging device○ Here, an overview of the photographing device 10 constituting the image processing system 1 will be described with reference to Figs. 2 to 11. Fig. 2 is a diagram showing an example of a celestial sphere image photographed by the photographing device. The image shown in Fig. 2 is a celestial sphere image photographed by the photographing device 10 of a room of a real estate property, which is an example of the internal space of a structure. A celestial sphere image is suitable for viewing a real estate property because the interior of the room can be photographed in all directions. There are various forms of celestial sphere images, but they are often generated by the equirectangular projection method described later. The advantages of an image generated by this equirectangular projection method are that the outer shape of the image is rectangular, making it efficient and easy to store image data, and that there is little distortion near the equator and vertical straight lines are not distorted, making it look relatively natural.

[0018] ○How to generate spherical images Next, a method for generating a celestial sphere image will be described with reference to Figs. 3 to 9. First, an outline of a process for generating a celestial sphere image from an image captured by the image capture device 10 will be described with reference to Figs. 3 and 4. Fig. 3(A) is a diagram showing a hemispherical image (front side) captured by the image capture device, Fig. 3(B) is a hemispherical image (rear side) captured by the image capture device, and Fig. 3(C) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 4(A) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(B) is a diagram showing a celestial sphere image.

[0019] The photographing device 10 is provided with imaging elements on both the front side (front side) and the back side (rear side). These imaging elements (image sensors) are used in conjunction with optical components such as lenses capable of photographing hemispherical images (angle of view of 180° or more). The photographing device 10 can obtain two hemispherical images by capturing images of subjects around the user using the two imaging elements, respectively.

[0020] 3(A) and (B), the image captured by the imaging element of the photographing device 10 is a curved hemispherical image (front and rear). The photographing device 10 then synthesizes the hemispherical image (front) and the hemispherical image (rear) flipped 180 degrees to create an equirectangular projection image EC as shown in FIG. 3(C).

[0021] Then, the photographing device 10 uses OpenGL ES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image EC so as to cover the sphere as shown in FIG. 4A, and creates a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 4B. In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. In addition, the celestial sphere image CE may be a still image or a video. Furthermore, the conversion method is not limited to OpenGL ES, and may be any method capable of converting a hemispherical image into an equirectangular projection, for example, a calculation by a CPU or a calculation by OpenCL.

[0022] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they see it. Therefore, the imaging device 10 can display a predetermined region T (hereinafter, referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to Figs. 5 and 6.

[0023] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the celestial sphere image is a three-dimensional 3D sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the celestial sphere image CE displayed as a three-dimensional 3D sphere. FIG. 5 shows the celestial sphere image CE as a three-dimensional 3D sphere CS. If the celestial sphere image CE generated in this way is a 3D sphere CS, as shown in FIG. 5, the virtual camera IC is located inside the celestial sphere image CE. The predetermined area T in the celestial sphere image CE is the shooting area of ​​the virtual camera IC, and is specified by the predetermined area information indicating the shooting direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the celestial sphere image CE. In addition, the zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined area image Q is an image of the predetermined area T in the celestial sphere image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0024] The predetermined area image Q is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The following description will be given using the shooting direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be indicated by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0025] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 6. FIG. 6 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 6, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP(x, y) of the predetermined area T, which is the shooting area of ​​the virtual camera IC. As shown in FIG. 6, the center point CP(x, y) when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is α becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of the predetermined area T and the center point CP(x, y) (2L is the diagonal line). In FIG. 6, the trigonometric function shown in the following formula 1 generally holds.

[0026]

number

[0027] Next, a state of photographing by the photographing device 10 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of a state of photographing by the photographing device. In order to photograph a room of a real estate property or the like so that the entire room can be viewed, it is preferable to install the photographing device 10 at a position close to the height of human eyes. For this reason, as shown in FIG. 7, the photographing device 10 is generally fixed to a support member 20 such as a monopod or a tripod for photographing. As described above, the photographing device 10 is a celestial sphere photographing device capable of acquiring light rays in all directions around the entire periphery, and it can also be said that the photographing device 10 acquires an image (a celestial sphere image CE) on a unit sphere around the photographing device 10. When the photographing direction of the photographing device 10 is determined, the coordinates of the celestial sphere image are determined. For example, in FIG. 7, point A is at a distance (d, -h) from the center point C of the photographing device 10. If the angle between the line segment AC and the horizontal direction is θ, the angle θ can be expressed by the following (Equation 2).

[0028]

number

[0029] Assuming that point A is at a depression angle θ, the distance d between points A and B can be expressed by the following (Equation 3) using the installation height h of the image capturing device 10.

[0030]

number

[0031] Here, a process of converting position information on a celestial sphere image into coordinates on a planar image converted from the celestial sphere image will be outlined. Fig. 8 is a diagram for explaining an example of a celestial sphere image. Fig. 8(A) is a diagram showing the hemispherical image shown in Fig. 3(A) by connecting points where the horizontal and vertical incidence angles with respect to the optical axis are equal with lines. Hereinafter, the horizontal incidence angle with respect to the optical axis is referred to as "θ", and the vertical incidence angle with respect to the optical axis is referred to as "φ".

[0032] Also, Fig. 9(A) is a diagram for explaining an example of an image processed by equirectangular projection. Specifically, the image shown in Fig. 8 is made to correspond to a pre-generated LUT (Look Up Table) or the like, processed by equirectangular projection, and the processed images shown in Fig. 8(A) and Fig. 8(B) are synthesized, whereby the planar image shown in Fig. 9(A) corresponding to the omnidirectional image is generated by the photographing device 10. The equirectangular projection image EC shown in Fig. 3(C) is an example of the planar image shown in Fig. 9(A).

[0033] As shown in FIG. 9(A), in an image processed by equirectangular projection, latitude (θ) and longitude (φ) are orthogonal to each other. In the example shown in FIG. 9(A), the center of the image is (0,0), and the latitude direction is expressed as -90 to +90, and the longitude direction is expressed as -180 to +180, so that any position in the omnidirectional image can be indicated. For example, the coordinates of the upper left corner of the image are (-180, -90). The coordinates of the omnidirectional image may be expressed in a format using 360 degrees as shown in FIG. 9(A), or may be expressed in radians or in the number of pixels as in a real image. The coordinates of the omnidirectional image may be converted and expressed as two-dimensional coordinates (x, y) as shown in FIG. 9(B).

[0034] Note that the synthesis process for the planar image shown in Fig. 9(A) or Fig. 9(B) is not limited to the process of simply arranging the hemispherical images shown in Fig. 8(A) and Fig. 8(B) consecutively. For example, if the horizontal center of the omnidirectional image is not θ=180°, in the synthesis process, the image capturing device 10 first preprocesses the hemispherical image shown in Fig. 3(C) and arranges it at the center of the omnidirectional image. Next, the image capturing device 10 may divide the hemispherical image shown in Fig. 3(B) into left and right parts of the image to be generated into a size that allows the preprocessed image to be arranged on the left and right parts, and synthesize the hemispherical images to generate the equirectangular projection image EC shown in Fig. 3(C).

[0035] Also, in the planar image shown in Fig. 9(A), the point corresponding to the pole (PL1 or PL2) of the hemispherical image (spherical image) shown in Fig. 8(A) and Fig. 8(B) is the line segment CT1 or CT2. This is because, as shown in Fig. 4(A) and Fig. 4(B), the celestial sphere image (for example, the celestial sphere image CE) is created by pasting the planar image (equirectangular projection image EC) shown in Fig. 9(A) onto a spherical surface by using OpenGL ES.

[0036] Examples of imaging devices that can be used with image processing systems Next, an example of the photographing device 10 applicable to the image processing system 1 according to the embodiment will be described with reference to FIG. 10 and FIG. 11. FIG. 10 is a schematic diagram for describing an example of a photographing device applicable to the image processing system. FIG. 10(A) shows a special photographing device equipped with a plurality of image pickup elements capable of generating a celestial sphere image by the above-mentioned generation method. The special photographing device uses a wide-angle lens or a fisheye lens with a wide angle of view, and can acquire an image in which all directions are photographed by combining the outputs of the plurality of image pickup elements. FIG. 10(B) shows a general photographing device, which is a so-called normal camera. The general photographing device is, for example, a normal digital camera or a mobile terminal such as a smartphone equipped with a camera. The photographer takes pictures while holding the general photographing device in his / her hand and rotating it. The general photographing device can acquire an image in all directions by synthesizing the obtained images. Both the special photographing device and the general photographing device generate a final photographed image by stitching together multiple photographing results by image processing (stitching process). It is preferable that the optical center of the photographing device 10 for obtaining multiple photographing results is the same.

[0037] FIG. 11 is a diagram for explaining an example of a photographed image photographed by a general photographing device. FIG. 11 shows a photographed image when a photographer photographs a general photographing device as shown in FIG. 10(B) while holding it in his / her hand and rotating it. A general photographing device has a small angle of view (generally 100 degrees or less), so there is a problem that the upper and lower poles cannot be photographed as shown in FIG. 11. In addition, when the photographer rotates it, the position of the optical center of the photographing device is shifted, causing parallax during photographing, and unnatural errors such as steps are likely to occur in the stitching process. That is, as the photographing device 10 in this embodiment, both a special photographing device as shown in FIG. 10 and a general photographing device are applicable, but it is preferable to apply a special photographing device as shown in FIG. 10(A). By using a special photographing device (all-sphere photographing device) as shown in FIG. 10(A), the image processing system 1 can use a natural, high-quality all-sphere image without any defects as an image to be used for a virtual tour that requires quality such as an advertisement. In the following description, the image capturing device 10 will be described as a special image capturing device (omnidirectional image capturing device).

[0038] ○Outline of image processing device processing○ Next, an overview of the processing executed by the image processing device 50 will be described with reference to Fig. 12. Fig. 12 is a schematic diagram for explaining an example of the processing executed by the image processing device. Fig. 12 shows the relationship between the shooting positions of images shot by the image shooting device 10 and paths that associate multiple shot images.

[0039] The image processing device 50 estimates the shooting positions (FIG. 12(A)) of the captured images acquired from the image capturing device 10 using a method such as visual SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion). The estimation result of the shooting positions is expressed as the relative positions of the shooting positions of the multiple captured images as shown in FIG. 12(B). Moreover, the estimation result of the shooting positions is expressed in such a way that each shooting position is expressed in one coordinate system.

[0040] In addition to estimating the shooting positions, the image processing device 50 can also restore the route traveled when the captured image was taken, and can also restore the order and route taken to each shooting position. The image processing device 50 generates a path (tour path) based on the estimated shooting positions and the route traveled when the image was taken. The tour path indicates the connection relationship between a plurality of captured images on a tour image. As shown in FIG. 12(C), a virtual tour using captured images can be created by connecting the captured images taken at each estimated shooting position and allowing users to move between the connected captured images.

[0041] ●Hardware configuration Next, the hardware configuration of each device or terminal constituting the image processing system according to the embodiment will be described with reference to Figures 13 and 14. Note that components may be added or deleted from the hardware configurations shown in Figures 13 and 14 as necessary.

[0042] ○Hardware configuration of the imaging device○ First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the hardware configuration of the imaging device. In the following, the imaging device 10 is an omnidirectional (all-directional) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0043] As shown in FIG. 13, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F (Interface) 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120 and a network I / F 121.

[0044] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter, when there is no need to distinguish between them, they will be referred to as lenses 102) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set.

[0045] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I / F 121 are also connected to the bus 110.

[0046] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to create equirectangular projection image data such as that shown in Figure 3(C).

[0047] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0048] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display (for example, a display of an external terminal such as a smartphone that performs short-distance communication with the imaging device 10 using the short-distance communication circuit 117). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / min).

[0049] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through an I / F bus and performs a predetermined process on the sound data.

[0050] The CPU 111 controls the overall operation of the photographing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0051] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. A user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0052] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0053] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0054] The electronic compass 118 calculates the direction of the image capturing device 10 from the earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of the captured image. The related information also includes each data of the image capture date and time and the data capacity of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) accompanying the movement of the image capturing device 10. The angle changes are an example of related information (metadata) according to Exif and are used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The image capturing device 10 calculates the attitude (angle with respect to the direction of gravity) of the own device (image capturing device 10) based on the acceleration detected by the acceleration sensor 120. The accuracy of image correction is improved by providing the acceleration sensor 120 in the image capturing device 10. The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like.

[0055] ○Hardware configuration of image processing device○ First, the hardware configuration of the image processing device 50 will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the hardware configuration of the image processing device. Each hardware configuration of the image processing device 50 is indicated by a reference number in the 500 series. The image processing device 50 is constructed by a computer, and includes a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516, as shown in Fig. 14.

[0056] Among these, the CPU 501 controls the operation of the entire image processing device 50. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The display 506 may be a touch panel display equipped with an input means. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory. The network I / F 509 is an interface for data communication using the communication network 100. The bus line 510 is an address bus or a data bus for electrically connecting each component such as the CPU 501 shown in FIG. 14.

[0057] The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 511 and the pointing device 512, but also a touch panel or a voice input device, etc. The DVD-RW drive 514 controls reading or writing of various data from a DVD-RW 513 as an example of a removable recording medium. The removable recording medium is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray (registered trademark) Disc, etc. The media I / F 516 controls reading or writing (storing) of data from a recording medium 515 such as a flash memory.

[0058] ○Hardware configuration of communication terminal○ Fig. 14 also shows an example of the hardware configuration of the communication terminal 90. Each hardware configuration of the communication terminal 90 is indicated by a reference number in the 900 series in parentheses. The communication terminal 90 is constructed by a computer, and as shown in Fig. 14, has the same configuration as the image processing device 50, so a description of each hardware configuration will be omitted. In addition to the same configuration as the image processing device 50, the communication terminal 90 has a short-range communication circuit 917 and an antenna 917a of the short-range communication circuit 917. The short-range communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0059] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc, an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the image processing device 50 realizes the image processing method according to the present invention by executing the program according to the present invention.

[0060] ●Function configuration Next, the functional configuration of the image processing system according to the embodiment will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of the functional configuration of the image processing system. Fig. 15 shows devices or terminals shown in Fig. 1 that are related to the processes or operations described below.

[0061] ○Functional configuration of the imaging device○ First, the functional configuration of the photographing device 10 will be described with reference to Fig. 15. The photographing device 10 has a transmitting / receiving unit 11, an operation receiving unit 12, a photographing control unit 13, a video photographing unit 14, a still image photographing unit 15, a movement detection unit 16, a communication unit 17, and a storage / reading unit 19. Each of these units is a function or means realized by operating any of the components shown in Fig. 13 by an instruction from the CPU 111 according to a program for the photographing device expanded from the SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in Fig. 13. The storage unit 1000 stores a GUID (Globally Unique Identifier) ​​of the own device.

[0062] The transmission / reception unit 11 is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. Furthermore, the transmission / reception unit 11 uses the network I / F 121 to perform data communication with other devices or terminals via the communication network 100.

[0063] The operation reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user who is the photographer.

[0064] The shooting control unit 13 is mainly realized by the processing of the CPU 111 for the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and captures an image of a subject such as a landscape, and acquires captured image data. The shooting control unit 13 performs shooting by, for example, switching between video shooting by the video shooting unit 14 and still image shooting by the still image shooting unit 15 in a time-division manner.

[0065] The video shooting unit 14 is mainly realized by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and performs video shooting by the imaging device 10. The video shooting unit 14 shoots moving images while moving within a structure such as a real estate property, which is a predetermined base, for example. The video shooting unit 14 shoots video in continuous frames at low resolution while the photographer holding the imaging device 10 is moving, and stores the captured image data in the storage unit 1000. The video shooting unit 14 shoots video while the photographer holding the imaging device 10 is moving from a first point to a second point within a real estate property, which is a predetermined base, for example.

[0066] The still image shooting section 15 is mainly realized by the processing of the imaging unit 101, and captures an image of a subject such as a landscape, and performs still image capture by the imaging device 10. The still image shooting section 15 captures a plurality of still images captured at different shooting positions within a structure such as a real estate office, which is a predetermined base, for example. The still image shooting section 15 captures a still image (photograph) with a higher resolution than a moving image captured by the moving image shooting section 14, for example, and stores the captured image data in the storage section 1000. The still image captured by the still image shooting section 15 may be a single frame image, or may be an HDR (High Dynamic Range) image obtained by combining a plurality of images.

[0067] Here, the still images captured by the still image capturing unit 15 preferably have a high resolution of, for example, 4K resolution or higher. On the other hand, the moving images captured by the video capturing unit 14 are images used for position estimation, and only require that the subject appearing in the moving images be identifiable, and therefore have a lower resolution than the still images. The moving images may have a resolution of, for example, about 480p or lower. By capturing low-resolution moving images, the imaging device 10 can reduce the overall amount of data in the tour capture.

[0068] The movement detection unit 16 is mainly realized by the processing of the CPU 111 on the gyro sensor 119 and the acceleration sensor 120, and detects the movement state of the photographing device 10. For example, the movement detection unit 16 detects whether the photographer holding the photographing device 10 is moving (in a moving state) or standing still (in a stationary state) while photographing with the photographing device 10.

[0069] The communication unit 17 is mainly realized by the processing of the CPU 111 on the input / output I / F 116 or the short-range communication circuit 117, and transmits and receives various data or information to and from other devices or terminals. The communication unit 17 uses the input / output I / F 116 to perform data communication with the communication terminal 90 via various cables, for example. The communication unit 17 also uses the short-range communication circuit 117 to perform data communication with the communication terminal 90 by short-range wireless communication technology.

[0070] The storage / readout unit 19 is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000. The storage unit 1000 also stores image data captured by the video capture unit 14 and the still image capture unit 15. The image data stored in the storage unit 1000 is associated with the capture time of the captured image as metadata.

[0071] ○Functional configuration of image processing device○ First, the functional configuration of the image processing device 50 will be described with reference to Fig. 15. The image processing device 50 has a transmitting / receiving unit 51, a receiving unit 52, a determining unit 53, a use determining unit 54, a shape estimating unit 55, a position estimating unit 56, a detecting unit 57, a path generating unit 58, an image processing unit 60, and a storing / reading unit 59. Each of these units is a function or means realized by operating any of the components shown in Fig. 14 by an instruction from the CPU 501 according to the program for the image processing device loaded from the HD 504 onto the RAM 503. The image processing device 50 also has a storage unit 5000 constructed by the ROM 502, the RAM 503, and the HD 504 shown in Fig. 14.

[0072] The transmitting / receiving unit 51 is mainly realized by processing of the CPU 501 for the network I / F 509, and transmits and receives various data or information to and from other devices or terminals via the communication network 100. The transmitting / receiving unit 51 receives (acquires), for example, a moving image captured by the imaging device 10 from the imaging device 10 or the communication terminal 90. The transmitting / receiving unit 51 also receives (acquires), for example, a still image captured by the imaging device 10 from the imaging device 10 or the communication terminal 90.

[0073] The reception unit 52 is realized mainly by the processing of the CPU 501 on the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user. The determination unit 53 is realized by the processing of the CPU 501, and performs various determinations.

[0074] The use determination unit 54 is realized by the processing of the CPU 501, and determines the use of the space shown in the captured image based on the captured image showing the interior space of the structure in all directions. In other words, the use is the classification, genre, or purpose of use of the space actually captured by the image capturing device 10. When a real estate property is captured as the base captured by the image capturing device 10, the use of the room that is the space shown in the captured image is, for example, a toilet, a bathroom, or an entrance.

[0075] The shape estimation unit 55 is realized by processing of the CPU 501, and estimates the shape of the space shown in a captured image based on the captured image showing the internal space of a structure in all directions.

[0076] The position estimation unit 56 is realized by the processing of the CPU 501, and estimates the relative shooting position of a still image acquired from the image capturing device 10 based on a moving image acquired from the image capturing device 10. The position estimation unit 56 estimates the relative position of a still image from which a still image was captured, from a moving image that is a low-resolution continuous frame, using a method such as visual SLAM or SfM. This method such as visual SLAM or SfM can calculate the position of a feature point on an image, the position and parameters of a camera from multiple images. On the other hand, the calculated position is relative, and a reference position is required to obtain an absolute position.

[0077] The detection unit 57 is realized by processing of the CPU 501, and detects the shooting points of the moving images that belong to the spatial shape estimated by the shape estimation unit 55. The shooting points of the moving images are, for example, coordinate values ​​obtained by restoring the movement path during shooting of the moving images.

[0078] The path generating unit 58 is realized by processing of the CPU 501, and generates paths indicating the connection relationships of a plurality of still images based on the shooting positions estimated by the position estimating unit 56. For example, the path generating unit 58 generates paths that associate nearest still images among the plurality of still images based on the estimated shooting positions.

[0079] The image processing unit 60 is realized by processing of the CPU 501, and performs image processing to generate a tour image for a virtual tour based on the shooting position estimated by the position estimation unit 56. The image processing unit 60 generates a tour image, which is a processed image including a plurality of still images associated based on the estimated shooting position, for example. The image processing unit 60 determines the position at which the still image should be placed in the tour image based on the estimated shooting position of the still image, the shooting time of the still image, and the path (tour path) generated by the path generation unit 58.

[0080] The storage / reading unit 59 is mainly realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000. The storage unit 5000 also stores tour images created by the image processing unit 60.

[0081] ○Functional configuration of communication terminal○ Next, the functional configuration of the communication terminal 90 will be described with reference to Fig. 15. The communication terminal 90 has a transmission / reception unit 91, a reception unit 92, a display control unit 93, a communication unit 94, and a storage / readout unit 99. Each of these units is a function or means realized by operating any of the components shown in Fig. 15 in response to an instruction from the CPU 901 in accordance with a program for the communication terminal loaded from the HD 904 onto the RAM 903. The communication terminal 90 also has a storage unit 9000 constructed by the ROM 902, RAM 903, and HD 904 shown in Fig. 15.

[0082] The transmission / reception unit 91 is mainly realized by the processing of the CPU 901 for the network I / F 909, and transmits and receives various data or information to and from other devices or terminals via the communication network 100.

[0083] The reception unit 92 is realized mainly by the processing of the CPU 901 on the keyboard 911 or the pointing device 912, and receives various selections or inputs from the user.

[0084] The display control unit 93 is mainly realized by the processing of the CPU 901, and causes various images, characters, etc. to be displayed on the display 906. The display control unit 93 accesses the image processing device 50 using, for example, a Web browser or a dedicated application, and causes an image corresponding to data distributed from the image processing device 50 to be displayed on the display 906.

[0085] The communication unit 94 is mainly realized by the processing of the CPU 901 on the external device connection I / F 908 or the short-range communication circuit 917, and transmits and receives various data or information to and from other devices or terminals. For example, the communication unit 94 uses the external device connection I / F 908 to perform data communication with the photographing device 10 via various cables, etc. Also, the communication unit 94 uses the short-range communication circuit 917 to perform data communication with the photographing device 10 by short-range wireless communication technology.

[0086] The storage / reading unit 99 is realized mainly by the processing of the CPU 901 , and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000 .

[0087] Processing or operation of the embodiment ○Tour photography processing○ Next, the processing or operation of the image processing system according to the embodiment will be described with reference to Figs. 16 to 29. In the following description, an example of a room in a real estate property will be shown as an example of the internal space of a structure that is a specific base. First, the acquisition process of photographed images used for generating tour images by the image processing device 50 will be described with reference to Figs. 16 to 20. Fig. 16 is a flowchart showing an example of tour photographing process by the photographing device.

[0088] First, the photographing apparatus 10 starts a tour photographing process in response to a predetermined request from the communication terminal 90 (step S11). Here, tour photographing refers to photographing within a predetermined base in order to provide a virtual tour. Specifically, the communication terminal 90 executes the tour photographing function by starting and executing an installed dedicated application or accessing a predetermined website using a web browser. Then, the communication unit 94 of the communication terminal 90 transmits a tour photographing request to the photographing apparatus 10. The photographing apparatus 10 starts a tour photographing process in response to the tour photographing request received by the communication unit 17.

[0089] Next, the video shooting unit 14 of the photographing device 10 starts shooting video while moving around the base holding the photographing device 10 (step S12). Specifically, when the tour shooting starts, the shooting control unit 13 of the photographing device 10 requests the video shooting unit 14 to start shooting video. Then, in response to the request from the shooting control unit 13, the video shooting unit 14 starts shooting a moving image.

[0090] Here, a method of photographing using the photographing device 10 will be described with reference to Figs. 17 and 18. Fig. 17 is a diagram showing an example of a method of fixing the photographing device. The photographing device 10 is preferably fixed to a support member 20 such as a monopod 20a as shown in Fig. 17(A) or a tripod 20b as shown in Fig. 17(B) so that the photographer is not photographed. The photographing device 10 is fixed at a height desired for viewing images. For example, the photographing device 10 is preferably placed at a height of a person's line of sight so that a real estate property appears natural.

[0091] Fig. 18 is a diagram for explaining the difference between handheld and fixed shooting. Fig. 18 shows the difference in physical movement between a photographer holding the camera 10 in his / her hand (handheld shooting) and a photographer fixing the camera 10 to a monopod 20a (fixed shooting). As shown in Fig. 17, the simplest method for moving around while shooting is to move the camera 10 while it is attached to a support member 20 such as a monopod 20a, but there is a difference in physical movement between handheld and fixed shooting. Specifically, for example, by attaching a relatively long and large monopod 20a, the physical moment becomes large and the movement of the camera itself is attenuated.

[0092] In Patent Document 1, the shooting position is estimated by position estimation using a sensor such as an IMU and PDR. However, when the monopod 20a is attached, as shown in Fig. 18, the variation in acceleration and angular velocity is attenuated, so there is a problem that the shooting position estimation method described in Patent Document 1 does not provide sufficient accuracy. Therefore, the image processing system 1 employs an image processing method described later so as not to affect the processing accuracy even when shooting with the imaging device 10 fixed to a support member 20 such as a monopod 20a so as not to include the photographer in the image.

[0093] Returning to FIG. 16, when the photographing device 10 detects a change in the photographing mode (YES in step S13), the process proceeds to step S14. In this case, the photographing device 10 switches from the video shooting mode to the still image shooting mode. For example, when the shutter button of the operation unit 115 of the photographing device 10 is pressed, the photographing device 10 switches from the video shooting mode to the still image shooting mode. The photographing device 10 detects the change in the photographing mode. The photographing device 10 may also be configured to switch from the video shooting mode to the still image shooting mode when the movement detection unit 16 detects that the photographing device 10 has stopped (still state). On the other hand, when the photographing unit 14 of the photographing device 10 does not detect a change in the photographing mode (NO in step S13), the photographing device 10 continues the video shooting in step S12.

[0094] Next, the video shooting unit 14 of the imaging device 10 stops video shooting in response to detection of the change in shooting mode in step S13 (step S14), and the still image shooting unit 15 performs still image shooting in response to the user pressing the shutter button of the operation unit 115 (step S15).

[0095] Next, similarly to the process of step S13, when the photographing device 10 detects a change in the photographing mode (YES in step S16), the process proceeds to step S18. In this case, the photographing device 10 switches from the still image photographing mode to the video photographing mode. For example, when the movement detection unit 16 detects the resumption of movement, the photographing device 10 switches from the still image photographing mode to the video photographing mode. The photographing device 10 may be configured to automatically switch from the still image photographing mode to the video photographing mode after the still image photographing unit 15 has photographed a still image.

[0096] The video shooting unit 14 resumes video shooting in response to detection of the switch in shooting mode in step S16 (step S17). As in step S12, the video shooting unit 14 continues shooting while moving around the base holding the camera device 10. Then, when the tour shooting is finished (YES in step S18), the camera device 10 ends the process. The end of the tour shooting is determined, for example, by a predetermined operation on the camera device 10 or the communication terminal 90 by the photographer. On the other hand, when the tour shooting is not finished (NO in step S18), the camera device 10 repeats the process from step S13 to continue the tour shooting.

[0097] On the other hand, in step S16, if the photographing device 10 does not detect a change in the photographing mode (NO in step S16), the photographing device 10 transitions to step S19. As in step S18, if the tour photographing has ended (YES in step S19), the photographing device 10 ends the process. On the other hand, if the tour photographing has not ended (NO in step S19), the photographing device 10 repeats the process from step S19 and continues still image photographing by the still image photographing unit 15.

[0098] Here, video shooting and still image shooting in the imaging device 10 will be described with reference to Figs. 19 and 20. Fig. 19 is a diagram for explaining switching between video shooting and still image shooting by the imaging device. As described above, the imaging device 10 is used for two purposes: video shooting and still image shooting. Of these, video shooting is performed for the purpose of position estimation, and still image shooting is performed for the purpose of obtaining images for viewing. Also, video shooting is performed while moving, and still image shooting is performed while stationary.

[0099] Here, the shooting specifications required for video shooting and still image shooting differ depending on the shooting purpose. In the case of video shooting, continuous images (continuous frames) are acquired at a high frame rate for use in position estimation, but high resolution and color layers of the captured images are not required, and low resolution and grayscale are sufficient. In addition, in the case of still image shooting, while acquisition of continuous frames is not required, shooting with high resolution, color information (RGB), and high dynamic range is required. Furthermore, video shooting is performed while moving, and still image shooting is performed while stationary.

[0100] 20 is a diagram showing an example of the timing of switching between moving image shooting and still image shooting, and shows a time chart of switching between moving image shooting and still image shooting in the process shown in FIG.

[0101] The image capturing device 10 switches between video capture and still image capture in a time-division manner. As described above, video capture is performed to obtain images for position estimation, so it is necessary to capture consecutive frames. On the other hand, still image capture is performed to obtain images for viewing the virtual tour, so it is not necessary to capture consecutive frames, and a configuration in which one image is captured may be sufficient, but an image may be obtained by high dynamic range synthesis using images of multiple frames.

[0102] Still image shooting is an item in which the position of the photographer should be reflected, and shooting is performed by some explicit action of the photographer. For example, still image shooting is performed by switching from video shooting when the shutter button of the operation unit 115 of the photographing device 10 is pressed. In this case, the shutter pressed by the operator may be the shutter button of the main body of the photographing device 10, or may be a shutter based on the operation of an app or controller that can be remotely operated. For example, when photographing a real estate property, it is preferable that the photographer does not appear in the photographed image. Therefore, when operating the shutter button of the main body of the photographing device 10, it is preferable that the photographing device 10 photographs and stores a still image a predetermined time after the shutter button is pressed. Thereby, the photographer can avoid appearing in the photographed image by moving outside the photographing range after pressing the shutter button. Also, when performing remote operation, it is preferable that the photographer performs operation from outside the photographing range to photograph.

[0103] As a method for switching from video recording to still image recording, when the image recording device 10 fixed to the support member 20 is placed in a desired position or when the photographer himself stands still, the movement detection unit 16 may detect the still state and switch to still image recording, or when a predetermined time has elapsed since video recording started, still image recording may be automatically switched to. These methods have the advantage that tour recording can be performed smoothly since the operator does not need to operate the shutter.

[0104] On the other hand, since video shooting is an item that is shot regardless of the photographer's intention, it is preferable that the shooting is performed automatically. For example, video shooting is automatically performed by switching from still image shooting after still image shooting. Also, in order to reduce the amount of data of the shot image, video shooting may be configured to be switched from still image shooting when the movement detection unit 16 detects that movement has resumed.

[0105] In this way, the image capturing device 10 can efficiently acquire captured images for generating tour images by switching in time series between video capture for capturing low-resolution video images for position estimation and still image capture for capturing high-resolution still images for viewing in the virtual tour. In the above description, a configuration in which video capture is performed at the start of tour capture has been described, but the image capturing device 10 may be configured to first capture still images at the start of tour capture.

[0106] The photographing device 10 uploads the captured video and still images to the image processing device 50 via the communication terminal 90 as needed. The image processing device 50 temporarily stores the video and still images transmitted from the photographing device 10 in a storage unit 5000 for generating a tour image. Note that the photographing device 10 may be configured to directly upload the video and still images to the image processing device 50 without going through the communication terminal 90.

[0107] ○Tour image generation processing○ Next, a process for generating tour images using images captured by the image capturing device 10 will be described with reference to FIGS.

[0108] FIG. 21 is a diagram for explaining an example of a tour path in a virtual tour. It is preferable that the tour path is generated so as to be equivalent to the adjacency relationship of rooms, which are the internal spaces of an actual real estate property. For example, if rooms A and B are adjacent to each other and can be accessed by a door, the still images taken in rooms A and B should be connected by a path. On the other hand, if rooms A and C are adjacent to each other but cannot be accessed by a door, the still images taken in rooms A and C should not be connected by a path. Therefore, when automatically generating a virtual tour, it is necessary to automatically determine which captured images should be connected by a path using the capture positions of the still images and the route between them.

[0109] As shown in FIG. 21, a tour path is basically generated by connecting still images in the order they were taken. For example, in the example shown in FIG. 21(A), paths are connected in the order they were taken, generating a path of "Room A ⇔ Room B ⇔ Room C." On the other hand, in the example shown in FIG. 21(B), even though rooms B and C are consecutively taken, they are not physically connected, so in the actual space, you must return to room A before entering room C. In such a case, if paths are simply connected in the order they were taken, rooms that should not be visited would be connected by paths, resulting in a tour that differs from reality. Therefore, as shown in FIG. 21(B), a path connecting rooms B and C is generated by taking into account the structure of the actual space and passing through room A again.

[0110] In this way, the image processing device 50 generates a path using an algorithm that takes into consideration which room each shooting point on the movement route during shooting is located in, so that the path can be correctly connected in the generated tour image. The algorithm for automatically generating a path in the image processing device 50 will be described in detail below.

[0111] Fig. 22 is a flowchart showing an example of a tour image generation process by an image processing device. The image processing device 50 executes the tour image generation process using moving images and still images that are images captured by the image capture device 10. Note that the image processing device 50 will be described as having acquired moving images and still images captured by the image capture device 10 in advance. At this time, the transmitting / receiving unit 51 of the image processing device 50 receives (acquires) moving images and still images transmitted from the image capture device 10. Also, in the examples shown in Figs. 23 to 27, it is assumed that the still images are images of rooms A, B, and C captured at the capture positions ("1 to 3" in each drawing) in rooms A, B, and C, respectively.

[0112] First, the purpose determination unit 54 of the image processing device 50 determines the purpose of the room shown in the acquired video (step S31). Specifically, the purpose determination unit 54 determines the purpose (genre) of the room shown in the video by using general machine learning or parameters such as feature amounts of the video. Here, the purpose of the room determined is, for example, a toilet, a bathroom, or an entrance.

[0113] Next, the shape estimation unit 55 estimates the shape of the room shown in the acquired video (step S32). Specifically, the shape estimation unit 55 detects straight lines of the subject shown in the video, obtains the vanishing points of the detected straight lines, estimates the shape of the room shown in the video by a method of estimating the structure of the room from the boundaries of the floor, walls, ceiling, etc., a method of using three-dimensional information acquired from the imaging device 10, or a method of using general machine learning, and performs three-dimensional reconstruction to acquire the shape of the room as viewed from above.

[0114] The shape estimation unit 55 also estimates the size of the room shape whose structure was estimated in step S32 (step S33). Specifically, the shape estimation unit 55 adjusts the size of the estimated room shape to match the coordinate system of each point of the video on the movement path. As shown in FIG. 23(A), when the room shape estimated in step S32 is restored from a single image, the absolute size is unknown. As shown in FIG. 23(B), in order to arrange each room shape in the correct size on the movement path during video shooting, the shape estimation unit 55 determines the size of each room shape in the coordinate system of each point of the video on the movement path during video shooting. The shape estimation unit 55 estimates the size of each room shape using the shooting height during video shooting and the average value of the size that makes the gap between adjacent room shapes zero. In this way, the shape estimation unit 55 estimates the shape and size of the room shown in the video, and the position estimation unit 56 estimates the relative shooting positions of the video and the multiple still images shot in a time-division manner.

[0115] Next, the detection unit 57 determines in which room shape each point of the moving image on the moving path is located among the room shapes estimated in steps S32 and S33 (step S34). The detection unit 57 detects in which room each point on the moving path is located among the room shapes arranged in step S33 from the coordinate values. In this way, the detection unit 57 detects in which room each point on the moving path was located. For example, as shown in FIG. 24(B), the image processing device 50 uniquely determines in which room each point is located based on the coordinate values ​​on the moving path at the time of shooting the restored moving image. In the example shown in FIG. 24(B), it is determined that the point on the path R1 is located in room A, the point on the path R2 is located in room B, and the point on the path R3 is located in room C.

[0116] Furthermore, when the detection unit 57 detects that a point of the moving image on the moving path does not exist within a single room shape (NO in step S34), it shifts the process to step S35. When a point on the moving path does not belong to a single room, as in the section S1 shown in FIG. 25(A), that is, when the point on the moving path is located within a plurality of room shapes (room A and room B in the example of FIG. 25), it is not possible to uniquely determine which room the point was located in when it was photographed. Also, when a point on the moving path is not located within any room shape, as in the section S2 shown in FIG. 25(A), it is not possible to uniquely determine which room the point was located in when it was photographed. This occurs due to an error in the restoration of the moving path, an error in the estimation of the room shape, and an error in the size of the room shape.

[0117] Therefore, when the points of the moving image on the moving path are not present in a single room shape, the detection unit 57 determines that the room in which the point on the moving path closest to the target point is located is the room in which the target point is located, in order to uniquely determine which room shape the target point is present in (step S35). The detection unit 57 calculates the points on the moving path closest to the points in the sections S1 and S2 in Fig. 25(A), and determines the rooms in which the points on the moving path closest to each other are located, such as the sections M1 and M2 in Fig. 25(B), as the room in which the target point is located.

[0118] On the other hand, in step S34, when the detection unit 57 detects that each point of the moving image on the movement path exists within a single room shape (YES in step S34), the process proceeds to step S36.

[0119] Next, the path generating unit 58 generates a path connecting the room in which the point of the moving image on the moving path exists and the room in which the next point exists (step S36). Since the process of steps S34 and S35 uniquely determines which room shape each point on the moving path is located in, the path generating unit 58 connects the room in which a certain point exists and the room in which the next point exists as the next room with a path for each point on the moving path. In the example of FIG. 26, the path generating unit 58 connects the room B entered after the room A with a path (N1). In addition, for the movement of returning from the room B to the room A and entering the room C, since there is a section (N2 and N3) of the room A between the shooting position 2 and the shooting position 3, the path generating unit 58 can correctly connect the path from the room A to the room C based on the estimation results in steps S32 and S33. Finally, the path generating unit 58 can generate a path of the route "R1⇔R2⇔R1⇔R3" as shown in FIG. 26. That is, a path of "Room A⇔Room B" and a path of "Room A⇔Room C" are generated. That is, the path generating unit 58 generates paths connecting each room, thereby making it possible to associate the nearest still images among the multiple still images captured in each room with each other.

[0120] Next, if the use of the next room determined in step S31 is a toilet or a bathroom (YES in step S37), the judgment unit 53 shifts the process to step S38. On the other hand, if the use of the next room determined in step S31 is not a toilet or a bathroom (NO in step S37), the judgment unit 53 shifts the process to step S39.

[0121] In most houses, there is only one room leading to a toilet or bathroom, i.e., there is only one room to return to, and this is utilized to prevent the wrong path from being connected. For example, as shown in FIG. 27(A), when there are many overlapping areas due to errors in the above-mentioned process, a path may be generated that connects room A to room B via the toilet or bathroom. In the example of FIG. 27(A), the room in which the overlapping section J1 point exists is determined to be room B by the judgment based on the nearest point on the movement route in step S35, and a path connecting the toilet or bathroom is generated. However, in real houses, there are almost no cases where a toilet or bathroom leads to two different rooms.

[0122] Therefore, when the use of the next room is a toilet or a bathroom, the determination unit 53 sets the current room as a room adjacent to the next room (step S38). For example, when the use of a room where a point next to a certain point on the movement route is located is a toilet or a bathroom, the determination unit 53 stores the room where the certain point exists as a room adjacent to the toilet or the bathroom.

[0123] Next, if the use of the previous room determined in step S31 is a toilet or a bathroom (YES in step S39), the judgment unit 53 shifts the process to step S40. On the other hand, if the use of the previous room determined in step S31 is not a toilet or a bathroom (NO in step S39), the judgment unit 53 shifts the process to step S41. If the room previous to a certain point on the movement path is a toilet or a bathroom, that point means that the toilet or the bathroom has been left. For example, the point in section J1 in FIG. 27(A) indicates the point when the toilet or the bathroom has been left.

[0124] When the previous room is used as a toilet or a bathroom, the path generating unit 58 generates a path to a room adjacent to the previous room (step S40). When the room where the point immediately before a certain point on the movement path is located is a toilet or a bathroom, the path generating unit 58 connects the room where the certain point is located to the adjacent room set in step S38 with a path. This makes it possible to make the room to which the user exits the toilet or bathroom the same as the room in which the user was located before entering the toilet or bathroom set in step S38. As shown in FIG. 27(B), since the room A in which the user was located immediately before entering the toilet or bathroom is set in step S38, the path generating unit 58 connects the room A in which the user was located immediately before entering the toilet or bathroom with a path (section J2) as a point on the movement path at the time of exiting the room, rather than the nearest room B. This forces the image processing device 50 to return to the room from which the user entered when exiting the toilet or bathroom, and prevents the generation of an incorrect path as shown in FIG. 27(A).

[0125] Next, when all the processes for the moving images captured by the imaging device 10 are completed (YES in step S41), the image processing device 50 shifts the process to step S42. On the other hand, the image processing device 50 repeats the processes from step S31 until all the processes for the moving images captured by the imaging device 10 are completed (NO in step S41).

[0126] Next, the path generating unit 58 sets the start point of the generated path based on the purpose of the room determined in step S31 (step S42). For example, the path generating unit 58 sets a position corresponding to a room whose purpose is determined to be an entrance as the start point of the path.

[0127] Then, the image processing unit 60 generates a tour image, which is a processed image including a plurality of still images captured by the image capturing device 10, based on the above-mentioned processing result (step S42). Specifically, as shown in FIG. 28, the image processing unit 60 generates a tour image in which a plurality of still images are associated by associating the shooting position estimated by the position estimation unit 56 with the shape estimation result by the shape estimation unit 55. Also, as shown in FIG. 29(A), in addition to the configuration shown in FIG. 28, the image processing unit 60 superimposes the path generated by the path generation unit 58 on the plurality of still images associated by the relative shooting position. Furthermore, as shown in FIG. 29(B), the image processing unit 60 can display the start point (S) of the path set in step S42 on the tour image.

[0128] In this way, the image processing device 50 estimates the shooting position of a high-resolution still image based on the movement path in the moving image captured by the image capturing device 10. Furthermore, the image processing device 50 generates a path indicating the order in which the images were captured based on the estimated shooting position, the shape of the room shown in the captured image, and the movement path of the image capturing device 10.

[0129] The tour images generated by the image processing device 50 can be viewed by a viewer using the communication terminal 90. When the generation of the tour images is completed, the image processing device 50 notifies the communication terminal 90, which is the requester of the tour shooting. The display control unit 93 of the communication terminal 90 causes the tour images provided from the image processing device 50 to be displayed on the display 906. The viewer can use the communication terminal 90 to view a virtual tour of a real estate property or the like using the tour images provided from the image processing device 50.

[0130] Effect of the embodiment As described above, the image processing system 1 can improve the estimation accuracy of the shooting positions of the captured images by accurately associating high-resolution still images used to create a virtual tour with the shooting positions of a plurality of still images estimated using moving images captured by the imaging device 10. The image processing system 1 can then automatically create a virtual tour with an appropriate path (route) (high-quality virtual tour).

[0131] In addition, the image processing system 1 can efficiently acquire captured images for creating a virtual tour by using the photographing device 10 to switch between capturing high-resolution still images to obtain images for viewing and capturing low-resolution videos for position estimation in a time-division manner.

[0132] Furthermore, the image processing system 1 performs shooting using an omnidirectional shooting device that shoots the interior of a given base such as a real estate property in all directions at once, and uses natural images without errors due to parallax during shooting for the virtual tour, thereby making it possible to create a natural virtual tour (high-quality virtual tour). Also, the image processing system 1 can create a virtual tour with improved position estimation accuracy even when shooting is performed with the shooting device 10 fixed to a support member 20 such as a monopod so that the photographer is not captured in the captured images.

[0133] ●Summary● As described above, the image processing method according to one embodiment of the present invention is an image processing method executed by the image processing device 50 for processing captured images captured in all directions within a specified base, and includes a still image acquisition step for acquiring multiple still images captured at different capture positions within the base, a video acquisition step for acquiring video images captured while moving from a first point to a second point within the base, a position estimation step for estimating the relative capture positions of the multiple still images based on the acquired video images, and an image processing step for generating a processed image (e.g., a tour image) including the multiple still images associated based on the estimated capture positions. This makes it possible for the image processing method to provide a virtual tour with improved estimation accuracy of the capture positions of the captured images.

[0134] Moreover, the image processing method according to one embodiment of the present invention further executes a shape estimation step of estimating the shape of the space (e.g., a room) in the base that appears in the moving image, and a path generation step of generating a path that associates a plurality of still images based on the shooting position estimated by the position estimation step and the shape estimated by the shape estimation step, and the position estimation step estimates the relative shooting positions of the plurality of still images based on the shape estimated by the shape estimation step, and the image processing step generates a processed image (e.g., a tour image) including paths corresponding to the plurality of still images. Thereby, the image processing method can accurately associate the plurality of still images with the shooting positions of each still image based on the shooting positions of the still images estimated using the moving image, the shape of the space that appears in the shot image, and the generated path. And, the image processing method can automatically create a virtual tour (high-quality virtual tour) with high position accuracy and an appropriate path (route).

[0135] In the image processing method according to an embodiment of the present invention, the video images are images with a lower resolution than the still images. In the image processing method, the still images and the video images are images captured by an omnidirectional imaging device. This allows the image processing method to efficiently create a virtual tour with an appropriate path (route) (high-quality virtual tour).

[0136] Furthermore, the image processing system according to an embodiment of the present invention is an image processing system 1 including an image processing device 50 and an image capturing device 10, which captures still images and moving images by switching between them in a time-division manner. The image capturing device 10 also switches between capturing still images and moving images depending on the movement state within a base. This allows the image processing system 1 to efficiently acquire captured images for creating a virtual tour.

[0137] ●Additional Information● Each function of the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), and a conventional circuit module designed to execute each function described above.

[0138] So far, an image processing method, a program, an image processing device, and an image processing system according to one embodiment of the present invention have been described. However, the present invention is not limited to the above-described embodiment, and other modifications, such as additions, changes, or deletions, can be made within the scope of what a person skilled in the art can conceive. In any aspect, as long as the functions and effects of the present invention are achieved, it is included in the scope of the present invention. [Explanation of symbols]

[0139] 1. Image Processing System 10. Imaging Device 11 Transmitter / receiver 12 Operation reception section 13 Shooting control unit (an example of a shooting control means) 14 Videography Department 15 Still Image Shooting Section 16 Motion detection unit 20 Support member 50 Image processing device 51 Transmitting / receiving unit (an example of a still image acquiring means, an example of a moving image acquiring means) 54 Application determination section 55 Shape estimation part 56 Position estimation unit (an example of a position estimation means) 57 Detection unit 58 Path generation unit (an example of a path generation means) 60 Image processing unit (an example of an image processing means) 90 Communication terminals 91 Transmitter / receiver 92 Reception 93 Display control unit (an example of a display control means)

Claims

1. An image processing method executed by an image processing device for processing images captured in all directions within a predetermined location, comprising: a still image acquisition step of acquiring a plurality of still images taken at different shooting positions within the base; a moving image acquisition step of acquiring moving images captured while moving from a first point to a second point within the base; a position estimating step of estimating relative shooting positions of the plurality of still images based on the acquired moving image; an image processing step of generating a processed image including the plurality of still images associated based on the estimated shooting positions; An image processing method that performs

2. The image processing method according to claim 1, further comprising: A path generating step is performed to generate a path that associates the plurality of still images; The image processing method, wherein the image processing step generates the processed image including the paths corresponding to the plurality of still images.

3. The image processing method according to claim 2, further comprising: A shape estimation step is carried out to estimate a shape of a space within the base shown in the moving image; the position estimation step estimates relative shooting positions of the plurality of still images based on the shape estimated by the shape estimation step; The image processing method, wherein the path generating step generates the path based on the shooting position estimated in the position estimating step and the shape estimated in the shape estimating step.

4. The image processing method according to claim 3, further comprising: A detection step is performed to detect the shape to which the points on the travel route along which the moving images were captured belong; The path generating step is an image processing method in which, when a point at which the video image was captured that belongs to a plurality of the shapes is detected, the path is generated so that the point belongs to the nearest one of the shapes.

5. The image processing method according to claim 4, further comprising: A use determination step is carried out to determine a use of the space; The image processing method, in which the path generating step generates the path based on the determined use.

6. 6. The image processing method according to claim 5, wherein the path generating step sets a start point of the generated path based on the determined purpose.

7. The image processing method according to claim 1 , wherein the moving image has a lower resolution than the still image.

8. The image processing method according to claim 1 , wherein the still image and the moving image are images captured by an omnidirectional imaging device.

9. A program for causing a computer to execute the image processing method according to any one of claims 1 to 8.

10. An image processing device that processes images captured in all directions within a predetermined base, a still image acquisition means for acquiring a plurality of still images taken at different shooting positions within the base; a moving image acquisition means for acquiring moving images captured while moving from a first point to a second point within the base; a position estimation means for estimating relative shooting positions of the plurality of still images based on the acquired moving images; an image processing means for generating a processed image including the plurality of still images associated based on the estimated photographing positions; An image processing device comprising:

11. An image processing system comprising the image processing device according to claim 10 and an image capturing device, The imaging device is The image processing system includes an image capture control means for switching between capturing the still images and the moving images in a time-division manner.

12. 12. The image processing system according to claim 11, The image capture control means is an image processing system that switches between capturing still images and capturing moving images depending on a moving state within the base.

13. 13. The image processing system according to claim 11, further comprising a communication terminal capable of communicating with the image processing device, The communication terminal includes: An image processing system comprising a display control means for displaying the processed image generated by the image processing device.

14. 14. The image processing system according to claim 13, The image processing device includes: a path generating means for generating a path that associates the plurality of still images based on the photographing positions estimated by the position estimating means, The image processing system includes a processing unit for generating a processed image including the paths corresponding to the plurality of still images.

15. 15. The image processing system according to claim 14, The display control means displays the processed image in which the corresponding paths are superimposed on the plurality of still images.

Images