Omnidirectional image processing method and server for performing this method

Abstract

A method and system for enabling effective mapping or linking (or matching) of positional relationships between multiple omni-directional images taken at different positions is disclosed. According to one aspect of the present invention, there is provided an omni-directional image processing method performed on a server, which includes: an acquisition step of acquiring multiple omni-directional images corresponding to different positions within a predetermined indoor space (wherein at least two of the multiple omni-directional images have a common area where a common space is captured); and a processing step of determining link relationships between the multiple omni-directional images.

Description

[Technical field]

[0001] The present invention relates to an omnidirectional image processing method and a server for performing this method, and more particularly to a method and a server for performing this method that enables effective mapping or linking (or matching) of positional relationships between multiple omnidirectional images taken at different positions. [Background technology]

[0002] In recent years, technologies related to virtual tours of specific spaces have been emerging. For example, there is a virtual tour that allows a user to move to a specific position in an indoor space of a building such as an apartment so that the user can experience the indoor space in a realistic manner, and displays images taken at the moved position. In order to generate such a virtual tour, it is necessary to correctly set the positional relationship between images inside the space by reflecting the structural characteristics of the space. In other words, in order to form a virtual tour that matches the actual reality, it is necessary to know the exact position where each image was taken and to associate it with the corresponding image, which has been a hassle that the person taking the image has to do manually.

[0003] Meanwhile, a technique for phase mapping between different images is widely known. Phase mapping is a technique for grasping the relative positional relationship between different images or for linking (or aligning) images.

[0004] Generally, in order to link two different images, a method is used in which feature points are detected from each of the two images that include a common space, and the images are transformed and linked using a transformation function (transformation matrix) that overlaps the detected feature points with the least amount of error.

[0005] Also, a technical concept (e.g., epipolar geometry) is known that uses matching points that exist in each of the two images (e.g., points in two different images that correspond to the same point in space) to grasp the positional relationship between two images without linking (matching) the different images.

[0006] However, there may be cases where there are multiple images and it is not known which of these images should be concatenated together or which images can be mapped to each other (e.g., images that occupy the same space), i.e. the respective positions and / or orientations of the images are not known.

[0007] For example, when images (e.g., 360-degree images) are taken at different positions in an indoor space, various services such as indoor space navigation can be smoothly performed by identifying the positional relationship between each image. However, if the positions at which each image was taken are not known, it is not possible to know which images can be mapped to each other.

[0008] In such a case, it is necessary to determine whether each combination of images can be mapped to each other. However, such a task may require a lot of resources and time. For example, when there are five different images, in order to find a pair of images that can be mapped to each other, the first image can be paired with each of the second image to the fifth image, and it can be determined whether the images can be mapped to each other. For example, the image that has the most common feature points with the first image can be the image to be mapped to the first image. Images that are mapped to each other are images when there is an area where a common space is photographed, and the same feature points are found in the area where the common space is photographed.

[0009] Only when such a task is performed for every image pair can the phase relationship between the images be determined, and then mapping between adjacent images can be performed. In this specification, mapping can be defined as including grasping the relative positional relationship between two images when it is not necessary to link (match) two images, such as images taken at two different positions, although it may be the case that two images are matched when they can be linked (matched).

[0010] Thus, there is a problem in that the cost required to grasp images that can be mapped and to specify the positional relationships between the grasped images increases exponentially as the number of images increases. Summary of the Invention [Problem to be solved by the invention]

[0011] Therefore, the technical problem that the present invention aims to achieve is to provide a method and system that can quickly and effectively perform mapping between multiple images taken inside a building.

[0012] Furthermore, it is an object of the present invention to provide a technical concept that enables images taken at each position within a particular space to be associated with the exact position at which the image was taken. [Means for solving the problem]

[0013] According to one aspect of the present invention, there is provided an omni-directional image processing method performed by a server, the method including: and a processing step of determining a connection relationship between the plurality of omnidirectional images, the acquisition step including: receiving an information set from a mobile computing device, the information set including the omnidirectional image and location information of the mobile computing device at the time the omnidirectional image is captured; and storing the omnidirectional image included in the information set in association with the location information included in the information set, the mobile computing device being mounted on a mobile platform, communicating with an omnidirectional image capturing device mounted on the mobile platform, tracking a location of the mobile computing device, and when the omnidirectional image capturing device captures an image, acquiring location information of the mobile computing device at the time of capturing the image, and transmitting an information set including an omnidirectional image generated by stitching a plurality of partial captured images captured by the omnidirectional image capturing device and location information of the mobile computing device at the time the plurality of partial captured images were captured to the omnidirectional image map generation server.

[0014] In one embodiment, an omnidirectional image processing method is provided in which the mobile computing device further tracks the attitude of the mobile computing device, and when an image is captured by the omnidirectional image capturing device, attitude information of the mobile computing device at the time of capture is further obtained, and the set of information transmitted by the mobile computing device further includes attitude information of the mobile computing device at the time the plurality of partial captured images were captured.

[0015] In one embodiment, the mobile computing device includes a camera module, and the omnidirectional image capturing device and the mobile computing device are mounted on the movable stand such that the shooting direction of a front camera module included in the omnidirectional image capturing device and the shooting direction of a camera module included in the mobile computing device match within a predetermined error range, and the mobile computing device can perform visual simultaneous localization and mapping (VSLAM) on images captured by the camera module to track the position and attitude of the mobile computing device.

[0016] In one embodiment, the method may further include the steps of acquiring a floor plan corresponding to the indoor space, receiving a first point on the floor plan corresponding to first position information included in a first information set, which is one of the plurality of information sets, and a second point on the floor plan corresponding to second position information included in a second information set, which is the other of the plurality of information sets, and arranging the plurality of omnidirectional images on the floor plan based on a positional relationship between the first position represented by the first position information and the second position represented by the second position information, and a positional relationship between the first point and the second point.

[0017] In one embodiment, the processing step may include extracting features from each of the plurality of omnidirectional images using a neural network-based feature extractor, and determining a mapping image for each of the plurality of omnidirectional images based on the features extracted from each of the plurality of omnidirectional images.

[0018] In one embodiment, the step of determining a mapping image for each of the plurality of omnidirectional images may include the steps of: constructing a vector in a DB corresponding to features extracted from each of the plurality of omnidirectional images; vector searching the constructed DB for a vector set that is at least a portion of a first vector corresponding to a first feature extracted from a specific first image among the plurality of omnidirectional images; and determining a second image extracted based on a result of the vector search as a mapping image of the first image.

[0019] In one embodiment, the processing step may further include mapping the first image and the second image determined to be a mapping image of the first image.

[0020] In one embodiment, the step of mapping the first image and the second image determined to be a mapping image of the first image may include a step of determining first feature corresponding positions on the first image corresponding to each first feature extracted from the first image and second feature corresponding positions on the second image corresponding to each second feature extracted from the second image, and a step of determining a relative positional relationship between the first image and the second image based on the determined first feature corresponding positions and second feature corresponding positions.

[0021] In one embodiment, the neural network may be a neural network trained to output a transformation relationship that can optimally match corresponding points extracted from the overlapping common areas of each of the divided images after dividing a given image so that there are overlapping areas.

[0022] In one embodiment, the processing step may include a step of extracting features from each of the plurality of omnidirectional images, a step of vector searching the constructed DB for a vector set that is at least a portion of a first vector corresponding to a first feature extracted from a predetermined first image among the plurality of omnidirectional images, and a step of determining a second image extracted based on a result of the vector search as a mapping image of the first image.

[0023] According to another aspect of the present invention, there is provided an omni-directional image processing method performed by a server, the method including: acquiring a plurality of omni-directional images corresponding to different positions in a predetermined indoor space (wherein at least two of the plurality of omni-directional images have a common area in which a common space is captured); and determining a connection relationship between the plurality of omni-directional images, the acquiring step including: receiving an information set from a mobile computing device (wherein the information set includes the omni-directional images and location information of the mobile computing device at the time when the omni-directional images are captured); and storing the omni-directional images included in the information set in association with the location information included in the information set, the mobile computing device storing the omni-directional images in the information set. In accordance with another aspect of the present invention, there is provided an omnidirectional image processing method in which a mobile computing device is mounted on a mobile stand, communicates with an omnidirectional image capturing device mounted on the mobile stand, tracks a position of the mobile computing device, and when the omnidirectional image capturing device captures an image, acquires position information of the mobile computing device at the time of capturing the image, stitches together a plurality of partial captured images captured by the omnidirectional image capturing device to generate an omnidirectional image, and transmits an information set including the generated omnidirectional image and position information of the mobile computing device at the time the plurality of partial captured images were captured to the omnidirectional image map generation server.

[0024] According to another aspect of the present invention, there is provided an omni-directional image processing method performed by a server, the method including: acquiring a plurality of omni-directional images corresponding to different positions in a predetermined indoor space (wherein at least two of the plurality of omni-directional images have a common area in which a common space is captured); and determining a connection relationship between the plurality of omni-directional images, the acquiring step including the steps of receiving an information set including a plurality of partial images and position information of the mobile computing device at a time when the plurality of partial images are captured; stitching the plurality of partial images included in the information set to generate an omni-directional image corresponding to the information set; and stitching the omni-directional image corresponding to the information set to the information set. and storing the location information included in the set in association with the location information included in the set, wherein the mobile computing device is mounted on a mobile base and communicates with an omnidirectional image capturing device mounted on the mobile base to track the position of the mobile computing device, and when the omnidirectional image capturing device captures an image, obtains location information of the mobile computing device at the time of capturing, and transmits to the omnidirectional image map generation server an information set including a plurality of partial captured images captured by the omnidirectional image capturing device and location information of the mobile computing device at the time the plurality of partial captured images were captured.

[0025] According to another aspect of the present invention, there is provided a computer readable recording medium having recorded thereon a program for carrying out the above-mentioned method.

[0026] According to another aspect of the invention there is provided a computer program stored on a computer readable recording medium for installation in a data processing device and for carrying out the method as described above.

[0027] According to another aspect of the present invention, there is provided a server including a processor and a memory storing a program, the program being configured to cause the server to perform an omni-directional image processing method when executed by the processor, the omni-directional image processing method including: and a processing step of determining a connection relationship between the plurality of omnidirectional images, the acquisition step including: receiving an information set including a plurality of partial images and location information of the mobile computing device at a time when the plurality of partial images are captured from a mobile computing device; stitching the plurality of partial images included in the information set to generate an omnidirectional image corresponding to the information set; and storing the omnidirectional image corresponding to the information set in association with the location information included in the information set. The mobile computing device is mounted on a mobile platform, communicates with an omnidirectional image capturing device mounted on the mobile platform, tracks the location of the mobile computing device, and when an image is captured by the omnidirectional image capturing device, obtains location information of the mobile computing device at the time of capturing the image, and transmits an information set including the captured image captured by the omnidirectional image capturing device and the location information of the mobile computing device at the time when the captured image is captured to the omnidirectional image map generation server.

[0028] In one embodiment, the omnidirectional image processing method may further include the steps of acquiring a floor plan corresponding to the indoor space; receiving a first point on the floor plan corresponding to first position information included in a first information set, which is one of the plurality of information sets, and a second point on the floor plan corresponding to second position information included in a second information set, which is the other of the plurality of information sets; and arranging the plurality of omnidirectional images on the floor plan based on a positional relationship between the first position represented by the first position information and the second position represented by the second position information, and a positional relationship between the first point and the second point.

[0029] In one embodiment, the processing step may include extracting features from each of the plurality of omnidirectional images using a neural network based feature extractor, and determining a mapping image for each of the plurality of omnidirectional images based on the features extracted from each of the plurality of omnidirectional images.

[0030] In one embodiment, the step of determining a mapping image for each of the plurality of omnidirectional images may include the steps of: constructing a vector in a DB corresponding to features extracted from each of the plurality of omnidirectional images; vector searching the constructed DB for a vector set that is at least a portion of a first vector corresponding to a first feature extracted from a specific first image among the plurality of omnidirectional images; and determining a second image extracted based on a result of the vector search as a mapping image of the first image.

[0031] In one embodiment, the processing step may further include mapping the first image and the second image determined to be a mapping image of the first image.

[0032] In one embodiment, the step of mapping the first image and the second image determined to be a mapping image of the first image may include a step of determining first feature corresponding positions on the first image corresponding to each first feature extracted from the first image and second feature corresponding positions on the second image corresponding to each second feature extracted from the second image, and a step of determining a relative positional relationship between the first image and the second image based on the determined first feature corresponding positions and second feature corresponding positions.

[0033] In one embodiment, the neural network may be a neural network trained to output a transformation relationship that can optimally match corresponding points extracted from the overlapping common areas of each of the divided images after dividing a given image so that there are overlapping areas.

[0034] In one embodiment, the processing step may include a step of extracting features from each of the plurality of omnidirectional images, a step of vector searching the constructed DB for a vector set that is at least a portion of a first vector corresponding to a first feature extracted from a predetermined first image among the plurality of omnidirectional images, and a step of determining a second image extracted based on a result of the vector search as a mapping image of the first image. Effect of the Invention

[0035] According to one embodiment of the present invention, a system and method are provided that can quickly and efficiently perform mapping between multiple images taken inside a building.

[0036] Additionally, images taken at each location within a particular space can be associated with the exact location where the image was taken. [Brief description of the drawings]

[0037] In order to more fully understand the drawings referred to in the detailed description of the present invention, a brief description of each of the drawings is provided. [Figure 1] FIG. 2 illustrates an example of an omnidirectional image capture assembly in accordance with one embodiment of the present invention. [Diagram 2] 2 is a diagram for illustrating a schematic operation between an omnidirectional image capturing assembly and a server according to an embodiment of the present invention. FIG. [Diagram 3] 1 is a block diagram illustrating a schematic configuration of an omnidirectional image capturing device according to an embodiment of the present invention. [Figure 4] 1 is a block diagram illustrating a schematic configuration of a mobile computing device according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of a plan view of a specific indoor space and different shooting positions in the indoor space. [Figure 6] 2 shows a flowchart illustrating an omnidirectional image processing method according to an embodiment of the present invention. [Figure 7] 13 is a flowchart showing an example of a process for acquiring an omnidirectional image corresponding to one shooting position. [Figure 8] 1 is a diagram illustrating an automatic phase mapping processing method according to an embodiment of the present invention; [Figure 9] FIG. 1 is a diagram showing a schematic configuration of an automatic phase mapping processing system according to an embodiment of the present invention. [Figure 10] FIG. 1 is a diagram for explaining the concept of using neural network features for an automatic phase mapping processing method according to an embodiment of the present invention. [Figure 11] FIG. 2 is a diagram for illustrating a schematic logical configuration of an automatic phase mapping processing system according to an embodiment of the present invention. [Figure 12] FIG. 13 is a diagram illustrating the advantages of using neural network features according to an embodiment of the present invention. [Figure 13] FIG. 2 is a diagram illustrating feature locations corresponding to neural network features according to an embodiment of the present invention. [Figure 14]1 is a flowchart illustrating a method for searching for a mapping image between images in an automatic phase mapping processing method according to an embodiment of the present invention. [Figure 15] 1 is a flowchart illustrating a method for mapping an image in an automatic phase mapping process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] Since the present invention can be modified in various ways and can have various embodiments, a specific embodiment is shown in the drawings and described in detail in the detailed description. However, it is not intended to limit the present invention to the specific embodiment, and it should be understood that the present invention includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. In the description of the present invention, if a detailed description of related publicly known technology is deemed to make the gist of the present invention unclear, the detailed description will be omitted.

[0039] Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

[0040] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless otherwise clearly defined in the context.

[0041] As used herein, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and are to be understood as not precluding the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0042] Also, as used herein, when a component "sends" data to another component, it means that the component can send data directly to the other component, or can send data to the other component via at least one other component. Conversely, when a component "sends" data to another component, it means that the component sends data to the other component without going through another component.

[0043] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, in which the same reference numerals indicate the same elements.

[0044] FIG. 1 is a diagram showing an example of an omnidirectional image capturing assembly according to one embodiment of the present invention, and FIG. 2 is a diagram for generally explaining the operation between the omnidirectional image capturing assembly according to one embodiment of the present invention and a server.

[0045] Referring to FIG. 1, an omnidirectional image capture assembly 200 may include an omnidirectional image capture device 300 , a mobile computing device 400 , and a mobile platform 500 .

[0046] The omnidirectional image capturing device 300 is also called a 360-degree camera, and may be a device capable of capturing images with a spherical angle of view or capturing multiple partial images to generate an image having a spherical angle of view.

[0047] For example, the omnidirectional image capturing device 300 may include four camera modules (e.g., 320, 340) provided on the front, rear, left side, and right side, respectively, and images captured through these camera modules may be stitched together to generate an omnidirectional image. It goes without saying that there may be various other types of omnidirectional image capturing devices, such as a type in which a camera module having a hemispherical angle of view is provided on the front and rear, respectively.

[0048] The camera modules (eg, 320, 340) included in the omnidirectional image capturing device 300 may include a fisheye lens.

[0049] The mobile computing device 400 may include a mobile processing device such as a smartphone, a tablet, a PDA, or the like.

[0050] The omnidirectional image capture device 300 and the mobile computing device 400 may be placed on a mobile platform 500. The mobile platform 500 may be equipped with wheels 520 for moving forward, backward, left and right, and may be equipped with a mounting means for mounting the omnidirectional image capture device 300 and the mobile computing device 400.

[0051] Meanwhile, in one embodiment, the movable mount 500 may further include a rotating body 600 that rotates around a mount rod 510 of the movable mount as a rotation axis, and the mobile computing device 400 may be mounted on the rotating body 600. In another embodiment, the omnidirectional image capturing assembly 200 may be realized in a form in which the omnidirectional image capturing device 300 is mounted on the rotating body 600 instead of the mobile computing device 400.

[0052] 2, the mobile computing device 400 and the omnidirectional image capturing device 300 are connected to each other via wired or wireless communication and can transmit and receive various information, signals and / or data required to realize the technical idea of ​​the present invention. For example, the wireless communication may include long-distance mobile communication such as 3G, LTE, LTE-A, 5G, Wi-Fi, WiGig, Ultra Wide Band (UWB), LAN card, or short-distance wireless communication such as MST, Bluetooth, NFC, RFID, ZigBee, Z-Wave, IR, etc.

[0053] The mobile computing device 400 can transmit a predetermined signal to the omnidirectional image capturing device 300 to control the omnidirectional image capturing device 300 to capture an image. The omnidirectional image capturing device 300 can transmit the captured image to the mobile computing device 400.

[0054] Meanwhile, the mobile computing device 400 can be connected to the remote server 100 via wired or wireless communication (e.g., the Internet). The mobile computing device 400 can transmit images captured by the omnidirectional image capturing device 300 to the server 100. The server 100 can perform an omnidirectional image processing method, which will be described later, on the images transmitted from the mobile computing device 400.

[0055] Meanwhile, the mobile computing device 400 can track its own position and / or attitude, and can further transmit the position / attitude information at the time when the omnidirectional image capturing device 300 captures an image to the server 100 together with the captured image.

[0056] FIG. 3 is a block diagram showing a schematic configuration of an omnidirectional image capturing device 300 according to an embodiment of the present invention.

[0057] 3, the omnidirectional image capturing device 300 may include a front camera 310, a rear camera 320, a left side camera 330, a right side camera 340, a communication device 350, and a control device 360. Depending on the embodiment of the present invention, some of the above-mentioned components may not necessarily correspond to components essential to realizing the present invention, and it goes without saying that depending on the embodiment, the omnidirectional image capturing device 300 may include more components than the above.

[0058] The front camera 310, rear camera 320, left side camera 330, and right side camera 340 may be provided on the front, rear, left side, and right side of the omnidirectional image capturing device 300, respectively, and may include a fisheye lens.

[0059] The four partial images captured by the front camera 310, the rear camera 320, the left side camera 330, and the right side camera 340 can be stitched together into one omnidirectional image. The image stitching can be performed by the server 100, the omnidirectional image capturing device 300, or the mobile computing device 400.

[0060] According to an embodiment, the omnidirectional image capturing device 300 may include at least some of the front camera 310, the rear camera 320, the left side camera 330, and the right side camera 340. For example, the omnidirectional image capturing device 300 may include only the front camera 310 and the rear camera 320.

[0061] The communication device 350 is capable of performing wired and wireless communication with the mobile computing device 400 .

[0062] The control device 360 ​​can control functions and / or resources of other components of the omnidirectional image capturing device 300 (e.g., the front camera 310, the rear camera 320, the left side camera 330, the right side camera 340, the communication device 350, etc.) The control device 360 ​​may include a CPU, an APU, a mobile processing device, etc.

[0063] For example, when a predetermined signal occurs (e.g., when a predetermined image capture signal is received from the mobile computing device 400), the control device 360 ​​can control the front camera 310, the rear camera 320, the left side camera 330, and the right side camera 340 to capture images, and can control the communication device 350 to transmit the captured images to the mobile computing device 400.

[0064] FIG. 4 is a block diagram that generally illustrates the configuration of a mobile computing device 400 in accordance with one embodiment of the present invention.

[0065] 4, the mobile computing device 400 may include a communication module 410, a tracking module 420, a control module 430, a storage module 440, and a transmission module 450. According to an embodiment, the mobile computing device 400 may further include a rotor control module 460. It goes without saying that, depending on the embodiment of the present invention, some of the above-mentioned components may not necessarily correspond to components essential to the realization of the present invention, and depending on the embodiment, the mobile computing device 400 may include more components than the above.

[0066] In this specification, the term "module" may refer to a functional and structural combination of hardware for carrying out the technical idea of ​​the present invention and software for driving the hardware. For example, a module may refer to a logical unit of a specific code and a hardware resource for executing the specific code, and a person skilled in the art can easily deduce that a module does not necessarily refer to physically connected code or a type of hardware.

[0067] The control module 430 may control the functionality and / or resources of other components included in the mobile computing device 400 (eg, the communication module 410, the tracking module 420, the storage module 440, the transmission module 450, etc.).

[0068] The communication module 410 can communicate with an external device and transmit and receive various signals, information, and data. For example, the communication module 410 can perform wired or wireless communication with the omnidirectional image capturing device 300. The communication module 410 may include a long-distance communication module such as a 3G module, an LTE module, an LTE-A module, a Wi-Fi module, a WiGig module, an ultra-wideband (UWB) module, or a LAN card, or a short-distance communication module such as an MST module, a Bluetooth (registered trademark) module, an NFC module, an RFID module, a ZigBee module, a Z-Wave module, or an IR module.

[0069] The tracking module 420 may track the position and / or orientation of the mobile computing device 400 .

[0070] In one embodiment, the tracking module 420 may track the position and / or orientation of the mobile computing device 400 through visual simultaneous localization and mapping (VSLAM). To this end, the mobile computing device 400 may further include a camera module, and perform VSLAM on an image captured by the camera module to track the position and orientation of the mobile computing device 400. In this case, the omnidirectional image capturing device 300 and the mobile computing device 400 may be installed on the mobile platform 500 such that the shooting direction of the front camera 310 included in the omnidirectional image capturing device 300 and the shooting direction of the camera module included in the mobile computing device 400 match within a predetermined error range.

[0071] Additionally, in accordance with an embodiment, the tracking module 420 may track the location of the mobile computing device 400 in a variety of ways. For example, the tracking module 420 may track the location of the mobile computing device 400 via various Wi-Fi or Bluetooth based indoor positioning techniques, and may track the position and / or attitude of the mobile computing device 400 via an IMU and / or velocity sensor integrated into the mobile computing device 400.

[0072] The control module 430 can control the omnidirectional image capturing device 300 to transmit a predetermined capturing signal for capturing an image, and in response, when the omnidirectional image capturing device 300 captures an image, position information and / or attitude information of the mobile computing device 400 at the time of capturing the image can be obtained.

[0073] For example, the control module 430 may acquire position information and / or attitude information of the mobile computing device 400 based on the time when a capture signal is transmitted or the time when an image captured from the omnidirectional image capture device 300 is received.

[0074] Meanwhile, the storage module 440 may store images captured by the omnidirectional image capturing device 300 and position information and / or attitude information of the mobile computing device at the time the images were captured.

[0075] The transmission module 450 may transmit predetermined information to the server 100 via a wired or wireless communication network (e.g., the Internet). For example, the transmission module 450 may transmit to the server 100 an information set including a plurality of partial captured images captured by the omnidirectional image capturing device 300 and position information and / or attitude information of the mobile computing device 400 at the time when the plurality of partial captured images were captured.

[0076] In one embodiment, the information set may be in the form of JavaScript Object Notation.

[0077] The omnidirectional image capture assembly 200 enables the server 100 to capture omnidirectional images (or multiple partial captured images for generating an omnidirectional image) at various different locations in a given indoor space, and transmits the captured images to the server so that the server can perform an omnidirectional image processing method (described below) on the multiple omnidirectional images, which will be described with reference to FIG. 5.

[0078] FIG. 5 is a diagram showing an example of a plan view of a given indoor space and different shooting positions in the indoor space. Referring to FIG. 5, the omnidirectional image capture assembly 200 can move sequentially to various different positions 11 to 18 in the indoor space 10. At each of the positions 11 to 18, the omnidirectional image capture device 300 provided in the omnidirectional image capture assembly 200 can capture a plurality of partial images corresponding to the corresponding positions, and the mobile computing device 400 can transmit the partial images captured at the positions 11 to 18 to the server 100 together with information on the positions at which the corresponding images were captured and / or the attitude of the mobile computing device 400. As a result, the server 100 can receive sets of information corresponding to the positions 11 to 18, respectively, and, for example, the set of information corresponding to the position 11 may include a plurality of partial images captured at the position 11, information on the position 11, and attitude information of the mobile computing device 300 at the time of capturing images at the position 11.

[0079] Returning to FIG. 4, the rotor control module 460 may control the rotation of the rotor 600 so that the shooting direction of the front camera 310 included in the omnidirectional image capturing device 300 matches the shooting direction of the camera module included in the mobile computing device 300 within a predetermined error range.

[0080] In one embodiment, the rotor control module 460 may control the rotation of the rotor 600 based on images captured by the front camera 310 included in the omnidirectional image capture device 300 and images captured by a camera module included in the mobile computing device 300.

[0081] As described above, the server 100 can perform an omnidirectional image processing method. The omnidirectional image processing method may be a method for determining a connection relationship between a plurality of omnidirectional images corresponding to different positions in a predetermined indoor space. Here, at least two of the plurality of omnidirectional images may have a common area in which a common space is captured. This may mean that each of the plurality of omnidirectional images shares a captured space with at least one of the remaining omnidirectional images, and in this case, two images sharing a captured space may mean that at least a part of the space captured in the two images overlaps.

[0082] The omnidirectional image processing method performed by the server 100 will be described in more detail below.

[0083] FIG. 6 shows a flowchart for explaining an omnidirectional image processing method according to an embodiment of the present invention, which is performed by the server 100.

[0084] Referring to FIG. 6, the server 100 can acquire a plurality of omnidirectional images corresponding to different positions in a given indoor space (S10).

[0085] In one embodiment, the server 100 can acquire (S10) an omnidirectional image corresponding to the plurality of partial captured images based on the plurality of partial captured images received from the mobile computing device 400. A more detailed process for this is shown in Figure 7. Figure 7 is a flowchart showing an example of a process for acquiring an omnidirectional image corresponding to one capture position.

[0086] Referring to FIG. 7, in order to obtain an omnidirectional image, the server 100 receives from the mobile computing device 400 an information set including a plurality of partial captured images and position information and / or attitude information of the mobile computing device at the time the plurality of partial captured images are captured (S11), and stitches the plurality of partial captured images included in the information set to generate an omnidirectional image corresponding to the information set (S12).

[0087] For example, the server 100 can generate an omnidirectional image by performing image stitching on the front captured image, rear captured image, left captured image, and right captured image (which are included in the information set) captured by the omnidirectional image capturing device 300 from the mobile computing device 400.

[0088] Thereafter, the server 100 can store the omnidirectional image corresponding to the information set in association with the position information and / or attitude information included in the information set (S13).

[0089] Meanwhile, in another embodiment, image stitching may be performed by the mobile computing device 400. In this embodiment, the mobile computing device 400 may receive a plurality of partial captured images from the omnidirectional image capturing device 300 and stitch the received plurality of partial captured images to generate an omnidirectional image. In this case, the mobile device 400 may transmit to the server 100 an information set including the stitched omnidirectional image and position information and / or attitude information of the mobile computing device 400 at the time when the plurality of partial captured images are captured.

[0090] In another embodiment, image stitching may be performed by the omnidirectional image capturing device 300. In this case, the omnidirectional image capturing device 300 may stitch the captured partial images to generate an omnidirectional image and then transmit the omnidirectional image to the mobile computing device 400. In this way, the mobile computing device may transmit to the server 100 an information set including the received omnidirectional image and position information and / or attitude information of the mobile computing device 400 at the time when the partial images are captured.

[0091] Referring again to FIG. 6, the server may determine connectivity between a plurality of omnidirectional images (S20).

[0092] In one embodiment, the server 100 can determine the connection relationship between the multiple omnidirectional images based on the order in which the multiple omnidirectional images are received or generated. More specifically, when a certain omnidirectional image is received or generated, the server 100 can determine that the corresponding omnidirectional image and the next omnidirectional image received or generated are connected. This is because the multiple omnidirectional images or the partial captured images used to generate them are captured sequentially while the above-mentioned omnidirectional image capturing assembly 200 moves. For example, when omnidirectional images 1 to 5 are received or generated in order, the server 100 can determine that omnidirectional image 1 and omnidirectional image 2 are connected, that omnidirectional image 2 and omnidirectional image 3 are connected, that omnidirectional image 3 and omnidirectional image 4 are connected, and that omnidirectional image 4 and omnidirectional image 5 are connected.

[0093] In another embodiment, the server 100 can determine the connection relationship between the multiple omnidirectional images by performing an automatic phase mapping processing method described below. The automatic phase mapping processing method according to the technical concept of the present invention and the server 100 that performs the method will be described in more detail below.

[0094] Fig. 8 is a diagram for explaining a schematic configuration for implementing an automatic phase mapping processing method according to an embodiment of the present invention, and Fig. 9 is a diagram showing a schematic configuration of a server 100 for implementing the automatic phase mapping processing method according to an embodiment of the present invention.

[0095] The server 100 may include a memory 2 in which a program for realizing the technical idea of ​​the present invention is stored, and a processor 1 for executing the program stored in the memory 2.

[0096] An average person skilled in the art of the present invention can easily deduce that processor 1 can be named by various names such as CPU, mobile processor, etc. depending on the embodiment of server 100.

[0097] The memory 2 may be realized by any type of storage device in which a program is stored and which can be accessed by a processor to run the program. Also, in a hardware embodiment, the memory 2 may be realized by multiple storage devices rather than any one storage device. Also, the memory 2 may include not only a main storage device but also a temporary storage device. Also, it may be realized as a volatile memory or a non-volatile memory, and may be defined to include any form of information storage means realized so that a program is stored and run by a processor.

[0098] Depending on the embodiment, the server 100 may be realized in various ways, such as a web server, a computer, a mobile phone, a tablet, a television, a set-top box, etc., and may be defined to include any form of data processing device capable of performing the functions defined in this specification.

[0099] In addition, the server 100 may further include various peripheral devices 3 depending on the embodiment of the server 100. For example, a person skilled in the art of the present invention can easily deduce that the automatic phase mapping processing system 100 may further include a keyboard, a monitor, a graphic card, a communication device, etc. as peripheral devices.

[0100] For ease of understanding, the server 100 that performs the automatic topological mapping processing method will be referred to as an automatic topological mapping processing system 100 hereinafter.

[0101] The automatic phase mapping processing system 100 according to the technical concept of the present invention can recognize images that can be mapped to each other, i.e., mapping images, among a plurality of images. In addition, depending on the embodiment, the automatic phase mapping processing system 100 may perform mapping between the recognized mapping images.

[0102] The mapping images may refer to images that have the closest phase relationship to each other. The closest phase relationship may be not only close in distance but also directly movable to each other in space, and such an example may be the image that contains the most common space. Also, performing mapping may refer to matching between two images as described above, but the present invention will be described mainly in the case of grasping the phase of two images, that is, the relative positional relationship.

[0103] For example, as shown in Fig. 8a, multiple (e.g., five) images may be input to the automatic phase mapping processing system 100. After that, the automatic phase mapping processing system 100 can determine which images among the multiple images can be mapped to each other, i.e., which mapping images are, and perform mapping of the determined mapping images.

[0104] For example, in an embodiment of the present invention, the images may be omnidirectional images (i.e., 360-degree images) taken at different positions, and the mapping images may be a pair of images that share the most common space with each other.

[0105] For example, as shown in FIG. 8b, the images taken at positions a, b, c, d, and e may be image 1, image 2, image 3, image 4, and image 5, respectively.

[0106] In such a case, image 1, image 2, and image 3 may have a significant amount of common space in a common captured image, but relatively more common space may be included between image 1 and image 2. Thus, the mapping image of image 1 may be image 2.

[0107] Then, a mapping image must be searched for image 2, at which point image 1, whose mapping image has already been determined, can be excluded. Then, the mapping image of image 2 can be image 3.

[0108] In this way, the mapped image of image 3 can be image 4, the mapped image of image 4 can be image 5, etc.

[0109] After that, the automatic phase mapping processing system 100 can perform mapping for image 2, which is a mapping image, with image 1 as a reference. That is, it is possible to grasp the phase of image 2 with respect to image 1, i.e., the relative position of image 2 with respect to image 1. Also, by grasping the phase of image 3 with respect to image 2, the phase of image 4 with respect to image 3, and the phase of image 5 with respect to image 4 in order, it is possible to identify the phase relationship between all images.

[0110] Ultimately, in the past, when multiple omnidirectional images existed and the exact position of each omnidirectional image was not known, it could take a significant amount of time and resources to determine the positional relationships between the multiple images.

[0111] For example, according to the conventional method, it is necessary to extract predetermined feature points for each image and use the extracted feature points to determine how many common feature points exist for each image pair. In addition, the image pair with the most common feature points can be recognized as mapping images, and mapping, i.e., relative positional relationship, can be determined according to the positions of the common feature points. If matching is required, a transformation matrix is ​​determined to overlap the common feature points with a minimum error, and the two images can be linked (matched) by transforming one of the images through such a transformation matrix.

[0112] However, the feature points used in such conventional methods require a considerable amount of time and calculation to extract the feature points. In addition, in order to grasp the mapping image, a calculation must be performed to compare the feature points for every image pair, and the more feature points an image has, the more time it takes to perform such a calculation.

[0113] However, as described above, according to the technical concept of the present invention, it is possible to quickly, accurately and automatically search for a mapping image from among such a plurality of images, and perform mapping for the found mapping image.

[0114] To solve this problem, the automatic phase mapping processing system 100 according to the teachings of the present invention can use neural network features.

[0115] A neural network feature as defined herein may refer to all or a portion of features selected in a feature map of a given layer of a neural network trained to achieve a given objective.

[0116] These features are used in a neural network (e.g., a convolutional neural network) trained to achieve a particular goal, and may be information derived by the trained neural network when the neural network is trained to achieve the particular goal.

[0117] For example, there is a neural network 20 as shown in FIG. 10a, which may be a convolutional neural network (CNN).

[0118] In such a case, multiple layers 21, 22, 23, 24 may be included in the neural network 20, and there may be an input layer 21, an output layer 24, and multiple hidden layers 22, 23. The output layer 24 may be a fully connected layer that is fully connected to the previous layer, and the automatic phase mapping processing system 100 according to the technical concept of the present invention may select neural network features f1, f2, f3 from the output layer 24 or any feature map-containing layer (e.g., 23) before the fully connected layer.

[0119] The neural network features f1, f2, and f3 used by the automatic phase mapping processing system 100 may be all the features included in the feature map of the corresponding layer, or may be some selected features from among them.

[0120] The automatic topological mapping processing system 100 can use these features to grasp mapping images or perform mapping between mapping images by using them instead of traditional handcrafted features such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented FAST and Rotated BRIEF (ORB). That is, features used in convolutional neural networks may be used instead of traditional handcrafted features.

[0121] This is because image features preferably have the same characteristics regardless of scale or orientation, and in a convolutional neural network, layers prior to the output layer 24 have such features through multiple nonlinear convolutional functions and / or pooling functions, etc. Furthermore, traditional handcrafted features are extracted only from human-defined characteristic locations, such as corners in an image, and typically only from where edges exist (e.g., where an edge bends, etc.).

[0122] However, neural network features have the advantage that the neural network 20 can be trained to look not only at these locations but also in flat regions of the image. Also, while handcrafted features often fail to detect feature points despite the need to detect them depending on image distortion and image quality, neural network features are far more robust to such image distortions, and the accuracy of feature extraction can be improved.

[0123] The neural network 20 may itself be a feature extractor. For example, when features are selected from the output layer 24 or the layer 23 immediately preceding the fully connected layer, the output layer 24 may be designed to output the selected features f1, f2, and f3 of the immediately preceding layer 23 themselves, and in such a case, the neural network 20 itself may function as a feature extractor.

[0124] Alternatively, the neural network 20 may be trained to achieve separate, specific goals (e.g., classification, object detection, etc.). In such cases, consistent features can still be selected from a given layer to be used as neural network features. For example, in the case of FIG. 10a, the combination of the remaining layers, excluding the output layer 24, can function as a feature extractor.

[0125] According to one embodiment of the present invention, the neural network 20 may be a neural network that has been trained to derive an optimal transformation relationship (e.g., one that minimizes error) so that, after dividing one of the images so that there are overlapping areas, corresponding points extracted from each of the overlapping common areas of each of the divided images are matched.

[0126] For example, as shown in Fig. 10b, all or part of a given image 6 may be divided so that there is an overlapping common area 6-3, and a given number of corresponding points (e.g., P11 to P14, P21 to P24) may be extracted from each of the divided images 6-1 and 6-2.

[0127] This makes it possible to realize a neural network 20 as a neural network that is trained (for example, to determine parameters of a transformation matrix) so that points P11 to P14 extracted from the first sub-image 6-1 can be transformed to points P21 to P24 extracted from the second sub-image 6-2 with minimal error.

[0128] At this time, the points (for example, P11 to P14, P21 to P24) may be arbitrarily selected points, or may be feature points extracted by a predetermined method from a common region of each image.

[0129] In any case, all or a portion of neural network 20, once sufficiently trained to achieve a desired purpose, can be used as a feature extractor to select and extract features from an image.

[0130] In addition, by using such a feature extractor, the same features can be extracted from common regions included in different images input to the automatic phase mapping processing system 100. Therefore, an image that has the most common features (mutually corresponding features) can be determined as a mapping image.

[0131] On the other hand, according to the technical idea of ​​the present invention, since neural network features are represented as vectors, in order to search for a mapping image of a specific image, it may be possible to determine the positional relationship more quickly by using a vector search engine capable of high-speed calculations, rather than comparing the features of each image pair as in the conventional method.

[0132] Techniques for quickly searching large vectors have been widely published recently.

[0133] A vector search engine is an engine that is built to quickly find vectors that are closest (in close proximity) to an input vector (or vector set). All vectors are indexed and stored in a DB, and the vector search engine may be designed to output vectors (or vector sets) that are closest to the input vector (or vector set).

[0134] Such a vector search engine can be constructed using known vector search techniques such as faiss, etc. When such a vector search engine is implemented on a GPU, it has the effect of enabling high-speed calculations of large amounts of data.

[0135] The vector search engine according to the technical concept of the present invention can receive a set of features extracted from a target image (e.g., image 1) and output the most similar (closest) vector or set of vectors in response to the input. It can also quickly determine the mapping image of the target image by determining which image the vector or set of vectors originates from.

[0136] For example, all of the features extracted from the first image can be input to a vector search engine, which can output the vector with the shortest distance to each of the input features from the vector DB, or the distance between each of the input features and the vector with the shortest distance. Such a task can be performed for each image.

[0137] For example, if there are five images and ten features are extracted from each image, 50 vectors can be indexed and stored in the vector DB. Information about each source image can also be stored.

[0138] The vector search engine can then receive the 10 vectors extracted from the first image. The vector search engine can output the 10 vectors that have the shortest distances from each of the 10 vectors to the vectors extracted from the second image, or the sum of the distances. In this way, when the vectors extracted from the third image, the vectors extracted from the fourth image, and the vectors extracted from the fifth image are searched for, an image containing a feature set that is most similar to the input vector set can be quickly searched for. The searched image can then be determined as the mapping image of the first image.

[0139] In some embodiments, the vector search engine may output the vector with the shortest distance from each of the 10 vectors output from the first image to the remaining vectors (40 vectors) excluding the 10 vectors extracted from the first image. For example, when the list of 10 vectors is output, the automatic topological mapping processing system 100 may analyze such a vector list to output a mapping image.

[0140] The results or methods output by the vector search engine may vary. In any case, however, according to the technical idea of ​​the present invention, features can be extracted from each input image and these features can be input to a DB that is constructed to enable vector search, and the vector search engine can perform a function of outputting the most similar (shortest distance) vector or set of vectors when a vector or set of vectors is input. This function allows for high-speed search of mapping images.

[0141] In some embodiments, not all features of a target image, i.e., an image for which a mapping image is to be found (e.g., a first image), may be input, and only some features may be input to the vector search engine. For example, only features corresponding to a predefined region of the image may be input to the vector search engine to grasp the positional relationship. The predefined region is usually not the center of the image but an area adjacent to the left, right, top, and bottom corners, so an outer region of the image may be arbitrarily set, and features at positions corresponding to the set region may be selectively used as input for vector search. Of course, only features corresponding to such an outer region may be input to the vector DB, or all features may be input.

[0142] Furthermore, the neural network features according to the technical concept of the present invention are not themselves localized in the extracted image. Therefore, the mapping can only be performed if the points in the original image corresponding to the neural network features are identified. Therefore, the technical concept of identifying the points in the original image corresponding to the neural network features is necessary, which will be described below with reference to FIG. 13.

[0143] An automatic phase mapping processing system 100 for realizing the above-mentioned technical idea can be defined as a functional or logical configuration as shown in FIG.

[0144] FIG. 11 is a diagram for explaining a schematic logical configuration of an automatic phase mapping processing system according to an embodiment of the present invention.

[0145] 11, an automatic phase mapping processing system 100 according to the technical concept of the present invention includes a control module 110, an interface module 120, and a feature extractor 130. The automatic phase mapping processing system 100 may further include a mapping module 140 and / or a vector search engine 150.

[0146] The automatic phase mapping processing system 100 may refer to a logical configuration including hardware resources and / or software necessary to realize the technical idea of ​​the present invention, and does not necessarily refer to one physical component or one device. In other words, the automatic phase mapping processing system 100 may refer to a logical combination of hardware and / or software provided to realize the technical idea of ​​the present invention, and may be realized as a set of logical configurations for realizing the technical idea of ​​the present invention by providing the hardware and / or software in devices separated from each other as necessary and performing their respective functions. In addition, the automatic phase mapping processing system 100 may refer to a set of configurations that are realized separately for each function or role for realizing the technical idea of ​​the present invention. For example, the control module 110, the interface module 120, the feature extractor 130, the mapping module 140, and / or the vector search engine 150 may be located in different physical devices or in the same physical device. Further, depending on the embodiment, the software and / or hardware combinations constituting each of the control module 110, the interface module 120, the feature extractor 130, the mapping module 140, and / or the vector search engine 150 may also be located on different physical devices, and the components located on the different physical devices may be organically combined with each other to realize each module.

[0147] The control module 110 can control other components included in the automatic phase mapping processing system 100 (e.g., the interface module 120, the feature extractor 130, the mapping module 140 and / or the vector search engine 150) to realize the technical concept of the present invention.

[0148] The interface module 120 may receive a plurality of images from the outside. The plurality of images may be images taken at different positions. In one example, the plurality of images may be, but is not limited to, omnidirectional images taken indoors.

[0149] Among the multiple images, there may be images of a common space taken at different positions, and two images that include the common space, i.e., a common area, can be defined as being in a mappable relationship. The image that includes the most common area among them can be defined as a mapping image, which may be defined as an image with the most corresponding features.

[0150] From each of the multiple images input via the interface module 120, the feature extractor 130 can extract features defined in accordance with the technical concept of the present invention, i.e., neural network features.

[0151] As mentioned above, the neural network features may be features of an image specified before the output layer in a given neural network (e.g., CNN). The feature extractor 130 may be the neural network 20 itself as shown in Fig. 10a, or may refer to a configuration from the input layer 21 to a predetermined layer (e.g., 23) before the output layer 24 in the neural network. All or some of the features included in the feature map defined by layer 23 may be neural network features.

[0152] Neural network 20 may be a neural network designed to align two images with minimal error as described above, or may be trained for the purpose of extracting neural network features, although it may be trained for purposes other than extracting neural network features (e.g., classification, detection, etc.).

[0153] For example, in the latter case, the neural network 20 may be trained to output handcrafted feature points that can adequately express the position and / or image features arbitrarily set by the user, in which case the neural network 20 itself may become the feature extractor 130.

[0154] The position arbitrarily set by the user may be set to a position set by the user on a specific object (e.g., a wall, a door, etc.) (e.g., the center position of the object). Furthermore, unlike conventional handcraft feature points, such a user-set position may be set in a flat region, i.e., an edge or a flat image region without edges. In such a case, the conventional handcraft feature points can define features even in flat image regions that are not extracted as feature points, and when this is utilized, more accurate mapping image determination and mapping can be performed.

[0155] FIG. 12 is a diagram for explaining the advantages of using neural network features according to an embodiment of the present invention.

[0156] As shown in FIG. 12, the feature extractor 130 may be trained to identify arbitrary positions within a predefined object (eg, a wall, a door, a table) as feature points fp1, fp2, and fp3.

[0157] Also, as shown in FIG. 12, the arbitrary position can be set within a normally flat image area, such as a predetermined position for each object (e.g., the center of a wall, the center of a table, the center of a door, etc.).

[0158] Of course, the feature extractor 130 may be trained to extract features that correspond to handcraft features, such as traditional edges or edge breaks.

[0159] For example, a user may annotate handcraft feature points and the set positions of flat regions set by the user for each object in multiple images, and use this as training data to train the neural network 20. In such a case, features corresponding to each of the feature points fp1, fp2, and fp3 can be extracted, and the feature points themselves may be output.

[0160] In any case, when using neural network features, it is possible to utilize locations that are not extracted by traditional handcraft features as shown in Figure 5, which may have a more advantageous effect in defining image characteristics and mapping images.

[0161] On the other hand, although a neural network feature is characteristic information of an image that is determined by multiple convolutions and / or pooling so that the neural network 20 outputs a desired objective, such a neural network feature itself may not represent a specific location within the corresponding original image.

[0162] Therefore, even if neural network features are extracted, it is still necessary to identify the locations in the original image that correspond to the neural network features, i.e., the feature locations, since only when such feature locations are identified can image mapping be performed.

[0163] The technical idea for identifying the feature locations of neural network features in this way will now be described with reference to FIG.

[0164] FIG. 13 is a diagram illustrating feature locations corresponding to neural network features according to an embodiment of the present invention.

[0165] As shown in Fig. 13, a neural network feature f can be extracted from a given layer. In this case, the neural network feature f corresponds to a given corresponding region S1 in a previous given layer 1, and pixel information included in such corresponding region S1 can be mapped to the neural network feature f by a predefined convolution and pooling function.

[0166] In this case, a predetermined position (for example, the center or a specific vertex) in the corresponding region S1 of the neural network feature f in the l layer can be defined as the corresponding position (PSl) of the neural network feature f in the l layer.

[0167] Similarly, a corresponding region So on the original image corresponding to a corresponding position PSl in the l layer can be identified by the convolution and pooling relationship between the original image and the l layer, and a predetermined position (e.g., the center) within the corresponding region So can be identified as the corresponding position on the original image of the neural network feature f, i.e., the feature position.

[0168] Once the feature position is determined for each neural network feature in this manner, each feature position can become a feature point for image mapping.

[0169] This allows the mapping module 140 to perform image mapping using the locations of corresponding features between the mapping images.

[0170] Image mapping between two images can be performed using corresponding points in each of the two images in the case of mapping that specifies a relative positional relationship between the two images. In such a case, the corresponding points may be feature points of neural network features extracted from each of the two images, and the corresponding feature points can be easily searched for via a vector search engine.

[0171] When corresponding points (representing the same position in space) exist in different images, the technical concept of determining the relative positions of these two images is known.

[0172] For example, it would be easy for an average person skilled in the art of the present invention to deduce that the relative positional relationship can be determined using epipolar geometry. Various other methods may also be possible.

[0173] According to another embodiment, if a mapping between two images, i.e., a mapping image, aligns the two images, identifying a transformation matrix to align the two images may be performing the mapping.

[0174] It is widely known that to identify such a transformation matrix, three pairs of corresponding features can be extracted and a transformation matrix can be defined so that the extracted three pairs can be transformed. Therefore, these three pairs of features can be searched so that all features are transformed with the minimum error, and it goes without saying that an algorithm such as RANSAC can be used.

[0175] The vector search engine 150 inputs vectors corresponding to each image feature extracted by the feature extractor 130 as described above into the DB, and may input a set of vectors corresponding to a set of features extracted from a target image (e.g., the first image), thereby enabling vector search results to be output as described above.

[0176] The control module 110 can then determine images that are in adjacent relationship with the target image based on the vector search results.

[0177] FIG. 14 is a flowchart illustrating a method for searching for a mapping image between images in the automatic phase mapping processing method according to an embodiment of the present invention.

[0178] 14, an automatic phase mapping processing system 100 according to the technical idea of ​​the present invention can extract neural network features from each of a plurality of images (S100). Then, the features are built into a vector DB, and a vector search can be performed on the vector set (feature set) extracted from the target image (S110, S120).

[0179] A mapping image for the target image can then be determined based on the vector search results (S130), and a mapping image for each image can be determined by performing the same task for all images (S140).

[0180] FIG. 15 is a flow chart illustrating a method for mapping an image in an automatic phase mapping process according to an embodiment of the present invention.

[0181] 15, an automatic phase mapping processing system 100 according to the technical idea of ​​the present invention can identify the position of a feature corresponding to a feature extracted from a first image in order to map a first image and a second image determined to be mapping images with each other (S200). For this purpose, a method as shown in FIG. 13 can be used.

[0182] Additionally, features corresponding to the features extracted from the second image may be located (S210).

[0183] Then, the automatic phase mapping processing system 100 can determine the relative positional relationship via an epipolar geometry algorithm based on the positions of the features in each image, or determine a transformation matrix for image stitching in a predetermined manner (e.g., a RANSAC algorithm) (S220).

[0184] Referring again to FIG. 6, the server 100 can arrange, on a floor plan corresponding to the indoor space, a plurality of omnidirectional images each corresponding to a different position in the indoor space (S30).

[0185] More specifically, the server 100 can acquire a floor plan corresponding to an indoor space. For example, the server 100 can receive a floor plan 10 as shown in FIG.

[0186] Thereafter, the server 100 can receive a first point on the floor plan corresponding to the first position information included in a first information set, which is one of the plurality of information sets, and a second point on the floor plan corresponding to the second position information included in a second information set, which is the other of the plurality of information sets. For example, the server 100 may receive coordinates on the floor plan 10 corresponding to position 11, and may receive coordinates on the floor plan 10 corresponding to position 12.

[0187] The server 100 can then arrange multiple omnidirectional images on the plan view based on the positional relationship between the first position represented by the first position information and the second position represented by the second position information, and the positional relationship between the first point and the second point.

[0188] As described above, the server 100 knows the relative positional relationship between the omnidirectional images corresponding to each position by the above-mentioned automatic phase mapping processing method. Therefore, the server calculates a predetermined parameter that matches the positional relationship between the first point and the second point (e.g., the distance and direction between the two points) and the positional relationship between the first point and the second point, and then applies the parameter to the relative positions between the positions calculated by the above-mentioned automatic phase mapping processing method, thereby determining the positions on the plan view where the multiple omnidirectional images are arranged.

[0189] Meanwhile, the method according to the embodiment of the present invention can be realized in the form of computer readable program instructions and stored in a computer readable recording medium, which includes any kind of recording device in which data that can be read by a computer system is stored.

[0190] The program instructions contained on the recording media may be those specially designed and constructed for the present invention, or they may be of the type well known and available to those having skill in the software arts.

[0191] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, floptical disks, magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, etc. Also, the computer-readable recording media can be distributed among computer systems connected via a network to store and execute computer-readable code in a distributed manner.

[0192] Examples of program instructions include not only machine code, such as produced by a compiler, but also higher level language codes that may be executed by a device for processing information electronically, such as a computer, using an interpreter or the like.

[0193] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of this invention, and vice versa.

[0194] The above description of the present invention is given for illustrative purposes only, and it will be understood by those skilled in the art to which the present invention pertains that the present invention can be easily modified into other specific forms without changing the technical concept or essential characteristics of the present invention.

[0195] Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not restrictive. For example, each component described in a single form may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form.

[0196] The scope of the present invention is indicated by the claims set forth below rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention. [Industrial Applicability]

[0197] The present invention can be used in an omnidirectional image processing method and a server that performs this method.

Claims

1. An omnidirectional image processing method performed on a server, comprising: acquiring a plurality of omni-directional images corresponding to different positions in a predetermined indoor space, where at least two of the plurality of omni-directional images have a common area in which a common space is captured; a processing step of determining connectivity between the plurality of omnidirectional images; Including, The acquiring step includes: receiving a set of information from a mobile computing device, the set of information including an omnidirectional image and location information of the mobile computing device at a time when the omnidirectional image is captured; storing the omnidirectional image included in the information set in association with the location information included in the information set; Including, The mobile computing device comprises: Mounted on a mobile stand, Communicating with an omnidirectional image capturing device provided on the mobile stand; Tracking the location of the mobile computing device; When the omnidirectional image capture device captures an image, location information of the mobile computing device at the time of capture is obtained; An omnidirectional image processing method, which transmits to the server an information set including an omnidirectional image generated by stitching together a plurality of partial images captured by the omnidirectional image capturing device, and location information of the mobile computing device at the time the plurality of partial images were captured.

2. The mobile computing device comprises: further tracking the pose of the mobile computing device; When an image is captured by the omnidirectional image capture device, further acquiring attitude information of the mobile computing device at the time of capture; The set of information transmitted by the mobile computing device further includes orientation information of the mobile computing device at a time when the plurality of partial images were captured; and The step of storing the omnidirectional image included in the information set in association with the position information included in the information set includes: storing the omnidirectional image included in the information set in association with the position information included in the information set and the attitude information included in the information set; The omnidirectional image processing method according to claim 1 ,

3. the mobile computing device includes a camera module; The omnidirectional image capturing device and the mobile computing device, The omnidirectional image capturing device is provided on the movable stand such that a capturing direction of a front camera module included in the omnidirectional image capturing device and a capturing direction of a camera module included in the mobile computing device are aligned within a predetermined error range; The mobile computing device comprises: The omnidirectional image processing method of claim 2 , further comprising: performing visual simultaneous localization and mapping (VSLAM) on the images captured by the camera module to track the position and orientation of the mobile computing device.

4. obtaining a floor plan corresponding to the indoor space; receiving a first point on the floor plan corresponding to first position information included in a first information set, which is one of the plurality of information sets, and a second point on the floor plan corresponding to second position information included in a second information set, which is the other of the plurality of information sets; 2. The omnidirectional image processing method according to claim 1, further comprising a step of arranging the plurality of omnidirectional images on the plan view based on a positional relationship between a first position represented by the first position information and a second position represented by the second position information, and a positional relationship between the first point and the second point.

5. The processing steps include: extracting features from each of the plurality of omnidirectional images using a neural network feature extractor; and determining a mapping image of each of the plurality of omnidirectional images based on features extracted from each of the plurality of omnidirectional images, thereby determining a connectivity relationship between the plurality of omnidirectional images.

6. determining a mapping image of each of the plurality of omnidirectional images to determine a connection relationship between the plurality of omnidirectional images, constructing, in a DB, vectors corresponding to features extracted from each of the plurality of omnidirectional images; performing a vector search in the constructed DB for a vector set that is at least a portion of first vectors corresponding to a first feature extracted from a predetermined first image among the plurality of omnidirectional images; and determining a second image extracted based on a vector search result as a mapping image of the first image.

7. The omnidirectional image processing method described in claim 6, characterized in that the step of determining the connection relationship between the plurality of omnidirectional images by determining a mapping image of each of the plurality of omnidirectional images further includes a step of mapping a first image and a second image determined to be a mapping image of the first image.

8. The step of mapping the first image and the second image determined to be a mapping image of the first image includes: determining first feature corresponding positions on the first image corresponding to each first feature extracted from the first image, and determining second feature corresponding positions on the second image corresponding to each second feature extracted from the second image; The omnidirectional image processing method according to claim 7 , further comprising a step of determining a relative positional relationship between the first image and the second image based on the determined first feature corresponding position and the determined second feature corresponding position.

9. The neural network comprises:

6. The omnidirectional image processing method according to claim 5, further comprising a neural network trained to output a transformation relationship that enables optimal matching of corresponding points extracted from the overlapping common areas of each of the divided images after dividing a predetermined image so that overlapping areas exist.

10. The processing steps include: extracting features from each of the plurality of omnidirectional images; constructing, in a DB, vectors corresponding to features extracted from each of the plurality of omnidirectional images; performing a vector search in the constructed DB for a vector set that is at least a portion of first vectors corresponding to a first feature extracted from a predetermined first image among the plurality of omnidirectional images; and determining a connection relationship between the plurality of omnidirectional images by determining a second image extracted based on a vector search result as a mapping image of the first image.

11. An omnidirectional image processing method performed on a server, comprising: acquiring a plurality of omni-directional images corresponding to different positions in a predetermined indoor space, where at least two of the plurality of omni-directional images have a common area in which a common space is captured; a processing step of determining connectivity between the plurality of omnidirectional images; Including, The acquiring step includes: receiving, from a mobile computing device, a set of information including a plurality of partial images and location information of the mobile computing device at the time the plurality of partial images are captured; stitching a plurality of partial images included in the information set to generate an omnidirectional image corresponding to the information set; storing an omnidirectional image corresponding to the information set in association with location information included in the information set; Including, The mobile computing device comprises: Mounted on a mobile stand, Communicating with an omnidirectional image capturing device provided on the mobile stand; Tracking the location of the mobile computing device; When the omnidirectional image capture device captures an image, location information of the mobile computing device at the time of capture is obtained; An omnidirectional image processing method that transmits to the server a set of information including a plurality of partial images captured by the omnidirectional image capturing device and location information of the mobile computing device at the time the plurality of partial images were captured.

12. A computer-readable recording medium having a program recorded thereon for carrying out the method according to any one of claims 1 to 11.

13. A computer program stored on a computer-readable recording medium for performing the method according to any one of claims 1 to 11.

14. A server, A processor; A memory in which a program is stored; Including, The program, when executed by the processor, causes the server to perform an omnidirectional image processing method; The omnidirectional image processing method includes: acquiring a plurality of omni-directional images corresponding to different positions in a predetermined indoor space, where at least two of the plurality of omni-directional images have a common area in which a common space is captured; determining connectivity between the plurality of omnidirectional images; The acquiring step includes: receiving a set of information from a mobile computing device, the set of information including an omnidirectional image and location information of the mobile computing device at a time when the omnidirectional image is captured; storing the omnidirectional image included in the information set in association with the location information included in the information set; Including, The mobile computing device comprises: Mounted on a mobile stand, Communicating with an omnidirectional image capturing device provided on the mobile stand; Tracking the location of the mobile computing device; When the omnidirectional image capture device captures an image, the location information of the mobile computing device at the time of capture is obtained; A server that transmits to the server an information set including an omnidirectional image generated by stitching together a plurality of partial images captured by the omnidirectional image capturing device, and location information of the mobile computing device at the time the plurality of partial images were captured.

15. The omnidirectional image processing method includes: obtaining a floor plan corresponding to the indoor space; receiving a first point on the floor plan corresponding to first position information included in a first information set, which is one of the plurality of information sets, and a second point on the floor plan corresponding to second position information included in a second information set, which is the other of the plurality of information sets; The server of claim 14, further comprising a step of arranging the plurality of omnidirectional images on the plan view based on a positional relationship between a first position represented by the first position information and a second position represented by the second position information, and a positional relationship between the first point and the second point.

16. The processing steps include: extracting features from each of the plurality of omnidirectional images using a neural network feature extractor; and determining a mapping image for each of the plurality of omnidirectional images based on features extracted from each of the plurality of omnidirectional images to determine a connectivity relationship between the plurality of omnidirectional images.

17. determining a mapping image of each of the plurality of omnidirectional images to determine a connection relationship between the plurality of omnidirectional images, constructing, in a DB, vectors corresponding to features extracted from each of the plurality of omnidirectional images; performing a vector search in the constructed DB for a vector set that is at least a portion of first vectors corresponding to a first feature extracted from a predetermined first image among the plurality of omnidirectional images; The server of claim 16, further comprising determining a second image extracted based on vector search results as a mapping image of the first image.

18. The step of determining a connection relationship between the plurality of omnidirectional images by determining a mapping image of each of the plurality of omnidirectional images, comprises: The server of claim 17, further comprising the step of: mapping a first image and a second image determined to be a mapping image of the first image.

19. The step of mapping the first image and the second image determined to be a mapping image of the first image includes: determining first feature corresponding positions on the first image corresponding to each first feature extracted from the first image, and determining second feature corresponding positions on the second image corresponding to each second feature extracted from the second image; and determining a relative positional relationship between the first image and the second image based on the determined first feature corresponding position and the determined second feature corresponding position.

20. The neural network comprises: The server of claim 18, which is a neural network trained to output a transformation relationship that can optimally match corresponding points extracted from the overlapping common areas of each of the divided images after a given image is divided so that there are overlapping areas.

21. The processing steps include: extracting features from each of the plurality of omnidirectional images; constructing, in a DB, vectors corresponding to features extracted from each of the plurality of omnidirectional images; performing a vector search in the constructed DB for a vector set that is at least a portion of first vectors corresponding to a first feature extracted from a predetermined first image among the plurality of omnidirectional images; and determining a second image extracted based on a vector search result as a mapping image of the first image, thereby determining a connection relationship between the plurality of omnidirectional images.

22. A server, A processor; A memory in which a program is stored; Including, The program, when executed by the processor, causes the server to perform an omnidirectional image processing method; The omnidirectional image processing method includes: acquiring a plurality of omni-directional images corresponding to different positions in a predetermined indoor space, where at least two of the plurality of omni-directional images have a common area in which a common space is captured; a processing step of determining connectivity between the plurality of omnidirectional images; Including, The acquiring step includes: receiving, from a mobile computing device, a set of information including a plurality of partial images and location information of the mobile computing device at the time the plurality of partial images are captured; stitching a plurality of partial images included in the information set to generate an omnidirectional image corresponding to the information set; storing an omnidirectional image corresponding to the information set in association with location information included in the information set; Including, The mobile computing device comprises: Mounted on a mobile stand, Communicating with an omnidirectional image capturing device provided on the mobile stand; Tracking the location of the mobile computing device; When an image is captured by the omnidirectional image capture device, location information of the mobile computing device at the time of capture is obtained; A server transmits to the server an information set including the captured image captured by the omnidirectional image capture device and location information of the mobile computing device at the time the captured image was captured.

Images