Map information generation device, position identification device, map information generation method, position identification method, program, and recording medium

Abstract

To provide a map information generation device with which it is possible to generate more accurate map information.SOLUTION: In this map information generation device 10, an image acquisition unit 11 acquires three-dimensional image information and two-dimensional image information in which a specific area is imaged, a three-dimensional model generation unit 12 generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information, the first and second three-dimensional models indicating a composition in the specific area and including point cloud data having shape information. Meanwhile, a map information generation unit 13 associates the position of point cloud data in the first three-dimensional model with the position of point cloud data in the second three-dimensional model and generates the map information of the specific area.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a map information generation device, a position identification device, a map information generation method, a position identification method, a program, and a recording medium.

Background Art

[0002] Techniques for estimating the position of a terminal from an image captured by the terminal have been reported (for example, Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in order to more accurately identify the position of the terminal that captured the image, it is necessary to use more accurate map information. For this reason, a technique for generating more accurate map information is required.

[0005] Therefore, an object of the present invention is to provide a map information generation device, a position identification device, a map information generation method, a position identification method, a program, and a recording medium that can generate more accurate map information.

Means for Solving the Problems

[0006] To achieve the above object, the map information generation device of the present invention includes an image acquisition unit, a three-dimensional model generation unit, and a map information generation unit, the image acquisition unit acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area, the three-dimensional model generation unit generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information, The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area, and include point cloud data having shape information. The map information generation unit is a device that associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model, and generates map information of the specific area.

[0007] The map information generation method of the present invention includes an image acquisition step, a three-dimensional model generation step, and a map information generation step. In the image acquisition step, three-dimensional image information and two-dimensional image information obtained by imaging a specific area are acquired. In the three-dimensional model generation step, a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information are generated. The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area, and include point cloud data having shape information. In the map information generation step, the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model are associated with each other, and map information of the specific area is generated.

Advantages of the Invention

[0008] According to the present invention, more accurate map information can be generated. Further, according to the present invention, using the map information, for example, the position of the terminal that captured the image can be specified more accurately.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0010] In the map information generation device of the present invention, for example, the three-dimensional image information may be in a form including a full-surround panoramic image obtained by three-dimensionally imaging the specific area.

[0011] The map information generation device of the present invention, for example, further includes a feature point extraction unit, the feature point extraction unit extracts feature points from the two-dimensional image information by machine learning, and the map information generation unit generates the map information associated with the feature points. This may also be the case.

[0012] The position identification device of the present invention includes a map information generation unit, a user terminal information acquisition unit, and an identification unit, the map information generation unit is the map information generation device of the present invention, the user terminal information acquisition unit acquires a captured image and sensor information from the user terminal, The specific unit is a device that identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

[0013] In the position identification device of the present invention, for example, The map information generation unit is a map information generation device including a feature point extraction unit, The feature point extraction unit extracts feature points from the captured image by machine learning, The specific unit identifies a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image, Identifies the position of the user terminal from the candidate points, and It may also be in a mode of identifying the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

[0014] In the map information generation method of the present invention, for example, The three-dimensional image information may also be in a mode including a full-surround panorama image obtained by three-dimensionally imaging the specific area.

[0015] The map information generation method of the present invention, for example, Further includes a feature point extraction step, In the feature point extraction step, feature points are extracted from the two-dimensional image information by machine learning, The map information generation step may be in a mode of generating the map information associated with the feature points.

[0016] The position identification method of the present invention Includes a map information generation step, a user terminal information acquisition step, and a specific step, The map information generation step is a step of executing each step of the map information generation method of the present invention, The user terminal information acquisition step acquires a captured image and sensor information from the user terminal, The specific process is a method of identifying the position of the user terminal based on the point cloud data of the map information and the captured image, and identifying the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

[0017] In the position identification method of the present invention, for example, The map information generation process is a process of executing each process of a map information generation method including a feature point extraction process. The feature point extraction process extracts feature points from the captured image by machine learning. The specific process identifies a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image. Identify the position of the user terminal from the candidate points, and It may be a mode of identifying the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

[0018] The program of the present invention is a program for causing a computer to execute each process of the method of the present invention as a procedure.

[0019] The recording medium of the present invention is a computer-readable recording medium recording the program of the present invention.

[0020] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following figures, the same parts are denoted by the same reference numerals. In addition, the descriptions of the respective embodiments can be mutually referred to unless otherwise specified, and the configurations of the respective embodiments can be combined unless otherwise specified.

[0021] [Embodiment 1] FIG. 1 is a block diagram showing a configuration example of a map information generation device 10 according to the present embodiment. As shown in FIG. 1, the device 10 includes an image acquisition unit 11, a three-dimensional model generation unit 12, and a map information generation unit 13. Further, the device 10 may further include a feature point extraction unit 14 as an arbitrary configuration, for example.

[0022] The device 10 may be, for example, one device including the above-described respective parts, or the above-described respective parts may be devices connectable via a communication line network. Further, the device 10 can be connected to an external device described later via the communication line network. The communication line network is not particularly limited, and a known network can be used. For example, it may be wired or wireless. Examples of the communication line network include an Internet line, WWW (World Wide Web), a telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (local 5G), and the like. Examples of the wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, LPWA, and the like. The wireless communication may be in a form in which each device directly communicates (Ad Hoc communication), infrastructure communication, indirect communication via an access point, or the like. The device 10 may be incorporated into a server as a system, for example. Further, the device 10 may be, for example, a personal computer (PC, for example, a desktop type or a notebook type), a smartphone, a tablet terminal, etc. in which the program of the present invention is installed. Furthermore, the device 10 may be in a form such as cloud computing or edge computing in which at least one of the above-described respective parts is on a server and the other above-described respective parts are on a terminal, for example.

[0023] FIG. 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, an imaging device 108, an audio input / output device 109, and the like. Each part of the device 10 is interconnected via the bus 103 by respective interfaces (I / F).

[0024] The central processing unit 101 is responsible for the overall control of the apparatus 10. In the apparatus 10, programs of the present invention and other programs are executed by the central processing unit 101, and various information is read and written. Specifically, for example, the central processing unit 101 functions as an image acquisition unit 11, a three-dimensional model generation unit 12, a map information generation unit 13, and a feature point extraction unit 14.

[0025] The bus 103 can be connected to, for example, an external device. Examples of the external device include an external storage device (such as an external database), a printer, an external input device, an external display device, an external imaging device, and the like. The apparatus 10 can be connected to an external network (the communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

[0026] The memory 102 includes, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operation programs such as the program of the present invention stored in the storage device 104 described later, and the central processing unit 101 receives data from the memory 102 and executes the program. The main memory is, for example, a RAM (random access memory). Further, the memory 102 may be, for example, a ROM (read-only memory).

[0027] The storage device 104 is also referred to as a so-called auxiliary storage device with respect to the main memory (primary storage device). As described above, operation programs including the program of the present invention are stored in the storage device 104. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited, and may be, for example, an internal type or an external type, and examples include an HD (hard disk), a CD-ROM, a CD-R, a CD-RW, an MO, a DVD, a flash memory, a memory card, and the like. The storage device 104 may be, for example, a hard disk drive (HDD) in which a recording medium and a drive are integrated, and a solid state drive (SSD).

[0028] In the present apparatus 10, the memory 102 and the storage device 104 can store various types of information such as log information, information obtained from an external database (not shown) or an external device, information generated by the present apparatus 10, and information used when the present apparatus 10 executes processing. Note that at least some of the information may be stored in an external server other than the memory 102 and the storage device 104, for example, or may be stored in a distributed manner using blockchain technology or the like in a plurality of terminals.

[0029] The present apparatus 10 may further include, for example, an input device 105, a display device 106, an imaging device 108, and an audio input / output device 109. The input device 105 is, for example, a touch panel, a keyboard, a mouse, or the like. The display device 106 includes, for example, an LED display, a liquid crystal display, or the like. The imaging device 108 includes, for example, a 2D camera, a 3D camera, an infrared camera, or the like. The audio input / output device 109 includes, for example, a microphone, a speaker, or the like.

[0030] Next, an example of the map information generation method of the present embodiment will be described based on the flowchart of FIG. 3. The map information generation method of the present embodiment is implemented as follows, for example, using the map information generation apparatus 10 of FIG. 1. Note that the map information generation method of the present embodiment is not limited to the use of the map information generation apparatus 10 of FIG. 1. In FIG. 3, the steps indicated in parentheses are optional steps and do not have to be processed.

[0031] First, the image acquisition unit 11 acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area (S11). The specific area is not particularly limited if it is indoors. Examples of indoor areas include exhibition halls, event venues, department stores, schools such as universities, etc. The three-dimensional image information may be, for example, an image captured by a three-dimensional camera, specifically, information including a full-surround panoramic image that three-dimensionally images the inside of the specific area. The three-dimensional camera means a camera that can capture not only vertical and horizontal information but also depth (depth), and specifically, for example, a three-dimensional camera such as matterport can be mentioned. The three-dimensional camera is, for example, an external device. On the other hand, the two-dimensional image information is information including an image captured by a two-dimensional camera. The two-dimensional camera means a camera that can capture vertical and horizontal information. Each of the images may be, for example, a still image or a moving image. Also, each image in the three-dimensional image information and the two-dimensional image information may be, for example, a plurality of images, and by connecting the plurality of images, it may be an image that can become a full-surround panoramic image. The image acquisition unit 11 may generate the full-surround panoramic image by acquiring, for example, a plurality of the images. On the one hand, from the plurality of images, the acquisition is executed, for example, via the communication network.

[0032] Next, the three-dimensional model generation unit 12 generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information (S12). The first and second three-dimensional models are models showing the configuration within the specific area and include point cloud data having shape information. The shape information is, for example, information showing the three-dimensional shape of the surface within the specific area. The first and second three-dimensional models have, for example, coordinate information based on the point cloud data. The coordinate information may be, for example, two-dimensional coordinate information (x, y) composed of values of horizontal (x) and depth (y), or may further be three-dimensional coordinate information (x, y, z) including a value of height (z). The configuration within the specific area includes, for example, not only the internal structure of the specific area but also the position and shape of objects arranged in the specific area.

[0033] Next, an example of generating the first three-dimensional model from the three-dimensional image information will be specifically described, but it is not limited thereto, and known techniques can be used. First, the three-dimensional model generation unit 12 extracts point cloud data from the three-dimensional image information and performs modeling on the point cloud data using a polygon mesh. The extraction of the point cloud data is not particularly limited, and the location that can be the shape information may be extracted as the point cloud data. Then, the three-dimensional model generation unit 12 arranges an image corresponding to the point cloud data on the surface of the polygon mesh.

[0034] The three-dimensional model generation unit 12 may generate the second three-dimensional model from the two-dimensional image information by, for example, SfM (Structure from Motion) processing. Note that it is not limited thereto, and the three-dimensional model generation unit 12 may generate the second three-dimensional model from the two-dimensional image information using a known technique other than SfM processing.

[0035] Then, the map information generation unit 13 associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model, generates map information of the specific area (S13), and ends (END). The association can also be referred to as, for example, alignment. That is, the map information generation unit 13 makes the coordinates of the first three-dimensional model and the second three-dimensional model coincide. The method of the association is not particularly limited. The map information is used, for example, to specify the position of a terminal that has captured an image (two-dimensional image).

[0036] As described above, the device 10 may further include a feature point extraction unit 14. For example, before the step (S13), the feature point extraction unit 14 extracts feature points from the two-dimensional image information by machine learning (S14). The machine learning is, for example, unsupervised learning and machine learning using a neural network as a model. The model extracts the feature points with an image as an input. The feature points indicate characteristic locations on the image and can be used to distinguish the image from other images. The step (S13) may be performed before the step (S13), for example, before the step (S12). In the step (S12), for example, the three-dimensional model generation unit 12 may associate the position of the feature points with the second three-dimensional model.

[0037] When the feature points are extracted in the step (S14), the map information generation unit 13 may generate the map information associated with the feature points in the step (S13), for example. The feature points and the map information may be associated, for example, by associating the second three-dimensional model with the feature points.

[0038] There is a problem that the accuracy of the position information of the three-dimensional model generated based on the SfM process is low. In contrast, according to the present embodiment, more accurate map information can be generated by associating each point cloud data using the first three-dimensional model and the second three-dimensional model. That is, the present embodiment improves the accuracy of the position information by using the first three-dimensional model generated using the images captured three-dimensionally within the specific area, and enables the identification of the position of the terminal by using the second three-dimensional model generated using the images captured two-dimensionally within the specific area. In addition, by extracting the feature points, map information that contributes to more accurate identification of the position of the terminal can be generated.

[0039] [Embodiment 2] FIG. 4 is a block diagram showing the configuration of an example of the position identification device 20 of the present embodiment. As shown in FIG. 4, the device 20 includes a map information generation unit 21, a user terminal information acquisition unit 22, and an identification unit 23. The device 20 may further include, for example, as an arbitrary configuration, a virtual image generation unit 24, a map information management unit 25, a composite image information output unit 26, and the like.

[0040] The device 20 may be, for example, one device including the above-described units, or may be a device in which the above-described units can be connected via a communication line network. Further, the device 20 can be connected to the above-described external device via the communication line network. The communication line network is not particularly limited and is, for example, the same as described above. Further, the device 20 may be, for example, a display device with a reflector in which the program of the present invention is installed. Furthermore, the device 20 may be in a form such as cloud computing or edge computing, for example, in which at least one of the above-described units is on a server and the other above-described units are on a terminal. The device 20 can communicate with the map information generation device 10 described in Embodiment 1 via the communication line network, for example.

[0041] FIG. 5 illustrates a block diagram of the hardware configuration of the device 20. The device 20 includes, for example, a central processing unit (CPU, GPC, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, an imaging device 108, an audio input / output device 109, and the like. Each unit of the device 20 is interconnected via the bus 103 by respective interfaces (I / F). Unless otherwise specified, the description of each unit can refer to the description of the hardware configuration of the map information generation device 10 shown in FIG. 2.

[0042] The central processing unit 101 functions as, for example, a map information generation unit 21, a user terminal information acquisition unit 22, an identification unit 23, a virtual image generation unit 24, a map information management unit 25, a composite image information output unit 26, and the like.

[0043] Next, an example of the position identification method of the present embodiment will be described based on the flowchart of FIG. 6. The position identification method of the present embodiment is implemented as follows, for example, using the position identification device 20 of FIG. 4. Note that the position identification method of the present embodiment is not limited to the use of the position identification device 20 of FIG. 4.

[0044] First, the map information generation unit 21 generates map information of a specific area (S21). Specifically, the map information generation unit 21 includes each part of the map information generation device 10 described in the first embodiment, and the foregoing description can be incorporated by reference.

[0045] Next, the user terminal information acquisition unit 22 acquires a captured image and sensor information from the user terminal (S22). The captured image is an image captured by the user terminal (also referred to as a captured image). The sensor information is information measured by various sensors (for example, an acceleration sensor, etc.) of the user terminal. The captured image and the sensor information are collectively referred to as user terminal information. The user terminal information acquisition unit 22 may acquire the sensor information, for example, in association with the captured image. Here, the user terminal is a terminal of a user in a specific area, and may be, for example, a personal computer (PC, for example, a desktop type, a notebook type), a smartphone, a tablet terminal, or the like. The user terminal is, for example, a device used in combination with the position identification device 10. The configuration of the user terminal is not particularly limited, and for example, it is sufficient that at least the image can be captured, the sensor information can be measured, and the image and the sensor information can be transmitted to the position identification device 20.

[0046] Then, the specific unit 23 identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information (S23), and ends (END).

[0047] When the map information generation device 10 in which the map information generation unit 21 includes the feature point extraction unit 14, the specifying unit 23 may, for example, specify the position of the user terminal by two-stage processing in the step (S23). Specifically, the specifying unit 23 may specify a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image, and specify the position of the user terminal from the candidate points. Further, the specifying unit 23 also specifies the imaging direction of the user terminal as described above. The candidate points are specified, for example, by the specifying unit 23 calculating the similarity between the feature points of the two-dimensional image information for each feature point and the feature points of the captured image. The specifying unit 23 specifies, for example, a point having the feature points of the two-dimensional image information with a high similarity as the candidate point.

[0048] According to the present embodiment, since more accurate map information is generated, the position of the user terminal can be specified more accurately. Here, the position of the user terminal is the position of the user terminal at the time when the captured image is taken, and corresponds to, for example, the position of the user. Further, the imaging direction of the user terminal is the imaging direction of the user terminal at the time when the captured image is taken, and corresponds to, for example, the front direction of the user. If the specific area is indoors, in the case of indoors, the position of the user terminal cannot be specified using a satellite positioning system such as GPS (Global Positioning System). However, according to the present embodiment, the position of the user terminal indoors can be specified by the map information.

[0049] (Modification Example 1) An example of displaying a virtual image on the user terminal will be described.

[0050] As described above, the position specifying device 20 of this modification example further includes a virtual image generation unit 24, a map information management unit 25, and a composite image information output unit 26.

[0051] Next, an example of the position identification method of this modified example will be described based on the flowchart of FIG. 7. The position identification method of this modified example is implemented as follows, for example, using the position identification device 20 of FIG. 4. Note that the position identification method of this modified example is not limited to the use of the position identification device 20 of FIG. 4.

[0052] First, the map information generation unit 21 generates map information of a specific area (S31). The step (S31) is the same as the step (S21) shown in FIG. 6, and the foregoing description can be incorporated by reference.

[0053] Next, the virtual image generation unit 24 generates a virtual image related to the specific area (S32). The virtual image is an image that is superimposed on the on-site captured image captured by the user terminal within the specific area. The virtual image generation unit 24 may generate, for example, a plurality of virtual images. Specifically, the virtual image includes, for example, virtual images for each object in the specific area. The object is not particularly limited and can be arbitrarily set, and may be an artificial object or a natural object. Examples of the artificial object include facilities, exhibition booths, tenants, classrooms, etc. Examples of the natural object include mountains, seas, rocks, etc. The virtual image for each object includes, for example, a position image of the object, and the position images are linked to each other for each object. The position image is, for example, an image related to the object. The generation of the virtual image may mean, for example, reading out the virtual image stored in the memory 102 or the storage device 104 in advance, or may mean acquiring the virtual image from an external database via the communication network. Further, the virtual image generation unit 24 may, for example, acquire information about the object (also referred to as object information) and generate the virtual image based on the object information.

[0054] In addition, the virtual image for each object generated by the virtual image generation unit 24 may further include a detailed information image of the object. In this case, the position image of the object and the detailed information image are linked to each other for each object. Then, the composite image information output unit 26 described later, for example, further provides a window portion in the captured image, and superimposes the detailed information image linked to the position image of the object existing in the captured image on the window portion. The window portion can be displayed, for example, on a screen capable of transitioning a plurality of the detailed information images. The detailed information image is, for example, an image having a one-layer or two-layer or more hierarchical structure. When the detailed information image is an image having a one-layer or two-layer or more hierarchical structure, the window portion can be displayed, for example, on a screen capable of transitioning the images for each layer of the detailed information image. Specifically, the detailed information image is, for example, an image based on the detailed information of the object, and one detailed information image may be linked to one position image of the object, or two detailed information images may be linked to one position image of the object.

[0055] Next, the map information management unit 25 manages by linking the map information, the virtual image, and the coordinate information of each virtual image (S33). The position information of the virtual image is, for example, information indicating the position where the virtual image is displayed. The position where the virtual image is displayed can be arbitrarily set by, for example, an administrator.

[0056] Next, the user terminal information acquisition unit 22 acquires a captured image and sensor information from the user terminal (S34). The step (S34) is the same as the step (S22) shown in FIG. 6, and the foregoing description can be incorporated by reference.

[0057] Next, based on the point cloud data of the map information and the captured image, the specifying unit 23 specifies the position of the user terminal, and based on the point cloud data of the map information and the sensor information, specifies the imaging direction of the user terminal (S35). The step (S35) is the same as the step (S23) shown in FIG. 6, and the foregoing description can be incorporated by reference.

[0058] Then, the composite image information output unit 26 outputs a composite image in which the virtual image is superimposed on the captured image (S36) and ends (END). Specifically, first, the composite image information output unit 26 refers to the map information based on the position and imaging direction of the user terminal, and determines whether there is a position corresponding to the coordinate information of the virtual image in the captured image. Then, when the position exists, the composite image information output unit 26 outputs a composite image in which the virtual image is superimposed on the position to the user terminal. The output may be executed, for example, via the communication network. The output of the composite image by the composite image information output unit 26 may be executed repeatedly, for example.

[0059] Hereinafter, an example of superimposing the virtual image on the captured image will be described.

[0060] FIG. 8 is a schematic diagram showing an example of the positional relationship between the user terminal 30 and the position information of the virtual image. As shown in FIG. 8, the composite image information output unit 26 determines, for example, based on the position and imaging direction X of the user terminal 30, that the coordinate information of the virtual images A to C exists in the imaging direction X.

[0061] FIG. 9 shows an example in which the composite image output from the position specifying device 20 to the user terminal 30 is displayed on the display of the user terminal 30. As shown in FIG. 9, the composite image information output unit 26 generates, for example, a composite image in which the virtual images A to C (for example, the position images; hereinafter also referred to as pins) are superimposed on the positions in the captured image corresponding to the objects A to C existing in the imaging direction X. The pins A to C may be superimposed with the pins farther from the user terminal 30 being smaller than the other pins, for example, based on the distance between the position of the user terminal 30 and the coordinate information of the pins A to C. On the other hand, the pins A to C may be superimposed with the pins closer to the user terminal 30 being larger than the other pins, for example, based on the distance between the position of the user terminal 30 and the coordinate information of the pins A to C. Also, the pins may be colored at the outer edges, for example, according to the distance between the position of the user terminal 30 and the coordinate information of the pins A to C.

[0062] Furthermore, as shown in FIG. 9, the detailed information image is specifically displayed in the window part (Window) 1. In the figure, the window part 1 is shown at the lower part of the captured image, which is an example and not limited thereto. For example, when the present invention is applied to an exhibition, a detailed information image generated based on detailed information such as displayed products and services and the company of its booth is displayed in the window part 1.

[0063] According to this modification example, since the position of the user terminal can be specified more accurately, the virtual image can be displayed at a more accurate position by using AR (Augmented Reality) technology. Note that this modification example is an example, and the map information generation device 10 and the map information generation method of the first embodiment and the position specification device 20 and the position specification method described in the second embodiment are not limited to the application to AR technology.

[0064] [Embodiment 3] The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, "procedure" may be read as "processing". Also, the program of this embodiment may be recorded, for example, on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.

[0065] The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0066] [Supplementary Note] Some or all of the above embodiments may be described as follows in the supplementary note, but are not limited thereto. [Supplementary Note 1] It includes an image acquisition unit, a three-dimensional model generation unit, and a map information generation unit. The image acquisition unit acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area. The three-dimensional model generation unit generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information. The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area and include point cloud data having shape information. The map information generation unit is a map information generation device that associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model to generate map information of the specific area. (Appendix 2) The three-dimensional image information includes a full-surround panorama image obtained by three-dimensionally imaging the inside of the specific area. The map information generation device according to Appendix 1. (Appendix 3) Furthermore, it includes a feature point extraction unit. The feature point extraction unit extracts feature points from the two-dimensional image information by machine learning. The map information generation unit generates the map information associated with the feature points. The map information generation device according to Appendix 1 or 2. (Appendix 4) It includes a map information generation unit, a user terminal information acquisition unit, and a specifying unit. The map information generation unit is the map information generation device according to any one of Appendices 1 to 3. The user terminal information acquisition unit acquires a captured image and sensor information from a user terminal. The specifying unit specifies the position of the user terminal based on the point cloud data of the map information and the captured image, and specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information. A position specifying device. (Appendix 5) The map information generation unit is the map information generation device according to Appendix 3. The feature point extraction unit extracts feature points from the captured image by machine learning. The specific part identifies a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image. Identify the position of the user terminal from the candidate points, and The position identification device according to Supplementary Note 4, which identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information. (Supplementary Note 6) Including an image acquisition step, a three-dimensional model generation step, and a map information generation step. The image acquisition step acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area. The three-dimensional model generation step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information. The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area and include point cloud data having shape information. The map information generation method in which the map information generation step associates the positions of the point cloud data in the first three-dimensional model with the point cloud data in the second three-dimensional model and generates map information of the specific area. (Supplementary Note 7) The three-dimensional image information includes a full-surround panoramic image obtained by three-dimensionally imaging the inside of the specific area. The map information generation method according to Supplementary Note 6. (Supplementary Note 8) Furthermore, it includes a feature point extraction step. The feature point extraction step extracts feature points from the two-dimensional image information by machine learning. The map information generation method according to Supplementary Note 6 or 7, in which the map information generation step generates the map information associated with the feature points. (Supplementary Note 9) Including a map information generation step, a user terminal information acquisition step, and a specific step. The map information generation step is a step of executing each step of the map information generation method according to any one of Supplementary Notes 6 to 8. The user terminal information acquisition step acquires a captured image and sensor information from the user terminal. The specific process is a position determination method that determines the position of the user terminal based on the point cloud data of the map information and the captured image, and determines the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information. (Appendix 10) The map information generation process is a process of executing each process of the map information generation method described in Appendix 8. The feature point extraction process extracts feature points from the captured image by machine learning. The specific process determines a plurality of candidate locations as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image. Determine the position of the user terminal from the candidate locations, and The position determination method described in Appendix 9 that determines the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information. (Appendix 11) A program for causing a computer to execute a procedure including an image acquisition procedure, a three-dimensional model generation procedure, and a map information generation procedure. The image acquisition procedure acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area. The three-dimensional model generation procedure generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information. The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area, and include point cloud data having shape information. The map information generation procedure associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model, and generates map information of the specific area. (Appendix 12) The three-dimensional image information includes a full-surround panoramic image obtained by three-dimensionally imaging the inside of the specific area. The program described in Appendix 11. (Appendix 13) Furthermore, it includes a feature point extraction procedure. The feature point extraction procedure extracts feature points from the two-dimensional image information by machine learning. The map information generation procedure is the program according to Appendix 11 or 12 that generates the map information associated with the feature points. (Appendix 14) A program for causing a computer to execute a procedure including a map information generation procedure, a user terminal information acquisition procedure, and a specification procedure; The map information generation procedure is a procedure for executing each procedure of the program according to any one of Appendices 11 to 13, The user terminal information acquisition procedure acquires a captured image and sensor information from a user terminal, The specification procedure specifies the position of the user terminal based on the point cloud data of the map information and the captured image, and specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information. (Appendix 15) The map information generation procedure is a procedure for executing each procedure of the program according to Appendix 13, The feature point extraction procedure extracts feature points from the captured image by machine learning, The specification procedure specifies a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image, specifies the position of the user terminal from the candidate points, and specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information, the program according to Appendix 14. (Appendix 16) A computer-readable recording medium recording the program according to any one of Appendices 11 to 15.

Industrial Applicability

[0067] According to the present invention, more accurate map information can be generated. Therefore, the present invention is particularly useful in fields using AR technology such as exhibitions and games.

Explanation of Signs

[0068] 10 Map information generation device 11 Image acquisition unit 12 Three-dimensional model generation unit 13 Map information generation unit 14 Feature point extraction unit 20 Location identification device 21 Map information generation unit 22 User terminal information acquisition unit 23 Identification unit 24 Virtual image generation unit 25 Map information management unit 26 Composite image information output unit 30 User terminal 101 Central processing unit 102 Memory 103 Bus 104 Storage device 105 Input device 106 Display device 107 Communication device 108 Imaging device 109 Audio input / output device

Claims

Claim 1: A position determination device including a map information generation unit, a virtual image generation unit, a map information management unit, a user terminal information acquisition unit, a specifying unit, and a composite image information output unit, wherein the map information generation unit includes an image acquisition unit, a three-dimensional model generation unit, and an area map information generation unit, the image acquisition unit acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area, the three-dimensional model generation unit generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information, the first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area and include point cloud data having shape information, the area map information generation unit associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model, and generates map information of the specific area, the virtual image generation unit generates a virtual image related to the specific area, the virtual image is an image superimposed on a local captured image captured by a user terminal within the specific area, the map information management unit manages by associating the map information, the virtual image, and each coordinate information of the virtual image, the user terminal information acquisition unit acquires a captured image and sensor information from the user terminal, the specifying unit specifies the position of the user terminal based on the point cloud data of the map information and the captured image, and specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information, the composite image information output unit outputs a composite image in which the virtual image is superimposed on the captured image.

2. The position determination device according to Claim 1, wherein the three-dimensional image information includes a full-surround panorama image obtained by three-dimensionally imaging the inside of the specific area.

3. Furthermore, it includes a feature point extraction unit, the feature point extraction unit extracts feature points from the two-dimensional image information by machine learning, and the map information generation unit generates the map information associated with the feature points. The position determination device according to Claim 1 or 2.

4. The feature point extraction unit extracts feature points from the captured image by machine learning, the specifying unit specifies a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image, specifies the position of the user terminal from the candidate points, and The position specifying device according to claim 3, which specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

5. Including a map information generation step, a virtual image generation step, a map information management step, a user terminal information acquisition step, a specification step, and a composite image information output step, The map information generation step includes an image acquisition step, a three-dimensional model generation step, and an area map information generation step, The image acquisition step acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area, The three-dimensional model generation step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information, The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area and include point cloud data having shape information, The area map information generation step associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model to generate map information of the specific area, The virtual image generation step generates a virtual image related to the specific area, The virtual image is an image superimposed on a local captured image captured by a user terminal within the specific area, The map information management step manages by associating the map information, the virtual image, and the coordinate information of each virtual image, The user terminal information acquisition step acquires a captured image and sensor information from the user terminal, The specification step specifies the position of the user terminal based on the point cloud data of the map information and the captured image, and specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information, The composite image information output step outputs a composite image obtained by superimposing the virtual image on the captured image, Position information specifying method.

6. Further including a feature point extraction step, The feature point extraction step extracts feature points from the two-dimensional image information by machine learning, The map information generation step generates the map information associated with the feature points. The position information specifying method according to claim 5.

7. A program for causing a computer to execute a procedure including a map information generation procedure, a virtual image generation procedure, a map information management procedure, a user terminal information acquisition procedure, a specification procedure, and a composite image information output procedure; The map information generation procedure includes an image acquisition procedure, a three-dimensional model generation procedure, and an area map information generation procedure, The image acquisition procedure acquires three-dimensional image information and two-dimensional image information obtained by imaging a specific area, The three-dimensional model generation procedure generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information, The first three-dimensional model and the second three-dimensional model are models showing the configuration within the specific area and include point cloud data having shape information, The area map information generation procedure associates the positions of the point cloud data in the first three-dimensional model and the point cloud data in the second three-dimensional model to generate map information of the specific area, The virtual image generation procedure generates a virtual image related to the specific area, The virtual image is an image superimposed on a local captured image captured by a user terminal within the specific area, The map information management procedure manages by associating the map information, the virtual image, and each coordinate information of the virtual image, The user terminal information acquisition procedure acquires a captured image and sensor information from the user terminal, The specific procedure specifies the position of the user terminal based on the point cloud data of the map information and the captured image, and specifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information, The composite image information output procedure outputs a composite image in which the virtual image is superimposed on the captured image.

Images