Wireless communication terminal, information processing method, and program

Abstract

To provide a wireless communication terminal, an information processing method, and a program capable of identifying a three-dimensional position.SOLUTION: A wireless communication terminal 100 includes a camera, a position sensor, and an angle sensor, and includes: a map data storage unit that stores three-dimensional map data including structural information indicating the position and shape of a structure; a terminal information acquisition unit that acquires captured images taken by the camera and position and angle information on the wireless communication terminal at the time when the images were taken; a structure identification unit that identifies the structure included in the captured images; a structural information acquisition unit that acquires structural information on the structure identified by the structure identification unit from the three-dimensional map data; and a three-dimensional position identification unit that identifies the three-dimensional position of the wireless communication terminal based on the captured images, position information, and angle information acquired by the terminal information acquisition unit, and the structural information acquired by the structural information acquisition unit.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a wireless communication terminal, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes "an optical axis alignment mechanism and an optical axis alignment method that can achieve accurate optical axis alignment with a simple configuration, enable pan-tilt operation in a small space, and realize miniaturization of the device in a space optical communication device." Patent Document 2 describes "an optical communication tracking device capable of performing optical axis alignment with an optical communication tracking device having the same configuration provided in a counterpart optical communication device." Patent Document 3 describes "an optical wireless system that performs wireless transmission using light, an imaging device that performs data transmission using the optical wireless system, and a control device." [Prior Art Documents] [Patent Documents] [Patent Document 1] International Publication No. 2021 / 005684 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 2022-103994 [Patent Document 3] International Publication No. 2013 / 084330

Summary of the Invention

Means for Solving the Problems

[0003] According to one embodiment of the present invention, a wireless communication terminal is provided. The wireless communication terminal may have a camera, a position sensor, and an angle sensor. The wireless communication terminal may include a map data storage unit that stores three-dimensional map data including structure information indicating the position and shape of a structure. The wireless communication terminal may include a terminal information acquisition unit that acquires an image captured by the camera, and the position information and angle information of the wireless communication terminal at the time the image was captured. The wireless communication terminal may include a structure identification unit that identifies a structure included in the image. The wireless communication terminal may include a structure information acquisition unit that acquires the structure information of the structure identified by the structure identification unit from the three-dimensional map data. The wireless communication terminal may include a three-dimensional position identification unit that identifies the three-dimensional position of the wireless communication terminal based on the image, position information, and angle information acquired by the terminal information acquisition unit, and the structure information acquired by the structure information acquisition unit.

[0004] In the wireless communication terminal, the 3D positioning unit may determine the relative direction between the wireless communication terminal and the structure from the shape of the structure included in the captured image and the shape of the structure indicated by the structure information, and determine the 3D position of the wireless communication terminal using the determined relative direction, the position of the wireless communication terminal indicated by the position information, the angle of the wireless communication terminal indicated by the angle information, and the position of the structure indicated by the structure information. The 3D positioning unit may determine the relative direction between the wireless communication terminal and the structure using the shape of the structure included in the captured image, the shape of the structure indicated by the structure information, the time when the wireless communication terminal took the captured image, and the shadow of the structure included in the captured image. The 3D positioning unit may further determine the relative direction between the wireless communication terminal and the structure using weather information of the area where the wireless communication terminal and the structure are located at the time the wireless communication terminal took the captured image.

[0005] In any of the above wireless communication terminals, the terminal information acquisition unit may further acquire altitude information of the wireless communication terminal at the time the captured image was taken, which is measured by an altitude sensor of the wireless communication terminal, and the three-dimensional position identification unit may further identify the three-dimensional position of the wireless communication terminal based on the altitude information.

[0006] In any of the above wireless communication terminals, the structure identification unit may identify a plurality of structures included in the captured image, the structure information acquisition unit may acquire the structure information of each of the plurality of structures identified by the structure identification unit from the 3D map data, and the 3D position identification unit may determine the 3D position of the wireless communication terminal based on the captured image, position information, and angle information acquired by the terminal information acquisition unit, and the plurality of structure information acquired by the structure information acquisition unit.

[0007] In any of the aforementioned wireless communication terminals, the terminal information acquisition unit may acquire a first captured image taken by the camera of the wireless communication terminal in a first direction, first position information and first angle information of the wireless communication terminal at the time the first captured image was taken, a second captured image taken by the camera of the wireless communication terminal in a second direction different from the first direction, and second position information and second angle information of the wireless communication terminal at the time the second captured image was taken, and the structure identification unit may acquire a first structure included in the first captured image and a second structure included in the second captured image. The structure may be identified, and the structure information acquisition unit may acquire first structure information and second structure information of the first structure and second structure identified by the structure identification unit from the 3D map data, and the 3D position identification unit may identify the 3D position of the wireless communication terminal based on the first captured image, first position information, and first angle information acquired by the terminal information acquisition unit, the second captured image, second position information, and second angle information, and the first structure information and second structure information acquired by the structure information acquisition unit.

[0008] Any of the above wireless communication terminals may include a target 3D position information acquisition unit that acquires target 3D position information indicating the 3D position of an object, and a display control unit that displays guide data on the display of the wireless communication terminal to adapt the position and angle of the wireless communication terminal to the 3D position indicated by the target 3D position information, based on the 3D position of the wireless communication terminal identified by the 3D position identification unit and the target 3D position information. The display control unit may display a marker at the position of the 3D position indicated by the target 3D position information, based on the position and angle of the wireless communication terminal, and may change the display mode of the marker depending on whether the position and angle of the wireless communication terminal are adapted to the 3D position indicated by the target 3D position information or not. The display control unit may change the color of the marker depending on whether the position and angle of the wireless communication terminal are adapted to the 3D position indicated by the target 3D position information or not.

[0009] According to one embodiment of the present invention, an information processing method is provided that is performed by a wireless communication terminal. The wireless communication terminal may have a camera, a position sensor, and an angle sensor. The information processing method may include a terminal information acquisition step of acquiring an image captured by the camera, and position information and angle information of the wireless communication terminal at the time the image was captured. The information processing method may include a structure identification step of identifying a structure included in the image. The information processing method may include a structure information acquisition step of acquiring structure information of the structure identified in the structure identification step from three-dimensional map data including structure information indicating the position and shape of the structure. The information processing method may include a three-dimensional position identification step of identifying the three-dimensional position of the wireless communication terminal based on the image, position information, and angle information acquired in the terminal information acquisition step, and the structure information acquired in the structure information acquisition step.

[0010] According to one embodiment of the present invention, a program is provided for a wireless communication terminal having a camera, a position sensor, and an angle sensor to perform the following steps: a terminal information acquisition step of acquiring an image captured by the camera, and position information and angle information of the wireless communication terminal at the time the image was captured; a structure identification step of identifying a structure included in the image; a structure information acquisition step of acquiring the structure information of the structure identified in the structure identification step from 3D map data including structure information indicating the position and shape of the structure; and a 3D position identification step of identifying the 3D position of the wireless communication terminal based on the image, position information, and angle information acquired in the terminal information acquisition step and the structure information acquired in the structure information acquisition step.

[0011] It should be noted that the above summary of the invention does not list all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention. [Brief explanation of the drawing]

[0012] [Figure 1] A schematic example of a wireless communication terminal 100 is shown below. [Figure 2] This is an explanatory diagram for describing the wireless communication terminal 100 used during filming. [Figure 3] This is an explanatory diagram for describing the wireless communication terminal 100 used during filming. [Figure 4] A schematic example of 300 captured images is shown below. [Figure 5] This is an explanatory diagram illustrating the processing details of the wireless communication terminal 100. [Figure 6] This is an explanatory diagram illustrating a specific example of the processing content of the wireless communication terminal 100. [Figure 7] A schematic example of the display of guide data by the wireless communication terminal 100 is shown. [Figure 8] An example of the functional configuration of the wireless communication terminal 100 is shown in a schematic manner. [Figure 9]A schematic example of the hardware configuration of a computer 1200 that functions as a wireless communication terminal 100 is shown. [Modes for carrying out the invention]

[0013] The present invention will be described below through embodiments of the invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0014] Optical wireless communication, as a backhaul wireless communication method, currently enables high-capacity communication of up to 25 Gbps, for example. Since it requires no licenses, no massive optical cable laying costs, and no construction work, it is considered ideal for event response and ensuring communication during disasters. Its use is also anticipated in inter-island and remote areas. In particular, during disasters, the ability to remotely check high-precision video signals for on-site conditions and diagnostics, in addition to general communication networks, provides valuable information for decision-making and is crucial for rapid response. However, optical wireless communication, with its extremely high directivity, requires very precise optical axis adjustment during installation. Currently, rough directional adjustments are performed manually using mirrors or on-board telescopes, resulting in very poor work efficiency and requiring a certain level of experience. This becomes more pronounced as the distance between two points increases. In the wireless communication terminal 100 according to this embodiment, for example, visualization through guide displays simplifies and shortens the work, thereby improving work efficiency. Furthermore, in the wireless communication terminal 100, it is expected that an optimal communication environment can be automatically constructed at all times by using AI (Artificial Intelligence) for operation and maintenance management of 3D management, terrain data, and past disaster data. As a specific example, in the wireless communication terminal 100 according to this embodiment, optical axis adjustment can be performed while visually and numerically confirming the precise position information based on 3D position (latitude / longitude / height) and guide displays indicating the installation position and direction of the optical wireless equipment, thereby improving work efficiency by performing the work based on visualized information, which was previously done intuitively.

[0015] Figure 1 schematically shows an example of a wireless communication terminal 100. The wireless communication terminal 100 may include a display 101, a camera 120, a position sensor 130, and an angle sensor 140.

[0016] The position sensor 130 measures the position of the wireless communication terminal 100. The position sensor 130 may measure the position of the wireless communication terminal 100 by GNSS (Global Navigation Satellite System) positioning such as GPS (Global Positioning System). The position sensor 130 may also use other positioning methods such as Wi-Fi (Wireless Fidelity) positioning or cell positioning, or a combination of these. The position sensor 130 may measure the planar position of the wireless communication terminal 100. Planar position is a two-dimensional position. The position sensor 130 may also measure the three-dimensional position of the wireless communication terminal 100. Three-dimensional position is a three-dimensional position.

[0017] The angle sensor 140 measures the angle of the wireless communication terminal 100. The angle sensor 140 may be, for example, a gyro sensor.

[0018] The wireless communication terminal 100 according to this embodiment identifies the three-dimensional position of the wireless communication terminal 100 by using a captured image captured by the camera 120, position information from the position sensor 130, angle information from the angle sensor 140, and three-dimensional map data (which may be referred to as 3D map data) including structure information indicating the position and shape of the structure. The wireless communication terminal 100 may identify the structure included in the captured image, obtain the structure information of the structure from the 3D map data, and use the captured image, position information, angle information, and structure information to identify the three-dimensional position of the wireless communication terminal 100. As an example, the wireless communication terminal 100 identifies the relative direction between the wireless communication terminal 100 and the structure from the shape of the structure included in the captured image and the shape of the structure indicated by the structure information, and uses the identified relative direction, the position of the wireless communication terminal 100 indicated by the position information, and the position of the structure indicated by the structure information to identify the three-dimensional position of the wireless communication terminal 100. The three-dimensional position identified by the wireless communication terminal 100 in this way can be expected to have higher position accuracy than the three-dimensional position measured by the position sensor 130.

[0019] The wireless communication terminal 100 can be used for various applications. The wireless communication terminal 100 is used, for example, for optical axis adjustment in optical wireless communication. In the example shown in FIG. 1, at the location where the first optical wireless communication device 200 is installed, the first wireless communication terminal 100 identifies its own three-dimensional position, and at the location where the second optical wireless communication device 200 is installed, the second wireless communication terminal 100 identifies its own three-dimensional position. Then, for example, the second wireless communication terminal 100 transmits its own three-dimensional position to the first wireless communication terminal 100. The first wireless communication terminal 100 uses its own three-dimensional position and the three-dimensional position of the second wireless communication terminal 100 to execute a guide for aligning the optical axes of the first optical wireless communication device 200 and the second optical wireless communication device 200. By the first wireless communication terminal 100 and the second wireless communication terminal 100 respectively identifying their accurate three-dimensional positions, a guide for aligning the optical axes of the first optical wireless communication device 200 and the second optical wireless communication device 200 can be accurately executed.

[0020] FIG. 2 and FIG. 3 are explanatory diagrams for explaining the wireless communication terminal 100 at the time of shooting. In the present embodiment, for example, a user of the wireless communication terminal 100 performs shooting such that a characteristic structure is located at the center. The wireless communication terminal 100 acquires a captured image including the structure, and position information and angle information of the wireless communication terminal 100 when the captured image is captured, and stores them in association with each other.

[0021] FIG. 4 schematically shows an example of the captured image 300. FIG. 4 shows an example of the captured image 300 when the structure 310 is the target structure. The wireless communication terminal 100 may extract the structure 310 located at the center of the captured image 300 as the target structure by analyzing the captured image 300.

[0022] The wireless communication terminal 100 may extract the structure 310 according to the 3D map data and the shooting conditions of the camera 120. For example, the wireless communication terminal 100 specifies the visible distance at the time of shooting from the lens telephoto magnification of the camera 120. As an example, when the lens telephoto magnification is 1 time, the visible distance is set to about 3.5 km. Then, the wireless communication terminal 100 searches within a radius of 3.5 km centered on the wireless communication terminal 100 from the position information of the wireless communication terminal 100 when the captured image 300 is captured, which is measured by the position sensor 130, and extracts the target structure 310 by an image processing technique. The wireless communication terminal 100 may identify the structure 310 by matching the characteristic points of the shape using the shape included in the structure information included in the 3D map data.

[0023] The wireless communication terminal 100 may store in association with each other the position information of the wireless communication terminal 100 when the captured image 300 is captured, which is measured by the position sensor 130, the angle information of the wireless communication terminal 100 when the captured image 300 is captured, which is measured by the angle sensor 140, and the captured image 300.

[0024] Figure 5 is an explanatory diagram illustrating the processing of the wireless communication terminal 100. The wireless communication terminal 100 may determine its three-dimensional position based on the position indicated by the position information of the wireless communication terminal 100 when the captured image 300 was taken, the altitude of the wireless communication terminal 100, and the precise position and shape (altitude) of the structure 310 indicated by the structure information of the structure 310 included in the captured image 300 obtained from 3D map data. The altitude of the wireless communication terminal 100 may be the altitude included in the position information of the wireless communication terminal 100, or it may be the altitude measured by the altitude sensor of the wireless communication terminal 100 when the captured image 300 was taken. In either case, it is often not the precise altitude.

[0025] The wireless communication terminal 100 may determine the relative direction between the wireless communication terminal 100 and the structure 310 based on the shape of the structure 310 included in the captured image 300 and the shape of the structure indicated by the structure information. The position and shape of the structure 310 indicated by the structure information included in the 3D map data are highly accurate, and the relative direction determined based on these may be relatively accurate. The wireless communication terminal 100 may also determine its 3D position by prioritizing the determined relative direction and correcting the position indicated by the position information of the wireless communication terminal 100 and the altitude of the wireless communication terminal 100.

[0026] The wireless communication terminal 100 may determine the relative direction between the wireless communication terminal 100 and the structure 310 by using the shape of the structure 310 included in the captured image 300, the shape of the structure indicated by the structure information, the time when the wireless communication terminal 100 took the captured image 300, and the shadow of the structure 310 included in the captured image 300. The wireless communication terminal 100 can accurately grasp the position and shape of the structure 310 using the structure information included in the 3D map data. The position of the sun can be estimated from the time when the wireless communication terminal 100 took the captured image 300, and the shadow of the structure 310 can be estimated from the estimated position of the sun. For example, the wireless communication terminal 100 tentatively determines the relative direction between the wireless communication terminal 100 and the structure 310 based on the shape of the structure 310 included in the captured image 300 and the shape of the structure indicated by the structure information. By comparing the estimated impression with the shadow of the structure 310 included in the captured image 300, the wireless communication terminal 100 corrects the tentatively determined relative direction as necessary to finalize the relative direction. For example, assuming the tentatively determined relative direction is correct, the wireless communication terminal 100 compares the estimated impression with the shadow of the structure 310 included in the captured image 300. If there is a difference, the wireless communication terminal 100 adjusts the tentatively determined relative direction to reduce the difference and finalize the relative direction.

[0027] Figure 6 is an explanatory diagram illustrating a specific example of the processing content of the wireless communication terminal 100. In this example, the wireless communication terminal 100 performs calculations using the captured image 402 including the structure 310, the planar information 404 measured by the position sensor 130 at the time the captured image 402 was taken, and the gyro information 406 measured by the angle sensor 140 at the time the captured image 402 was taken.

[0028] The wireless communication terminal 100 performs image recognition 408 on the captured image 402 and performs feature point extraction 410 and impression state extraction 412 on multiple structures contained in the captured image 402.

[0029] Furthermore, the wireless communication terminal 100 performs target area extraction 414 using the planar information 404. The wireless communication terminal 100 extracts the line-of-sight range centered on the planar position indicated by the planar information 404 as the target area. The wireless communication terminal 100 may further use the gyro information 406 to perform target area extraction 414. For example, the wireless communication terminal 100 extracts the line-of-sight range centered on the planar position indicated by the planar information 404 and in the direction indicated by the gyro information 406 as the target area.

[0030] The wireless communication terminal 100 performs object identification 418 using the results of feature point extraction 410, impression state extraction 412, and target area extraction 414, along with the 3D map data 416. The wireless communication terminal 100 identifies the portion of the 3D map data 416 corresponding to the target area extracted by the target area extraction 414, and uses the feature points and impression state to identify the target structure 310 (sometimes referred to as the target object) from the identified portion.

[0031] The wireless communication terminal 100 obtains structural information of the object identified by object identification 418 from 3D structural data 420 included in 3D map data 416, and obtains object position information 422 from the structural information. The wireless communication terminal 100 also identifies captured seal information 424 using the result of seal impression state extraction 412, the 3D structural data 420, and gyro information 406.

[0032] The wireless communication terminal 100 then uses the planar information 404, the gyro information 406, the object position information 422, and the captured image information 424 to determine the 3D position information 426.

[0033] The wireless communication terminal 100 may generate correction value calculation data 428 using the identified 3D position information 426, the planar information 404, and the gyro information 406. The correction value calculation data 428 may be data for correcting the 3D position of the wireless communication terminal 100, identified by the planar information 404 and the gyro information 406, to an accurate 3D position. By having the wireless communication terminal 100 perform the process shown in Figure 6 many times to identify many 3D position information 426, and generating correction value calculation data 428 using many combinations of the planar information 404 and the gyro information 406 and the 3D position information 426, the 3D position of the wireless communication terminal 100, identified by the planar information 404 and the gyro information 406, can be corrected to an accurate 3D position under any conditions.

[0034] The wireless communication terminal 100 may generate correction value calculation data 430 using the gyro information 406 and the impression state extraction 412. The correction value calculation data 430 may be data for correcting the angle information measured by the gyro information 406 to more accurate angle information.

[0035] Figure 7 schematically shows an example of guide data 150 from the wireless communication terminal 100. The guide data 150 may be data for fitting the position and angle of the wireless communication terminal 100 to the 3D position information of the target, which indicates the 3D position of the target. The wireless communication terminal 100 acquires the 3D position information of the target from an external source and displays the guide data 150 based on the 3D position information of the target and the 3D position information 426 that it has identified itself.

[0036] In the example shown in Figure 7, the guide data 150 includes a guide 152 and a marker 154. The guide 152 is a line that makes it easier to grasp the center point. The marker 154 indicates the 3D position indicated by the target 3D position information, based on the position and angle of the wireless communication terminal 100. The display position of the marker 154 changes as the position and angle of the wireless communication terminal 100 change. The wireless communication terminal 100 may display the marker 154 so that when the marker 154 coincides with the center of the guide 152, the position and angle of the wireless communication terminal 100 conform to the 3D position indicated by the target 3D position. This allows the user of the wireless communication terminal 100 to conform the position and angle of the wireless communication terminal 100 to the 3D position indicated by the target 3D position by adjusting the position and angle of the wireless communication terminal 100 so that the marker 154 is located at the center of the guide 152.

[0037] The wireless communication terminal 100 may change the display of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conform to the 3D position indicated by the target 3D position information or not. For example, the wireless communication terminal 100 may change the color of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conform to the 3D position indicated by the target 3D position information or not. As an example, the wireless communication terminal 100 may make the marker 154 red when the position and angle of the wireless communication terminal 100 does not conform to the 3D position indicated by the target 3D position information, and green when it does conform. This provides the user with a display that makes it easy to intuitively understand the conformance status.

[0038] Figure 8 schematically shows an example of the functional configuration of the wireless communication terminal 100. The wireless communication terminal 100 includes a storage unit 102, a map data acquisition unit 104, a terminal information acquisition unit 106, a structure identification unit 108, a structure information acquisition unit 110, a 3D position identification unit 112, a target 3D position information acquisition unit 114, a display control unit 116, and a weather information acquisition unit 118. However, it is not necessarily required that the wireless communication terminal 100 include all of these.

[0039] The map data acquisition unit 104 acquires 3D map data. The storage unit 102 stores the 3D map data acquired by the map data acquisition unit 104. The map data acquisition unit 104 may acquire 3D map data for the entire world. The map data acquisition unit 104 may acquire 3D map data for a target country. The map data acquisition unit 104 may acquire 3D map data for a target region. For example, if the target country is Japan, the target region may be at the municipal level, or it may be an arbitrarily designated area.

[0040] The 3D map data includes structural information that shows the location and shape of structures within the target area. Examples of structures include buildings and other structures, but are not limited to these; it includes any object with a structure.

[0041] The terminal information acquisition unit 106 acquires the captured image taken by the camera 120. The terminal information acquisition unit 106 acquires the location information measured by the position sensor 130. The terminal information acquisition unit 106 acquires the angle information measured by the angle sensor 140. When the camera 120 takes a picture, the terminal information acquisition unit 106 may acquire the captured image, as well as the location information and angle information at the time the image was taken.

[0042] The terminal information acquisition unit 106 may acquire altitude information measured by the altitude sensor if the wireless communication terminal 100 has an altitude sensor. The terminal information acquisition unit 106 may acquire the captured image, the position information, angle information, and altitude information at the time the image was taken, when the camera 120 takes a picture.

[0043] The structure identification unit 108 identifies structures included in the captured image acquired by the terminal information acquisition unit 106. For example, the structure identification unit 108 identifies a structure located in the center of the captured image. The structure identification unit 108 may also identify structures specified by the user of the wireless communication terminal 100 from the captured image. The structure identification unit 108 may identify multiple structures from a single captured image.

[0044] The structure information acquisition unit 110 acquires structure information of structures identified by the structure identification unit 108 from 3D map data. The structure information acquisition unit 110 may, for example, identify a shooting range from the position information and angle information acquired by the terminal information acquisition unit 106, identify structures whose characteristics match those of structures identified by the structure identification unit 108 from the portion of the 3D map data corresponding to that shooting range, and acquire structure information corresponding to the identified structures.

[0045] The structural information acquisition unit 110 may also accept designations from the user of the wireless communication terminal 100. For example, the structural information acquisition unit 110 displays 3D map data on the display 101 and accepts designations of structures from the user who performed the photography using the wireless communication terminal 100.

[0046] The 3D positioning unit 112 determines the 3D position of the wireless communication terminal 100 based on the captured image, position information, and angle information acquired by the terminal information acquisition unit 106, and the structural information acquired by the structural information acquisition unit 110. The 3D positioning unit 112 may use the planar position and altitude included in the position information. If the terminal information acquisition unit 106 has acquired altitude information measured by the altitude sensor, the 3D positioning unit 112 may use the planar position information included in the position information and the altitude information measured by the altitude sensor.

[0047] For example, the 3D positioning unit 112 determines the relative direction between the wireless communication terminal 100 and the structure based on the shape of the structure included in the captured image and the shape of the structure indicated by the structure information. Then, the 3D positioning unit 112 determines the 3D position of the wireless communication terminal 100 using the determined relative direction, the position of the wireless communication terminal 100 indicated by the position information, the altitude indicated by the position information or altitude information, the angle of the wireless communication terminal 100 indicated by the angle information, and the position of the structure indicated by the structure information.

[0048] The 3D positioning unit 112 may estimate the distance between the wireless communication terminal 100 and the structure from the structure included in the captured image and the structure information. For example, the 3D positioning unit 112 estimates the distance between the wireless communication terminal 100 and the structure from the magnification of the camera 120, the size of the structure in the captured image, and the actual size of the structure indicated by the structure information. The 3D positioning unit 112 may determine the 3D position of the wireless communication terminal 100 using the determined relative direction, the estimated distance, the position of the wireless communication terminal 100 indicated by the position information, the altitude indicated by the position information or altitude information, and the position of the structure indicated by the structure information.

[0049] The 3D positioning unit 112 may determine the relative direction between the wireless communication terminal 100 and the structure using the shape of the structure included in the captured image, the shape of the structure indicated by the structure information, the time when the wireless communication terminal 100 took the captured image, and the shadow of the structure included in the captured image. The position of the sun can be estimated from the time when the wireless communication terminal 100 took the captured image 300, and the shadow of the structure can be estimated from the estimated position of the sun. For example, the 3D positioning unit 112 provisionally determines the relative direction between the wireless communication terminal 100 and the structure from the shape of the structure included in the captured image and the shape of the structure indicated by the structure information, and then finalizes the relative direction by correcting the provisionally determined relative direction as necessary by comparing the estimated shadow with the shadow of the structure included in the captured image. For example, the 3D positioning unit 112 assumes that the provisionally determined relative direction is correct, compares the estimated shadow with the shadow of the structure included in the captured image, and if there is no difference, the provisionally determined relative direction is set as the final determined relative direction. For example, the 3D positioning unit 112 assumes that the provisionally determined relative direction is correct, compares the estimated shadow with the shadow of the structure included in the captured image, and if there is a difference, adjusts the provisionally determined relative direction to reduce the difference and then makes the final determination.

[0050] While the position and shape of structures included in the structural information are generally accurate, the relative direction determined from the captured image and structural information, the distance between the wireless communication terminal 100 and the structure estimated from the captured image and structural information, the position information measured by the position sensor 130, and the altitude information measured by the altitude sensor can all contain errors. In contrast, the 3D positioning unit 112 can improve the accuracy of determining the 3D position of the wireless communication terminal 100 by using these in combination.

[0051] The 3D positioning unit 112 may, for example, determine the 3D position of the wireless communication terminal 100 (sometimes referred to as the first 3D position) from the relative direction determined from the captured image and structural information, the distance between the wireless communication terminal 100 and the structure estimated from the captured image and structural information, and the position of the structure included in the structural information. Alternatively, it may determine the 3D position of the wireless communication terminal 100 (sometimes referred to as the second 3D position) from only the position information measured by the position sensor 130, or from the position information measured by the position sensor 130 and the altitude information measured by the altitude sensor, and then determine the 3D position of the wireless communication terminal 100 from the first 3D position and the second 3D position.

[0052] For example, the 3D positioning unit 112 identifies the midpoint between the first 3D position and the second 3D position as the 3D position of the wireless communication terminal 100. This makes it possible to determine the 3D position of the wireless communication terminal 100 by considering the 3D position identified from the captured image and structural information, as well as the 3D position identified from either the position sensor 130 alone or both the position sensor 130 and the altitude sensor.

[0053] The 3D positioning unit 112 may prioritize one of the first 3D position and the second 3D position over the other to determine the 3D position of the wireless communication terminal 100. For example, if the 3D positioning unit 112 prioritizes the first 3D position, it will determine the 3D position of the wireless communication terminal 100 as a point on the straight line connecting the first 3D position and the second 3D position, closer to the first 3D position. The degree to which the position is closer to the first 3D position may be predetermined. For example, if the priority of the first 3D position is set to 2 and the priority of the second 3D position is set to 1, the 3D position of the wireless communication terminal 100 will be determined as a point two-thirds of the way from the first 3D position on the straight line connecting the first 3D position and the second 3D position. The priorities of the first 3D position and the second 3D position can be set arbitrarily and can be changed after they have been set. For example, in situations where structures included in the captured image cannot be accurately photographed due to reasons such as bad weather, lowering the priority of the first 3D position to that of the second 3D position can be expected to improve the accuracy of the identified 3D position of the wireless communication terminal 100. Also, for example, in environments where the positioning accuracy of the position sensor 130 is low, lowering the priority of the second 3D position to that of the first 3D position can be expected to improve the accuracy of the identified 3D position of the wireless communication terminal 100. Furthermore, for example, in environments where the positioning accuracy of the position sensor 130 is considered to be very high, lowering the priority of the second 3D position to that of the first 3D position can be expected to improve the accuracy of the identified 3D position of the wireless communication terminal 100.

[0054] If the structure identification unit 108 identifies multiple structures from the captured image, the structure information acquisition unit 110 may acquire structure information for each of the multiple structures from the 3D map data. The 3D position identification unit 112 may use the multiple structure information acquired by the structure information acquisition unit 110 to determine the 3D position of the wireless communication terminal 100. For example, the 3D position identification unit 112 uses the structure included in the captured image and the structure information for each of the multiple structures to determine the 3D position of the wireless communication terminal 100. Then, the 3D position identification unit 112 determines the 3D position of the wireless communication terminal 100 from the multiple 3D positions of the wireless communication terminal 100 identified using the multiple structure information, and the 3D position of the wireless communication terminal 100 identified using location information, or location information and altitude information. By using multiple structures, it is possible to improve the accuracy of determining the 3D position of the wireless communication terminal 100.

[0055] The terminal information acquisition unit 106 may acquire multiple captured images. For example, the terminal information acquisition unit 106 may acquire a first captured image taken by the camera 120 of the wireless communication terminal 100 in a first direction, first position information and first angle information of the wireless communication terminal 100 at the time the first captured image was taken, a second captured image taken by the camera 120 of the wireless communication terminal 100 in a second direction different from the first direction, and second position information and second angle information of the wireless communication terminal 100 at the time the second captured image was taken. In this case, the structure identification unit 108 may identify a first structure included in the first captured image and a second structure included in the second captured image, and the structure information acquisition unit 110 may acquire first structure information and second structure information of the first and second structures identified by the structure identification unit from the 3D map data. The 3D positioning unit 112 may determine the 3D position of the wireless communication terminal 100 based on the first captured image, first position information, and first angle information acquired by the terminal information acquisition unit 106, the second captured image, second position information, and second angle information, and the first structure information and second structure information acquired by the structure information acquisition unit 110. For example, the 3D positioning unit 112 may determine the 3D position of the wireless communication terminal 100 using the 3D position of the wireless communication terminal 100 determined using the first captured image and first structure information, the 3D position of the wireless communication terminal 100 determined using the first position information, the 3D position of the wireless communication terminal 100 determined using the second captured image and second structure information, and the 3D position of the wireless communication terminal 100 determined using the second position information.

[0056] The target 3D position information acquisition unit 114 acquires target 3D position information indicating the 3D position of the target. For example, the target 3D position information acquisition unit 114 acquires target 3D position information received from another wireless communication terminal 100.

[0057] Based on the three-dimensional position of the wireless communication terminal 100 identified by the three-dimensional position identification unit 112 and the target three-dimensional position information acquired by the target three-dimensional position information acquisition unit 114, the display control unit 116 displays guide data 150 on the display 101 of the wireless communication terminal 100 to adjust the position and angle of the wireless communication terminal 100 to the three-dimensional position indicated by the target three-dimensional position information.

[0058] For example, the display control unit 116 displays a marker 154 at the 3D position indicated by the target 3D position information, based on the position and angle of the wireless communication terminal 100. The display control unit 116 may change the display mode of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 match the 3D position indicated by the target 3D position information or not.

[0059] For example, the display control unit 116 changes the color of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conforms to the 3D position indicated by the foreground 3D position information or not. For example, the display control unit 116 changes the shape of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conforms to the 3D position indicated by the foreground 3D position information or not. The display control unit 116 changes the size of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conforms to the 3D position indicated by the foreground 3D position information or not. The display control unit 116 changes the transparency of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conforms to the 3D position indicated by the foreground 3D position information or not. The display control unit 116 changes the type of the marker 154 depending on whether the position and angle of the wireless communication terminal 100 conforms to the 3D position indicated by the foreground 3D position information or not.

[0060] The weather information acquisition unit 118 acquires weather information for the area where the wireless communication terminal 100 and the structure are located at the time the camera 120 of the wireless communication terminal 100 takes the captured image. The 3D positioning unit 112 may further use the weather information acquired by the weather information acquisition unit 118 to determine the relative direction between the wireless communication terminal 100 and the structure. For example, the 3D positioning unit 112 estimates the position of the sun based on the time when the wireless communication terminal 100 took the captured image 300, and estimates the shadow of the structure from the estimated position of the sun and the weather information. By using weather information, it is possible to estimate the shadow of the structure while taking into account differences such as sunny, cloudy, and rainy conditions, thereby improving the accuracy of shadow estimation.

[0061] Figure 9 schematically shows an example of the hardware configuration of a computer 1200 that functions as a wireless communication terminal 100. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the apparatus according to this embodiment, or to cause the computer 1200 to execute operations associated with the apparatus according to this embodiment or such one or more "parts", and / or to cause the computer 1200 to execute a process or a stage of such process according to this embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0062] The computer 1200 according to this embodiment includes a CPU 1212, RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive and a solid-state drive, etc. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0063] The CPU 1212 operates according to the programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires the image data created by the CPU 1212 and stores it in the frame buffer provided in RAM 1214 or within itself, and enables the image data to be displayed on the display device 1218.

[0064] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0065] The ROM 1230 stores boot programs and / or hardware-dependent programs of the computer 1200, which are executed by the computer 1200 upon activation. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0066] The program is provided on a computer-readable storage medium such as a DVD-ROM or IC card. The program is read from the computer-readable storage medium and installed on a storage device 1224, RAM 1214, or ROM 1230, which are examples of computer-readable storage media, and executed by the CPU 1212. The information processing described within these programs is read by the computer 1200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the operation or processing of information in accordance with the use of the computer 1200.

[0067] For example, when communication is performed between a computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into RAM 1214 and, based on the processing described in the communication program, instruct the communication interface 1222 to perform communication processing. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM, or IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area provided on the recording medium.

[0068] Furthermore, the CPU 1212 may read all or necessary parts of a file or database stored on an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), or an IC card into the RAM 1214, and perform various types of processing on the data in the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

[0069] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 1212 may perform various types of processing on the data read from RAM 1214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 1214. The CPU 1212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 1212 may search among the multiple entries for an entry that matches the specified condition for the attribute value of the first attribute, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies the predetermined condition.

[0070] The program or software module described above may be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0071] In this embodiment, blocks in the flowchart and block diagram may represent a stage in a process in which an operation is performed or a "part" of a device that has the role of performing an operation. A particular stage and "part" may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits, such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, exclusive OR, negated AND, negated OR, and other logical operations, flip-flops, registers, and memory elements.

[0072] A computer-readable storage medium may include any tangible device capable of storing instructions that can be executed by a suitable device, and as a result, a computer-readable storage medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital multipurpose disc (DVD), Blu-ray® disc, memory stick, integrated circuit card, etc.

[0073] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and traditional procedural programming languages ​​such as the C programming language or similar programming languages.

[0074] Computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the internet to a processor or programmable circuit of a general-purpose computer, special-purpose computer, or other programmable data processing device, in order for the processor or programmable circuit of a programmable data processing device such as a computer to execute the instructions in order to create means for performing operations specified in a flowchart or block diagram. Here, the computer may be a PC (personal computer), tablet computer, smartphone, workstation, server computer, general-purpose computer, or special-purpose computer, and may also be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system and is a computer in a broad sense. In a distributed computing system, multiple computers execute a program collectively by each computer executing a part of the program and passing data during program execution between computers as needed.

[0075] Examples of processors include computer processors, central processing units, processing units, microprocessors, digital signal processors, controllers, and microcontrollers. A computer may have one or more processors. In a multiprocessor system with multiple processors, each processor executes a portion of the program, and the processors collectively execute the program by passing program execution data between them as needed. For example, in the execution of multitasks, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which part of a program each processor executes changes dynamically. Which part of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

[0076] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0077] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before" or "prior to," and that these can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," and "next," for convenience, this does not mean that it is essential to perform the operations in that order. [Explanation of Symbols]

[0078] 100 Wireless communication terminal, 101 Display, 102 Memory unit, 104 Map data acquisition unit, 106 Terminal information acquisition unit, 108 Structure identification unit, 110 Structure information acquisition unit, 112 3D position identification unit, 114 Target 3D position information acquisition unit, 116 Display control unit, 118 Weather information acquisition unit, 120 Camera, 130 Position sensor, 140 Angle sensor, 150 Guide data, 152 Guide, 154 Marker, 200 Optical wireless communication device, 300 Captured image, 310 Structure, 402 Captured image, 404 Planar information, 406 Gyro information, 408 Image recognition, 410 Feature point extraction, 412 Impression state extraction, 414 Target range extraction, 416 3D map data, 418 Target object identification, 420 3D structure data, 422 Object position information, 424 Image impression information, 426 3D position information, 428 Correction value calculation data, 430 Correction value calculation data, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / Output controller, 1222 Communication interface, 1224 Storage device, 1230 ROM, 1240 Input / Output chip

Claims

1. A wireless communication terminal having a camera, a position sensor and an angle sensor, A map data storage unit that stores three-dimensional map data including structural information showing the location and shape of structures, A terminal information acquisition unit that acquires the captured image taken by the camera and the position information and angle information of the wireless communication terminal at the time the captured image was taken. A structure identification unit that identifies structures included in the captured image, A structure information acquisition unit acquires the structure information of the structure identified by the structure identification unit from the three-dimensional map data, A three-dimensional positioning unit identifies the three-dimensional position of the wireless communication terminal based on the captured image, position information, and angle information acquired by the terminal information acquisition unit, and the structural information acquired by the structural information acquisition unit. Equipped with, The three-dimensional positioning unit determines the relative direction between the wireless communication terminal and the structure from the captured image and the structure information, and determines the three-dimensional position of the wireless communication terminal using the determined relative direction, the position of the wireless communication terminal indicated by the position information, the angle of the wireless communication terminal indicated by the angle information, and the position of the structure indicated by the structure information.

2. The wireless communication terminal according to claim 1, wherein the three-dimensional positioning unit determines the relative direction between the wireless communication terminal and the structure from the shape of the structure included in the captured image and the shape of the structure indicated by the structure information, and determines the three-dimensional position of the wireless communication terminal using the determined relative direction, the position of the wireless communication terminal indicated by the position information, the angle of the wireless communication terminal indicated by the angle information, and the position of the structure indicated by the structure information.

3. The wireless communication terminal according to claim 2, wherein the three-dimensional positioning unit determines the relative direction between the wireless communication terminal and the structure using the shape of the structure included in the captured image, the shape of the structure indicated by the structure information, the time when the wireless communication terminal captured the captured image, and the shadow of the structure included in the captured image.

4. The wireless communication terminal according to claim 3, wherein the three-dimensional positioning unit further uses weather information of the area where the wireless communication terminal and the structure are located at the time the wireless communication terminal took the captured image to determine the relative direction between the wireless communication terminal and the structure.

5. The terminal information acquisition unit further acquires the altitude information of the wireless communication terminal at the time the captured image was taken, which is measured by the altitude sensor of the wireless communication terminal. The wireless communication terminal according to any one of claims 1 to 4, wherein the three-dimensional positioning unit further determines the three-dimensional position of the wireless communication terminal based on the altitude information.

6. The aforementioned structure identification unit identifies multiple structures included in the captured image, The structure information acquisition unit acquires the structure information of each of the multiple structures identified by the structure identification unit from the three-dimensional map data. The wireless communication terminal according to any one of claims 1 to 4, wherein the three-dimensional positioning unit determines the three-dimensional position of the wireless communication terminal based on the captured image, position information, and angle information acquired by the terminal information acquisition unit, and a plurality of structural information acquired by the structural information acquisition unit.

7. The terminal information acquisition unit acquires a first captured image taken by the camera of the wireless communication terminal in a first direction, first position information and first angle information of the wireless communication terminal at the time the first captured image was taken, a second captured image taken by the camera of the wireless communication terminal in a second direction different from the first direction, and second position information and second angle information of the wireless communication terminal at the time the second captured image was taken. The structure identification unit identifies the first structure included in the first captured image and the second structure included in the second captured image. The structure information acquisition unit acquires the first structure information and the second structure information of the first structure and the second structure identified by the structure identification unit from the three-dimensional map data. The wireless communication terminal according to any one of claims 1 to 4, wherein the three-dimensional positioning unit determines the three-dimensional position of the wireless communication terminal based on the first captured image, the first position information, and the first angle information acquired by the terminal information acquisition unit, the second captured image, the second position information, and the second angle information, and the first structure information and the second structure information acquired by the structure information acquisition unit.

8. A target 3D position information acquisition unit acquires target 3D position information indicating the 3D position of the target, A display control unit displays guide data on the display of the wireless communication terminal to adjust the position and angle of the wireless communication terminal to the three-dimensional position indicated by the target three-dimensional position information, based on the three-dimensional position of the wireless communication terminal identified by the three-dimensional position identification unit and the target three-dimensional position information. A wireless communication terminal according to any one of claims 1 to 4, comprising:

9. The wireless communication terminal according to claim 8, wherein the display control unit displays a marker at the position of the three-dimensional location indicated by the target three-dimensional position information, based on the position and angle of the wireless communication terminal, and changes the display mode of the marker depending on whether the position and angle of the wireless communication terminal conform to the three-dimensional location indicated by the target three-dimensional position information or not.

10. The wireless communication terminal according to claim 9, wherein the display control unit changes the color of the marker depending on whether the position and angle of the wireless communication terminal match the three-dimensional position indicated by the target three-dimensional position information or not.

11. An information processing method performed by a wireless communication terminal having a camera, a position sensor and an angle sensor, A terminal information acquisition step involves acquiring the captured image taken by the camera and the location information and angle information of the wireless communication terminal at the time the captured image was taken. A structure identification step in which structures included in the captured image are identified, A structural information acquisition step involves acquiring the structural information of the structure identified in the structural identification step from three-dimensional map data that includes structural information indicating the location and shape of the structure, A three-dimensional position identification step in which the three-dimensional position of the wireless communication terminal is identified based on the captured image, position information, and angle information acquired in the terminal information acquisition step and the structure information acquired in the structure information acquisition step. Equipped with, The three-dimensional position determination step is an information processing method which involves determining the relative direction between the wireless communication terminal and the structure from the captured image and the structure information, and determining the three-dimensional position of the wireless communication terminal using the determined relative direction, the position of the wireless communication terminal indicated by the position information, the angle of the wireless communication terminal indicated by the angle information, and the position of the structure indicated by the structure information.

12. A wireless communication terminal having a camera, a position sensor and an angle sensor, A terminal information acquisition step involves acquiring the captured image taken by the camera and the location information and angle information of the wireless communication terminal at the time the captured image was taken. A structure identification step in which structures included in the captured image are identified, A structural information acquisition step involves acquiring the structural information of the structure identified in the structural identification step from three-dimensional map data that includes structural information indicating the location and shape of the structure, Based on the captured image, position information, and angle information acquired in the terminal information acquisition step, and the structure information acquired in the structure information acquisition step, the relative direction between the wireless communication terminal and the structure is determined from the captured image and the structure information, and the three-dimensional position determination step is determined using the determined relative direction, the position of the wireless communication terminal indicated by the position information, the angle of the wireless communication terminal indicated by the angle information, and the position of the structure indicated by the structure information, to determine the three-dimensional position of the wireless communication terminal. A program to execute.

Images