Information processing methods, computer programs, and information processing terminals

Abstract

The goal is to provide users with a more convenient three-dimensional virtual space equipped with high-precision location information. [Solution] An information processing method is provided that uses a portable information processing terminal, wherein the information processing terminal is equipped with a position acquisition unit capable of acquiring position data representing the position of the information processing terminal in real space, and comprises image data acquisition processing, virtual space generation processing, and assignment processing, wherein the image data acquisition processing acquires image data including an image of an object, the virtual space generation processing generates a three-dimensional virtual space based on the image data, and the assignment processing assigns coordinates in real space to the components of the virtual space based on the position data acquired by the position acquisition unit.

Description

Technical Field

[0006] , , , ,

[0001] The present invention relates to an information processing method, a computer program, and an information processing terminal.

Background Art

[0002] Patent Document 1 discloses a technique for generating a three-dimensional virtual space from image data. In the technique of this publication, photogrammetry processing is executed on a server on the Internet.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The inventor of the present application has been intensively researching a technique for imparting high-precision position information to a three-dimensional virtual space. Conventionally, there has been an inconvenience that necessary data has to be uploaded to a server on the Internet in order to generate a three-dimensional virtual space.

[0005] An object of the present invention is to provide a user with a three-dimensional virtual space to which high-precision position information is imparted with higher convenience.

Means for Solving the Problems

[0006] According to the present invention, an information processing method is provided that uses a portable information processing terminal, wherein the information processing terminal is equipped with a position acquisition unit capable of acquiring position data representing the position of the information processing terminal in real space, and comprises image data acquisition processing, virtual space generation processing, and assignment processing, wherein the image data acquisition processing acquires image data including an image of an object, the virtual space generation processing generates a three-dimensional virtual space based on the image data, and the assignment processing assigns coordinates in real space to the components of the virtual space based on the position data acquired by the position acquisition unit.

[0007] According to the present invention, a three-dimensional virtual space with high-precision location information can be generated on a portable information processing terminal without uploading data to a server, thereby improving user convenience. [Brief explanation of the drawing]

[0008] [Figure 1] This is an overall configuration diagram including the information processing terminal 1 and location acquisition terminal 2 of the embodiment. [Figure 2] Figure 2A shows the rear side of the information processing terminal 1 to which the location acquisition terminal 2 is attached, and Figure 2B shows the front side thereof. [Figure 3] This diagram illustrates the hardware configuration of information processing terminal 1 using functional blocks, and shows the connection between it and location acquisition terminal 2. [Figure 4] This diagram shows a functional block diagram of the computer program of the embodiment when it is executed on the information processing terminal 1, and the connection between it and the location acquisition terminal 2. [Figure 5] This is a processing flow diagram illustrating the information processing method and computer program of the embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other. Furthermore, each feature constitutes an independent invention.

[0010] 1. Description of the Configuration of the Embodiment 1.1. Overall Structure As shown in Figure 1, in this embodiment, as an example, user U uses a portable information processing terminal 1 to survey the object Y at the survey site X. The information processing terminal 1 is a portable terminal, and in this embodiment, it is a smartphone as an example. A location acquisition terminal 2 is attached to the information processing terminal 1. In this embodiment, the location acquisition terminal 2 is capable of communicating with an artificial satellite Z and a base station K, which are also illustrated in Figure 1. The imaging positions P1 to P6 in Figure 1 are examples of positions where user U takes images of the object Y.

[0011] 1.2. Hardware Configuration 1.2.1. External Configuration Figures 2A and 2B illustrate an example of an information processing terminal 1 with a location acquisition terminal 2 attached. In this embodiment, the location acquisition terminal 2 can be attached to the back of the information processing terminal 1, for example. The planar dimensions of the location acquisition terminal 2 are defined such that the portion other than the antenna 2b is almost entirely contained within the back of the information processing terminal 1.

[0012] In the location acquisition terminal 2 of this embodiment, various circuit configurations are housed in a plate-shaped housing 2a. The housing 2a has thickness in the depth direction of the paper in Figures 2A and 2B. In this embodiment, as shown in Figure 2A as an example, an antenna 2b is connected to the top of the housing 2a, and a battery 2c is provided inside the housing 2a, and these are connected to the main circuit 2d that realizes the location acquisition function. By adopting such a configuration, there is an advantage that the user U can grasp the information processing terminal 1 and the location acquisition terminal 2 as a single unit with one hand. Although the description is given as an example of a configuration in which the location acquisition terminal 2 has a battery 2c, it is not limited to this, and the location acquisition terminal 2 may not have a battery 2c and may be configured to receive power from another terminal (for example, the information processing terminal 1), or to receive power from an external power source.

[0013] 1.2.2. Functional Blocks of Hardware Configuration As shown in Figure 3, the information processing terminal 1 includes a communication unit 10, a storage unit 11, an arithmetic processing unit 12, a touch panel 13, and a camera 14, and these components are electrically connected within the information processing terminal 1 via a communication bus 15.

[0014] The memory unit 11 and the arithmetic processing unit 12 shall have information processing specifications sufficient to execute various processes of the virtual space generation unit 102 (see Figure 4), which will be described later, within the terminal. The touch panel 13 is an input unit that receives user U's operation input and also has the function of a display unit that displays various screens. In this embodiment, the touch panel is described as having the input unit and display unit configured as an integrated unit, but it is not limited to this. The input unit that receives user U's operation input and the display unit that displays images, etc., may be configured separately. In this case, the input unit may be configured to receive input from, for example, an instruction unit (e.g., a button or lever) provided on the information processing device 1, or it may be configured to receive voice instructions (input) from a microphone. The display unit may employ various display devices such as liquid crystal displays, organic EL displays, and plasma displays.

[0015] As can be seen in Figure 2A, in this embodiment, three different cameras 14 are provided as an example. Of the three cameras 14, one may be a wide-angle camera, another an ultra-wide-angle camera, and another a telephoto camera. User U may use these cameras interchangeably to image the survey target Y. Note that the configuration of the camera 14 described here is merely an example and is not limited to this configuration.

[0016] As shown in Figure 3, the location acquisition terminal 2 can communicate with the storage unit 11 and other components within the information processing terminal 1 via the communication unit 10 and the communication bus 15. There are no limitations on the connection configuration between the location acquisition terminal 2 and the communication unit 10; any wireless connection can be used. Specifically, for example, Bluetooth® or Wi-Fi can be used. Furthermore, it is not limited to wireless connections; any wired connection (e.g., USB-TypeC) may also be used.

[0017] 1.3 Software Configuration (Functional Block Diagram) When the computer program of the embodiment is executed on the information processing terminal 1, the functional block diagram shown in FIG. 4 is realized on the information processing terminal 1. As shown in FIG. 4, the information processing terminal 1 includes an image acquisition unit 101, a virtual space generation unit 102, an allocation unit 103, and a coordinate map generation unit 104.

[0018] In the embodiment, as an example, the information processing terminal 1 is a smartphone. Therefore, as an example, an application program is downloaded from an external server to the information processing terminal 1, installed in the information processing terminal 1, and each functional unit is realized by the arithmetic processing unit 12 executing this application program. However, this is not limited thereto. As another example, the program may be stored in and provided by a non-temporary computer-readable recording medium. Further, each functional unit is not limited to being realized by software, and may be realized by hardware. Also, in the embodiment, various information and concepts including the same are handled, and these are represented by the level of signal values or quantum bits as a set of binary bits composed of 0 or 1, and communication and arithmetic can be executed in the above software or hardware modes.

[0019] As shown in FIG. 4, the position acquisition terminal 2 is communicably connected to each functional unit of the information processing terminal 1. The position data 105 that can be acquired by the position acquisition terminal 2 may include, for example, latitude, longitude, and altitude, and may further include, for example, geoid height in addition to these. Here, the case where the position data 105 uses a coordinate system such as latitude and longitude is described as an example, but it is not limited thereto, and a plane rectangular coordinate system may be adopted.

[0020] The image acquisition unit 101 of the information processing terminal 1 is configured to be able to acquire the image data 101a from the storage unit 11. The image data 101a may be the one captured by the camera 14 at the survey site X. Alternatively, the image data 101a may be any image data acquired (for example, downloaded) from the outside via the communication unit 10 and transmitted to the storage unit 11. Further, the image data 101a may be a combination of the data captured by the camera 14 and the data acquired from the outside. The image data 101a may be a still image or a moving image.

[0021] The virtual space generation unit 102 generates a three-dimensional virtual space 102a from the image data 101a. Specifically, in the embodiment, as an example, the virtual space generation unit 102 generates the virtual space 102a from the image data 101a including a plurality of images by means of SfM (Structure from Motion) processing. SfM is a general term for technologies that restore the shape of an object from a plurality of images of the object. According to the SfM processing, for example, a point cloud or a mesh, etc. is output as a component of the virtual space 102a. In the embodiment, as an example, photogrammetry processing is adopted for the SfM processing. In photogrammetry, it is possible to generate a virtual space representing the object in three dimensions by analyzing a large number of captured images obtained by photographing the object to be photographed (the survey object Y in the embodiment) from multiple directions. Since each of the technologies of SfM and photogrammetry is already a known technology and not a new matter, detailed description thereof is omitted.

[0022] The allocation unit 103 assigns coordinates to the components of the virtual space 102a based on the position data 105 acquired by the position acquisition terminal 2. The components of the virtual space 102a can be any components determined according to the virtual space generation process adopted. For example, the components of the virtual space 102a may be a three-dimensional point cloud, a mesh, a voxel, or any 3D object. The coordinates assigned to each component may include latitude, longitude, and elevation, and may also include geoid height in addition to these. The information to be included in these coordinates (latitude, longitude, elevation, geoid height, plane orthogonal coordinate system, etc.) can be determined by the user depending on the situation. There are no limitations on the specific calculation method for coordinate assignment in the allocation unit 103. In this embodiment, as an example, the virtual space 102a is generated using SfM processing from image data 101a captured at each of the imaging positions P1 to P6. Here, since the position acquisition terminal 2 uses RTK, which will be described later, for position acquisition, the position acquisition terminal 2 can acquire high-precision position data 105 at imaging positions P1 to P6. As a result, coordinates based on the high-precision position data 105 can be assigned to each component of the virtual space 102a generated by SfM processing.

[0023] The correspondence between the elements of the virtual space 102a and the assigned coordinates is stored in the storage unit 11. As an example of how this correspondence is stored, in this embodiment, the coordinate map generation unit 104 generates a coordinate map 104a. The coordinate map 104a is a map that defines the correspondence between the elements in the virtual space 102a and the coordinates assigned by the assignment unit 103. The coordinate map 104a is stored in the storage unit 11.

[0024] As mentioned above, the information processing terminal 1 is further equipped with a touch panel 13 that also functions as a display unit. In response to user operations such as tapping a processing result display icon, a coordinate map 104a is displayed on the touch panel 13.

[0025] The position acquisition terminal 2 acquires position data 105. In this embodiment, as an example of how the position acquisition terminal 2 acquires position data 105, the position acquisition terminal 2 acquires position data 105 by communicating with the artificial satellite Z, and in this case, RTK (Real Time Kinematic) is used for communication between the position acquisition terminal 2 and the artificial satellite Z. The RTK technology itself is already publicly known and is not a new matter, so a detailed explanation is omitted.

[0026] 2. Description of the operation of the embodiment Referring to Figure 5, an example of an information processing method according to the embodiment will be described. The computer program according to the embodiment executes each of the steps (information processing method) described below on a portable information processing terminal 1 to which a location acquisition terminal 2 is attached.

[0027] In the information processing method defined in the processing flow of Figure 5, first, user U starts the computer program of the embodiment installed on the information processing terminal 1. After startup, the processing flow of Figure 5 begins.

[0028] In the processing flow shown in Figure 5, first, the location acquisition process is executed (step S1). In this embodiment, the location acquisition terminal 2 starts the process of acquiring location data 105 using RTK as an example. The acquired location data 105 is continuously transmitted to the information processing terminal 1.

[0029] Next, the image acquisition process (image data acquisition process) is performed (step S2). In this step, the user U uses the camera 14 of the information processing terminal 1 to capture images at multiple imaging positions P1 to P6 while moving around the survey object Y at the survey site X. For the sake of explanation, an example using imaging positions P1 to P6 is described, but it should be noted that the imaging positions are not limited to these. The multiple images obtained in this way were taken at multiple different locations at the survey site X, and each of them shows the survey object Y. Image data 101a is data that includes these multiple images.

[0030] There are no limitations on the survey object Y. The survey object Y may be, for example, a topographical feature, in which case it can be used for scanning the topography. The survey object Y is not limited to topography; it may be any building or any tangible object. The survey object Y may be of any size and shape.

[0031] Next, the virtual space generation process is executed (step S3). In this step, the virtual space generation unit 102 executes a process (SfM process) to generate a three-dimensional virtual space 102a from the image data 101a. In this embodiment, this processing program (SfM process) is, for example, photogrammetry. Through the photogrammetry process, a three-dimensional virtual space 102a is generated from the image data 101a, and a three-dimensional point cloud, which is a component of the virtual space 102a, is output.

[0032] In this example control flow, it is described that the virtual space generation process in step S3 is executed after the image acquisition in step S2, but it is not limited to this. The SfM process in step S3 may be executed in parallel with the image acquisition in step S2. This can shorten the processing time of this control flow and further improve convenience.

[0033] Next, the assignment process is performed (step S4). In this step, the assignment unit 103 assigns the coordinates of each component (three-dimensional point cloud) of the virtual space 102a based on the position data 105.

[0034] In this example control flow, it is described that the assignment process in step S4 is executed after the SfM process in step S3, but it is not limited to this. The assignment process in step S4 may be executed in parallel with the SfM process in step S3. This can shorten the processing time of this control flow and further improve convenience. As mentioned above, steps S2 and S3 may be executed in parallel, and as stated here, steps S3 and S4 may be executed in parallel, meaning that the processes in steps S2 to S4 may be executed in parallel.

[0035] Next, the coordinate map generation process is executed (step S5). This causes the coordinate map generation unit 104 to generate the coordinate map 104a and store it in the storage unit 11. Next, the display process is executed (step S6). This causes the coordinate map 104a to be displayed on the touch panel 13, allowing the coordinate map 104a to be presented to the user U. Once the above processes are completed, the processing flow shown in Figure 5 is finished.

[0036] In this embodiment, steps S1 to S5 are processed by the information processing terminal 1 and the location acquisition terminal 2. In other words, the information processing method according to this embodiment can be completed by the information processing terminal 1 and the location acquisition terminal 2, and coordinate assignment and coordinate map acquisition can be performed without uploading the location data acquired by the location acquisition terminal 2 and the image data acquired by the information processing terminal 1 to an external server or the like from the information processing terminal 1. In other words, in this embodiment, the information processing terminal 1 is configured to execute a first information processing mode in which it completes information processing such as coordinate assignment and coordinate map acquisition itself.

[0037] Furthermore, to accommodate situations where it becomes necessary to upload data to a server, the information processing terminal 1 may be capable of executing a second information processing mode. In the second information processing mode, for example, image data of the survey site X and its corresponding location data are uploaded from the information processing terminal 1 to an external server, the server assigns coordinates and generates a coordinate map, and the information processing terminal 1 retrieves this map from the server. In other words, in the second information processing mode, steps with a large amount of information processing, such as steps S3 to S5 described above, are handled by the server. By enabling the information processing terminal 1 to switch between the first and second information processing modes in this way, it is expected that convenience will be further enhanced. Mode switching can be achieved, for example, by displaying icons on the display unit of the information processing terminal 1 that allow the user to select the mode. Mode switching can be performed at any time; for example, the user may be prompted to select a mode before step S1, or the mode may be switched after step S2 (after the location acquisition process and image acquisition process are completed).

[0038] Furthermore, in the information processing of the information processing terminal 1, the second information processing mode may produce a higher-resolution point cloud (relatively higher density of points per unit space in the coordinate system) than the first information processing mode, while the first information processing mode may prioritize speed and have a shorter processing time than the second information processing mode. This allows users to switch between modes depending on the situation, further improving convenience.

[0039] 3. Description of the operation and effects of the embodiment According to the information processing method and computer program of this embodiment, a three-dimensional virtual space with high-precision location information can be generated without uploading data to a server, thereby improving user convenience. More specifically, according to this embodiment, it is not necessary to upload multiple image data and high-precision location information to an online server for virtual space generation when generating a three-dimensional virtual space. This has the advantage of reducing various server-related costs, as well as data communication costs and communication time. It also has the advantage of allowing the processing results (three-dimensional point cloud, etc.) at the information processing terminal 1 to be easily confirmed on-site. In particular, the configuration of this embodiment, which does not require uploading, is suitable when a communication environment for uploading to an online server is not in place. Furthermore, when uploading to an online server, the speed of various data processing operations depends on the number of users on the server. There is also concern that processing queues may occur on the server. In contrast, according to the information processing method and computer program of the embodiment, a three-dimensional virtual space can be generated without uploading data to a server, as described above, thus avoiding the impact of the number of users on the server and preventing queues.

[0040] Virtual space generation technologies such as photogrammetry, an example of SfM processing employed in the embodiment, and other SfM processing methods described later (e.g., NeRF, etc.), have the following advantages compared to known LiDAR (Light Detection and Ranging) technology. In other words, in the case of photogrammetry, for example, the measurement distance depends on the resolution of the image data, so if high-resolution image data is available, it is possible to generate a highly accurate three-dimensional virtual space for long distances.

[0041] Furthermore, according to this embodiment, there is the advantage that the high-precision location information acquired by the location acquisition terminal 2 using RTK can be added to the three-dimensional virtual space.

[0042] Furthermore, according to this embodiment, there is an advantage in that by performing SfM processing in the virtual space generation process (step S3), it is possible to generate a highly accurate three-dimensional virtual space from the image data 101a.

[0043] Furthermore, according to this embodiment, the inclusion of a coordinate map generation process (step S5) offers the advantage of conveniently saving the correspondence between the components of the virtual space 102a and their coordinates.

[0044] Furthermore, according to this embodiment, the inclusion of a display process (step S6) has the advantage of making it easy to present the location information coordinate map 104a to the user U.

[0045] Furthermore, according to this embodiment, the camera 14 mounted on the information processing terminal 1 can capture image data 101a. Therefore, user U has the advantage of being able to easily obtain a three-dimensional virtual space with high-precision location information using a palm-sized device.

[0046] 4. Variations 4.1. Variations related to virtual space generation In this embodiment, the virtual space generation unit 102 employs SfM processing as the process for generating a three-dimensional virtual space 102a from image data 101a, and as an example, SfM processing is assumed to be the process in photogrammetry. However, it is not limited to this, and instead of photogrammetry processing, neural rendering processing such as NeRF (Neural Radiance Fields) or 3D Gaussian splatting processing may be employed.

[0047] Neural rendering, such as NeRF, generates a three-dimensional image (three-dimensional scene) from multiple two-dimensional images using a neural network. In an embodiment, if SfM processing is a process in neural rendering, any components that constitute the three-dimensional image (three-dimensional scene) may be used as components of the virtual space 102a. These arbitrary components may be, for example, a three-dimensional point cloud, a mesh, a voxel, or a 3D object. In this case, the assignment unit 103 may assign coordinates to each of these arbitrary components based on the position data 105.

[0048] In an embodiment where the SfM process is a 3D Gaussian splatting process, individual 3D Gaussians may be used as components of the virtual space 102a, or more specifically, the center points of individual 3D Gaussians may be used. In this case, the assignment unit 103 may assign coordinates to each 3D Gaussian or each center point of the 3D Gaussians based on the position data 105.

[0049] 4.2. Modification Example 1 of Location Data Acquisition In one embodiment, the position acquisition terminal 2 acquires position data 105 by communicating with the satellite Z, and RTK is used for this communication. However, it is not limited to this, and GNSS (Global Navigation Satellite System), CLAS (Centimeter-Level Augmentation Service), or SLAS (Submeter-Level Augmentation Service) may be used for communication between the position acquisition terminal 2 and the satellite Z. As another example, the position acquisition terminal 2 may acquire position data 105 by communicating with the base station K. These communication technologies themselves are already publicly known and are not new, so a detailed explanation is omitted.

[0050] Furthermore, by employing CLAS or SLAS in this embodiment, there is the advantage of being able to generate a three-dimensional virtual space with high-precision location information even in locations with poor radio wave conditions or outside of radio wave coverage for the information processing terminal 1. Here, CLAS is a service provided by the Quasi-Zenith Satellite System "Michibiki" in Japan. By using this service, location information can be acquired with an accuracy of a few centimeters. For example, in civil engineering and construction work sites, infrastructure management sites, or disaster sites, there are many situations where mobile phone signals are out of range. In this respect, if CLAS is adopted as the communication method for the location acquisition terminal 2 in this embodiment, a three-dimensional virtual space with high-precision location information can be obtained quickly on-site, regardless of the communication environment. As a result, user convenience is dramatically improved.

[0051] Here, we will discuss one of the differences between RTK and CLAS, etc. RTK relies on receiving mobile phone signals from base stations K, etc., to obtain correction information. In contrast, CLAS, etc., communicates with satellite Z, so it can be used in areas where signals from base stations K, etc., do not reach. More specifically, RTK achieves high-precision positioning based on position correction information distributed from base stations K, etc., of mobile phone towers. In other words, RTK can be used in environments where mobile phone signals are available and there is an internet connection. In contrast, CLAS and SLAS are effective in areas with poor communication environments, and CLAS positioning using the Michibiki satellite is an example of this. Since position correction information is included in the signal from satellite Z (L6 signal), centimeter-precision positioning (CLAS) and sub-meter-precision positioning (SLAS) are possible even in places where mobile phone signals do not reach.

[0052] 4.3. Modification Example 2 of Location Data Acquisition In the embodiments and the modifications described in Section 4.2 above, a configuration was described in which the position acquisition terminal 2 communicates with the artificial satellite Z (using, for example, RTK, GNSS, CLAS, or SLAS) to acquire position data 105, but the invention is not limited to this. At the survey site, a base station K capable of communicating with the position acquisition terminal 2 may be installed, and the position acquisition terminal 2 may be configured to acquire position data 105 from this base station K. In this case, the base station K is configured to acquire base station coordinates corresponding to the position of the base station. Then, the position acquisition terminal 2 can acquire position data 105 based on the base station coordinates and distance information.

[0053] The location acquisition terminal 2 can obtain base station coordinates from base station K. Distance information may be obtained by the location acquisition terminal 2 by calculating the distance information through communication between the location acquisition terminal 2 and the base station K. Alternatively, the base station K may calculate and obtain the distance information through communication between the location acquisition terminal 2 and the base station K may transmit the distance information to the location acquisition terminal 2, thereby allowing the location acquisition terminal 2 to obtain the distance information.

[0054] Here, the distance information is based on the distance between base station K and location acquisition terminal 2. This distance information may, for example, represent the distance between base station K and location acquisition terminal 2. Alternatively, it may represent the relative positional relationship between base station K and location acquisition terminal 2. This positional relationship can be understood, for example, as the relative position (X, Y, Z) of location acquisition terminal 2 with respect to base station K.

[0055] Furthermore, when base station K acquires base station coordinates, it can employ communication methods such as RTK, GNSS, CLAS, or SLAS.

[0056] 4.4. Modifications relating to cameras and image data acquisition In this embodiment, for example, the camera 14 is mounted on the information processing terminal 1, but it is not limited to this. The camera 14 may be separate from the information processing terminal 1, or it may be connected to the information processing terminal 1 to enable communication of image data 101a.

[0057] In this embodiment, image data 101a is acquired at the survey site X using the camera 14. However, this is not the only example; as another example, the user U may refer to photo album data or the like stored in the storage unit 11 of the information processing terminal 1, or select multiple images that satisfy the requirements of photogrammetry processing, etc. (provided that each image is assigned location information that can be used as a substitute for location data 105), and use these as image data 101a. The processing from step S2 onward may be performed using the selected image data 101a.

[0058] 4.5. Switching the device used to acquire location data In this embodiment, the location data used in the allocation process was described as being the coordinates obtained by RTK from the location acquisition terminal 2. However, as described below, coordinates calculated by the information processing terminal 1 may also be used.

[0059] In other words, the location data used in the allocation process may be switchable from location data acquired by location acquisition terminal 2 to location data acquired by information processing terminal 1, depending on the accuracy of the location data from location acquisition terminal 2 (in the example of this embodiment, the accuracy of RTK), or conversely, it may be switchable from location data acquired by information processing terminal 1 to location data acquired by location acquisition terminal 2. Specifically, if the predetermined conditions regarding the accuracy of the location data from location acquisition terminal 2 are met (in this example, if RTK accuracy can be ensured), the location data acquired by location acquisition terminal 2 will be used as location data 105. If the predetermined conditions regarding the accuracy of the location data from location acquisition terminal 2 are not met (in this example, if RTK accuracy cannot be ensured), the coordinates acquired by information processing terminal 1 may be used as location data. Furthermore, the location data used in the allocation process may be constructed by combining the location data acquired by the information processing terminal 1 and the location data acquired by the location acquisition terminal 2, depending on the accuracy of the location data from the location acquisition terminal 2.

[0060] The above example describes switching between location data acquisition devices (information processing terminal 1 and location acquisition terminal 2) or combining location data depending on the accuracy of the location data, but it is not limited to this. For example, the location data acquisition devices may be switched or location data may be combined depending on various circumstances such as the processing speed of information processing terminal 1, the remaining battery level of battery 2c, the balance between these, and user requests. In other words, the location data acquisition devices may be switched or location data may be combined depending on predetermined circumstances, not just the accuracy of the location data.

[0061] For the coordinates acquired by the information processing terminal, for example, coordinates obtained using inertial positioning with an inertial sensor may be used, or coordinates obtained using SLAM (Simultaneous Localization and Mapping) may be used.

[0062] For inertial positioning, the inertial sensors used can include, for example, an acceleration sensor or a gyroscope sensor built into the information processing terminal 1. SLAM may employ SLAM using the camera 14, or if the information processing terminal 1 is equipped with LiDAR functionality, SLAM using LiDAR may be employed. In SLAM, for example, coordinates can be acquired by combining the aforementioned camera 14 or LiDAR with the aforementioned inertial sensor.

[0063] 4.6. Processing is shared between information processing terminal 1 and the server. In this embodiment, it has been explained that a second information processing mode can be implemented in which steps with a large amount of information processing capacity, such as steps S3 to S5 shown in Figure 5, are handled by the server, but this is not the only way to go. Some of steps S3 to S5 may be executed by the information processing terminal 1, and the rest of steps S3 to S5 may be executed by the server. As a concrete example, the information processing terminal 1 generates a rough point cloud, and then uploads the calculation data (data related to the rough point cloud) to the server, thereby distributing the relatively heavy processing to the server. After generating the rough point cloud on the information processing terminal 1, it is advisable to display the point cloud on the display unit of the information processing terminal 1. This allows for the quick display of the point cloud, albeit a rough one, improving user convenience, while also shifting the heavy processing burden to the server, which is expected to improve the display speed of the clean (dense) point cloud ultimately displayed on the information processing terminal 1.

[0064] An example of this modified form is given below. If the information processing method employs photogrammetry, the information processing terminal 1 performs SfM processing, and the server performs MVS (Multi-View Stereo) processing. • If the information processing method employs NeRF processing, information processing terminal 1 performs SfM processing and the server performs NeRF processing. • If the information processing method employs 3D Gaussian splatting processing, information processing terminal 1 performs SfM processing and the server performs 3D Gaussian splatting processing.

[0065] 4.7. Others Instead of a smartphone, a tablet device may be used as the information processing terminal 1. Unlike laptop computers, smartphones and tablet devices are primarily designed as information processing terminals suitable for image capture and held in one hand. The typical usage scenario involves holding the terminal in one hand and operating the touch panel with either the holding hand or the other. By attaching the small, lightweight position acquisition terminal 2 according to this embodiment to such an information processing terminal 1, high-precision positioning and image data collection can be easily performed with one hand. In particular, the position acquisition terminal 2 according to this embodiment is equipped with a battery 2c and an antenna 2b, and is small enough to fit in one hand when attached to the information processing terminal 1, thus providing high user convenience when moving during image acquisition. Furthermore, it can help prevent a decrease in the mobility of the user in civil engineering and construction work or infrastructure management sites (e.g., railways, roads, electricity, gas, water), or disaster sites. Furthermore, the embodiment can be applied not only to civil engineering and construction sites, but also to various fields such as on-site surveys by civil engineering consultants, archaeological excavations, agricultural applications, and livestock research (for example, recording the shape and location of cow dung). Moreover, while the user of the embodiment is not limited, it is suitable not only for civil engineering and construction workers, infrastructure workers, and researchers, but also for those who utilize three-dimensional virtual space data, such as land and building surveyors, media, police, and the Self-Defense Forces.

[0066] In this embodiment, the location data 105 is described as being acquired by the location acquisition terminal 2, but it is not limited to this. It is also possible to assign coordinates to the components of the virtual space 102a based on the location data 105 acquired by the information processing terminal 1. In other words, not only steps S2 to S6 shown in Figure 5, but also the location acquisition process S1 in step S1 may be executed by the information processing terminal 1. In this case, the information processing terminal 1 has a location acquisition unit (corresponding to the location acquisition function of the location acquisition terminal 2) as a functional block. Furthermore, in this configuration, the location acquisition terminal 2 may not be necessary.

[0067] 5. Addendum Various embodiments are illustrated below. The embodiments shown below can be combined with each other. [Note 1] An information processing method using a portable information processing terminal, The aforementioned information processing terminal includes a position acquisition unit capable of acquiring position data representing the position of the information processing terminal in real space, It includes image data acquisition processing, virtual space generation processing, and allocation processing. In the aforementioned image data acquisition process, image data including an image of the target is acquired. In the virtual space generation process, a three-dimensional virtual space is generated based on the image data. The assignment process is an information processing method that assigns coordinates in real space to the components of the virtual space based on the position data acquired by the position acquisition unit. [Note 2] The information processing method described in Appendix 1, The position acquisition unit communicates with the satellite to acquire the position data, An information processing method in which GNSS, RTK, CLAS, or SLAS is used for communication between the position acquisition unit and the artificial satellite. [Note 3] The information processing method described in Appendix 1 or Appendix 2, The location acquisition unit is an information processing method that acquires location data by communicating with a base station. [Note 4] The information processing method described in Appendix 3, An information processing method wherein the base station is configured to acquire base station coordinates corresponding to the location of the base station, the location acquisition unit acquires the location data based on the base station coordinates and distance information, and the distance information is based on the distance between the base station and the location acquisition unit. [Note 5] The information processing method described in Appendix 4, An information processing method in which GNSS, RTK, CLAS, or SLAS is used to obtain the base station coordinates. [Note 6] An information processing method described in any one of the appendices 1 to 5, The virtual space generation process is an information processing method that generates the virtual space from the image data using SfM processing. [Note 7] The information processing method described in Appendix 6, The SfM processing is an information processing method in which the processing is performed in photogrammetry, NeRF, or 3D Gaussian splatting. [Note 8] An information processing method described in any one of the appendices 1 to 7, The position acquisition unit acquires first position data by communication with an artificial satellite or base station, and acquires second position data by inertial positioning or SLAM, in this information processing method. [Note 9] The information processing method described in Appendix 8, An information processing method wherein the position data used in the allocation process can be switched between the first position data and the second position data depending on whether or not predetermined conditions regarding the accuracy of the first position data are met. [Note 10] The information processing method described in Appendix 8 or Appendix 9, The aforementioned assignment process is an information processing method that assigns the coordinates by combining the first position data and the second position data. [Note 11] An information processing method described in any one of the appendices 8 to 10, The SLAM is an information processing method that includes SLAM using LiDAR provided by the information processing terminal. [Note 12] An information processing method described in any one of the appendices 1 to 11, It also includes a coordinate map generation process, An information processing method in which, in the coordinate map generation process, the information processing terminal generates a coordinate map, and the coordinate map is a map in which the coordinates of the assignment process are assigned to the components of the virtual space. [Note 13] The information processing method described in Appendix 12, With further display processing capabilities, The display process is an information processing method in which the information processing terminal causes the information processing terminal to display the coordinate map on the display unit of the information processing terminal. [Note 14] An information processing method described in any one of the appendices 1 to 13, An information processing method wherein the coordinates in the aforementioned real space include at least one of latitude, longitude, and altitude. [Note 15] An information processing method described in any one of the appendices 1 to 14, In the image data acquisition process described above, the image data, which includes multiple images captured by the camera, is acquired. An information processing method wherein the camera is mounted on the information processing terminal, or is separate from the information processing terminal and connected to the information processing terminal to enable communication of the image data. [Note 16] The information processing method described in Appendix 15, An information processing method wherein the aforementioned multiple images were captured at each of multiple different imaging positions. [Note 17] The information processing method described in Appendix 16, The aforementioned information processing terminal is a portable mobile terminal designed to fit in one hand so that it can be held in one hand, and is suitable for use in the field. The image data includes a plurality of images captured by the camera while the user grasps the information processing terminal and moves between the plurality of imaging positions around the object. [Note 18] An information processing method described in any one of the appendices 1 to 17, The information processing terminal performs the virtual space generation process and the allocation process entirely within the information processing terminal without uploading the image data and location data to an external server. [Note 19] An information processing method described in any one of the appendices 15 to 18, The information processing terminal performs at least a portion of the virtual space generation process and the allocation process in parallel with capturing the plurality of images by the camera. [Note 20] An information processing method described in any one of the appendices 1 to 19, An information processing method that allows switching between a first information processing mode in which the information processing terminal performs all of the virtual space generation process and the allocation process, and a second information processing mode in which the information processing terminal has a server perform at least a part of the virtual space generation process and the allocation process, and obtains the processing results from the server. [Note 21] An information processing method described in any one of the appendices 1 to 20, The virtual space generation process and a portion of the allocation process are performed at the information processing terminal. Other parts of the virtual space generation process and the allocation process are executed on a server capable of communicating with the information processing terminal. An information processing method comprising: generating a point cloud as a component of the virtual space by executing a part of the information processing terminal; displaying the generated point cloud on the display unit of the information processing terminal; and uploading the calculation data related to the point cloud to the server. [Note 22] A computer program that causes a computer, which serves as an information processing terminal, to execute an information processing method described in any one of the appendices 1 to 21. [Note 23] A portable information processing terminal, The system comprises a location acquisition unit capable of acquiring location data representing the location of the information processing terminal in real space, an image acquisition unit, a virtual space generation unit, and an allocation unit. The image acquisition unit acquires image data including an image of the target, The virtual space generation unit generates a three-dimensional virtual space based on the image data, The assignment unit is an information processing terminal that assigns coordinates in real space to the components of the virtual space based on the position data acquired by the position acquisition unit. [Note 24] The information processing terminal described in Appendix 23, The position acquisition unit communicates with the satellite to acquire the position data, An information processing terminal in which GNSS, RTK, CLAS, or SLAS is used for communication between the position acquisition unit and the artificial satellite. [Note 25] An information processing terminal as described in Appendix 23 or Appendix 24, The aforementioned location acquisition unit is an information processing terminal that acquires location data by communicating with a base station. [Note 26] The information processing terminal described in Appendix 25, The base station is configured to acquire base station coordinates corresponding to the location of the base station, and the location acquisition unit acquires the location data based on the base station coordinates and distance information, wherein the distance information is based on the distance between the base station and the location acquisition unit, and this is an information processing terminal. [Note 27] The information processing terminal described in Appendix 26, An information processing terminal that uses GNSS, RTK, CLAS, or SLAS to acquire the base station coordinates. [Note 28] An information processing terminal described in any one of the appendices 23 to 27, The virtual space generation unit is an information processing terminal that generates the virtual space from the image data using SfM processing. [Note 29] The information processing terminal described in Appendix 28, The aforementioned SfM processing is a processing method in photogrammetry, NeRF, or 3D Gaussian splatting, and is an information processing terminal. [Note 30] An information processing terminal described in any one of the appendices 23 to 29, The position acquisition unit is an information processing terminal that acquires first position data by communication with an artificial satellite or base station, and acquires second position data by inertial positioning or SLAM. [Note 31] The information processing terminal described in Appendix 30, An information processing terminal in which the position data used in the allocation unit can be switched between the first position data and the second position data depending on whether or not predetermined conditions regarding the accuracy of the first position data are met. [Note 32] An information processing terminal as described in Appendix 30 or Appendix 31, The allocation unit is an information processing terminal that assigns the coordinates by combining the first position data and the second position data. [Note 33] An information processing terminal described in any one of the appendices 30 to 32, The aforementioned SLAM is an information processing terminal that includes SLAM using LiDAR provided by the information processing terminal. [Note 34] An information processing terminal described in any one of the appendices 23 to 33, It further includes a coordinate map generation unit, An information processing terminal wherein the coordinate map generation unit generates a coordinate map, and the coordinate map is a map in which the coordinates are assigned to the components of the virtual space by the assignment unit. [Note 35] The information processing terminal described in Appendix 34, It also includes a display unit, The aforementioned information processing terminal is an information processing terminal that causes the coordinate map to be displayed on the display unit. [Note 36] An information processing terminal described in any one of the appendices 23 to 35, The coordinates in the aforementioned real space include at least one of latitude, longitude, and altitude. [Note 37] An information processing terminal described in any one of the appendices 23 to 36, The image acquisition unit acquires the image data, which includes a plurality of images captured by the camera. The camera is mounted on the information processing terminal, or is separate from the information processing terminal and connected to the information processing terminal to enable communication of the image data. [Note 38] The information processing terminal described in Appendix 37, The aforementioned multiple images were captured at each of multiple different imaging positions in an information processing terminal. [Note 39] The information processing terminal described in Appendix 38, The aforementioned information processing terminal is a portable mobile terminal designed to fit in one hand so that it can be held in one hand, and is suitable for use in the field. The image data includes a plurality of images captured by the camera while the user grasps the information processing terminal and moves between the plurality of imaging positions around the object. [Note 40] An information processing terminal described in any one of the appendices 23 to 39, The aforementioned information processing terminal is an information processing terminal that completes the generation of the virtual space and the assignment of coordinates within the information processing terminal itself, without uploading the image data and the location data to an external server. [Note 41] An information processing terminal described in any one of the appendices 37 to 40, The information processing terminal performs at least a portion of the processing by the virtual space generation unit and the processing by the allocation unit in parallel with capturing the plurality of images by the camera. [Note 42] An information processing terminal described in any one of the appendices 23 to 41, The information processing terminal is configured to switch between a first information processing mode in which the information processing terminal performs all of the generation of the virtual space and the assignment of the coordinates, and a second information processing mode in which the server performs at least part of the generation of the virtual space and the assignment of the coordinates, and the processing results from the server are obtained. [Note 43] An information processing terminal described in any one of the appendices 23 to 42, The generation of the virtual space and the allocation of the coordinates are partially performed at the information processing terminal. The other parts of generating the virtual space and assigning the coordinates are performed on a server that can communicate with the information processing terminal. The information processing terminal generates a point cloud as a component of the virtual space by executing the part, displays the generated point cloud on the display unit of the information processing terminal, and uploads the calculation data related to the point cloud to the server. [Explanation of Symbols]

[0068] 1: Information processing terminal 2: Location acquisition device 2a: Enclosure 2b: Antenna 2c: Battery 2d: Main circuit 10: Communications Department 11: Storage section 12: Arithmetic Processing Unit 13: Touch panel 14: Camera 15: Communications bus 101: Image acquisition unit 101a: Image data 102: Virtual Space Generation Unit 102a: Virtual Space 103: Allocation Section 104: Coordinate map generation unit 104a: Coordinate Map 105: Location data K:Base station U: User X: Survey site Y: Survey target Z:Artificial satellite P1~P6: Imaging location

Claims

1. An information processing method using a portable information processing terminal, The aforementioned information processing terminal includes a position acquisition unit capable of acquiring position data representing the position of the information processing terminal in real space, It includes image data acquisition processing, virtual space generation processing, and allocation processing. In the aforementioned image data acquisition process, image data including an image of the target is acquired. In the virtual space generation process, a three-dimensional virtual space is generated based on the image data. The assignment process is an information processing method that assigns coordinates in real space to the components of the virtual space based on the position data acquired by the position acquisition unit.

2. The information processing method according to claim 1, The position acquisition unit communicates with the satellite to acquire the position data, An information processing method in which GNSS, RTK, CLAS, or SLAS is used for communication between the position acquisition unit and the artificial satellite.

3. The information processing method according to claim 1, The location acquisition unit is an information processing method that acquires location data by communicating with a base station.

4. The information processing method according to claim 3, An information processing method wherein the base station is configured to acquire base station coordinates corresponding to the location of the base station, the location acquisition unit acquires the location data based on the base station coordinates and distance information, and the distance information is based on the distance between the base station and the location acquisition unit.

5. The information processing method according to claim 4, An information processing method in which GNSS, RTK, CLAS, or SLAS is used to obtain the base station coordinates.

6. The information processing method according to claim 1, The virtual space generation process is an information processing method that generates the virtual space from the image data using SfM processing.

7. The information processing method according to claim 6, The SfM processing is an information processing method in which the processing is performed in photogrammetry, NeRF, or 3D Gaussian splatting.

8. The information processing method according to claim 1, The position acquisition unit acquires first position data by communication with an artificial satellite or base station, and acquires second position data by inertial positioning or SLAM, in this information processing method.

9. The information processing method according to claim 8, An information processing method wherein the position data used in the allocation process can be switched between the first position data and the second position data depending on whether or not predetermined conditions regarding the accuracy of the first position data are met.

10. The information processing method according to claim 8, The aforementioned assignment process is an information processing method that assigns the coordinates by combining the first position data and the second position data.

11. The information processing method according to claim 8, The SLAM is an information processing method that includes a LiDAR-based SLAM provided by the information processing terminal.

12. The information processing method according to claim 1, It also includes a coordinate map generation process, An information processing method in which, in the coordinate map generation process, the information processing terminal generates a coordinate map, and the coordinate map is a map in which the coordinates of the assignment process are assigned to the components of the virtual space.

13. The information processing method according to claim 12, With further display processing capabilities, The display process is an information processing method in which the information processing terminal causes the information processing terminal to display the coordinate map on the display unit of the information processing terminal.

14. The information processing method according to claim 1, An information processing method wherein the coordinates in the aforementioned real space include at least one of latitude, longitude, and altitude.

15. The information processing method according to claim 1, In the image data acquisition process described above, the image data, which includes multiple images captured by the camera, is acquired. An information processing method wherein the camera is mounted on the information processing terminal, or is separate from the information processing terminal and connected to the information processing terminal to enable communication of the image data.

16. The information processing method according to claim 15, An information processing method wherein the aforementioned multiple images were captured at each of multiple different imaging positions.

17. The information processing method according to claim 16, The aforementioned information processing terminal is a portable mobile terminal designed to fit in one hand so that it can be held in one hand, and is suitable for use in the field. The image data includes a plurality of images captured by the camera while the user grasps the information processing terminal and moves between the plurality of imaging positions around the object.

18. The information processing method according to claim 1, The information processing terminal performs the virtual space generation process and the allocation process entirely within the information processing terminal without uploading the image data and location data to an external server.

19. The information processing method according to claim 15, The information processing terminal performs at least a portion of the virtual space generation process and the allocation process in parallel with capturing the plurality of images by the camera.

20. The information processing method according to claim 1, An information processing method that allows switching between a first information processing mode in which the information processing terminal performs all of the virtual space generation process and the allocation process, and a second information processing mode in which the information processing terminal has a server perform at least a part of the virtual space generation process and the allocation process, and obtains the processing results from the server.

21. The information processing method according to claim 1, The virtual space generation process and a portion of the allocation process are performed at the information processing terminal. Other parts of the virtual space generation process and the allocation process are executed on a server capable of communicating with the information processing terminal. An information processing method comprising: generating a point cloud as a component of the virtual space by executing a part of the information processing terminal; displaying the generated point cloud on the display unit of the information processing terminal; and uploading the calculation data related to the point cloud to the server.

22. A computer program that causes a computer, which serves as an information processing terminal, to execute the information processing method described in any one of claims 1 to 21.

23. A portable information processing terminal, The system comprises a location acquisition unit capable of acquiring location data representing the location of the information processing terminal in real space, an image acquisition unit, a virtual space generation unit, and an allocation unit. The image acquisition unit acquires image data including an image of the target, The virtual space generation unit generates a three-dimensional virtual space based on the image data, The assignment unit is an information processing terminal that assigns coordinates in real space to the components of the virtual space based on the position data acquired by the position acquisition unit.

24. An information processing terminal according to claim 23, The position acquisition unit communicates with the satellite to acquire the position data, An information processing terminal in which GNSS, RTK, CLAS, or SLAS is used for communication between the position acquisition unit and the artificial satellite.

25. An information processing terminal according to claim 23, The aforementioned location acquisition unit is an information processing terminal that acquires location data by communicating with a base station.

26. An information processing terminal according to claim 25, The base station is configured to acquire base station coordinates corresponding to the location of the base station, and the location acquisition unit acquires the location data based on the base station coordinates and distance information, wherein the distance information is based on the distance between the base station and the location acquisition unit, and this is an information processing terminal.

27. An information processing terminal according to claim 26, An information processing terminal that uses GNSS, RTK, CLAS, or SLAS to acquire the base station coordinates.

28. An information processing terminal according to claim 23, The virtual space generation unit is an information processing terminal that generates the virtual space from the image data using SfM processing.

29. An information processing terminal according to claim 28, The aforementioned SfM processing is a processing method in photogrammetry, NeRF, or 3D Gaussian splatting, and is an information processing terminal.

30. An information processing terminal according to claim 23, The position acquisition unit is an information processing terminal that acquires first position data by communication with an artificial satellite or base station, and acquires second position data by inertial positioning or SLAM.

31. An information processing terminal according to claim 30, An information processing terminal in which the position data used in the allocation unit can be switched between the first position data and the second position data depending on whether or not predetermined conditions regarding the accuracy of the first position data are met.

32. An information processing terminal according to claim 30, The allocation unit is an information processing terminal that assigns the coordinates by combining the first position data and the second position data.

33. An information processing terminal according to claim 30, The SLAM is an information processing terminal that includes a LiDAR-based SLAM provided by the information processing terminal.

34. An information processing terminal according to claim 23, It further includes a coordinate map generation unit, An information processing terminal wherein the coordinate map generation unit generates a coordinate map, and the coordinate map is a map in which the coordinates are assigned to the components of the virtual space by the assignment unit.

35. An information processing terminal according to claim 34, It also includes a display unit, The aforementioned information processing terminal is an information processing terminal that causes the coordinate map to be displayed on the display unit.

36. An information processing terminal according to claim 23, The coordinates in the aforementioned real space include at least one of latitude, longitude, and altitude.

37. An information processing terminal according to claim 23, The image acquisition unit acquires the image data, which includes a plurality of images captured by the camera. The camera is mounted on the information processing terminal, or is separate from the information processing terminal and connected to the information processing terminal to enable communication of the image data.

38. An information processing terminal according to claim 37, The aforementioned multiple images were captured at each of multiple different imaging positions in an information processing terminal.

39. An information processing terminal according to claim 38, The aforementioned information processing terminal is a portable mobile terminal designed to fit in one hand so that it can be held in one hand, and is suitable for use in the field. The image data includes a plurality of images captured by the camera while the user grasps the information processing terminal and moves between the plurality of imaging positions around the object.

40. An information processing terminal according to claim 23, The aforementioned information processing terminal is an information processing terminal that completes the generation of the virtual space and the assignment of coordinates within the information processing terminal itself, without uploading the image data and the location data to an external server.

41. An information processing terminal according to claim 37, The information processing terminal performs at least a portion of the processing by the virtual space generation unit and the processing by the allocation unit in parallel with capturing the plurality of images by the camera.

42. An information processing terminal according to claim 23, The information processing terminal is configured to switch between a first information processing mode in which the information processing terminal performs all of the generation of the virtual space and the assignment of the coordinates, and a second information processing mode in which the server performs at least part of the generation of the virtual space and the assignment of the coordinates, and the processing results from the server are obtained.

43. An information processing terminal according to claim 23, The generation of the virtual space and the allocation of the coordinates are partially performed at the information processing terminal. The other parts of generating the virtual space and assigning the coordinates are performed on a server that can communicate with the information processing terminal. The information processing terminal generates a point cloud as a component of the virtual space by executing the part, displays the generated point cloud on the display unit of the information processing terminal, and uploads the calculation data related to the point cloud to the server.

Images