Information processing method and computer program

Abstract

The purpose is to provide effective support for subject positioning. [Solution] An information processing method executed by a portable information processing terminal includes display processing, positioning processing, and UI processing, in which the display processing involves displaying an image based on imaging data on a display screen and superimposing a UI image on the image based on the imaging data, the image based on the imaging data being a current status image or a processed image obtained by processing the current status image, and the UI image being used to point out a subject for which a positioning point is to be set, in which the positioning processing involves obtaining subject position data which is position data of the positioning point of the subject in response to an operation of setting the positioning point of the subject, and in which the UI processing involves changing the UI image in response to the positioning status in the positioning processing or the content of the image based on the imaging data in the display processing.

Description

[Technical field]

[0001] The present invention relates to an information processing method and a computer program executed on an information processing terminal. [Background technology]

[0002] Patent Document 1 describes an information processing device for displaying images of a building under construction. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6733127 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, a technology has been developed for performing object positioning on a terminal carried by a user. In such a technology, for example, an object captured by a camera is displayed on a display screen, and a positioning point is set on the object on the display screen by a user's operation, and the position of the positioning point in real space is obtained. Here, effective support is required to ensure accuracy and reduce the burden on the user when setting a desired positioning point.

[0005] There is also a need to understand whether the situation in the real world, such as a construction site, matches or differs from the expected state (design dimensions, etc.).

[0006] An object of a first invention of the present application is to effectively support object positioning.

[0007] A second object of the present invention is to conveniently compare a situation in real space with an assumed state. [Means for solving the problem]

[0008] According to a first invention, there is provided an information processing method executed by a portable information processing terminal, the information processing method including display processing, positioning processing, and UI processing, in which the display processing includes displaying an image based on imaging data on a display screen and superimposing a UI image on the image based on the imaging data, the image based on the imaging data being a current status image or a processed image obtained by processing the current status image, and the UI image being used to indicate a subject for which a positioning point is to be set, the positioning processing includes acquiring subject position data which is position data of the positioning point of the subject in response to an operation of setting the positioning point of the subject, and the UI processing includes changing the UI image in response to a positioning situation in the positioning processing or a content of an image based on the imaging data in the display processing.

[0009] According to the first aspect of the present invention, subject positioning can be effectively supported.

[0010] According to a second invention, there is provided an information processing method comprising an imaging object process, a virtual object process, and a comparison process, in which the imaging object process acquires an imaging object, the virtual object process acquires a virtual object, and the comparison process acquires a comparison result between the imaging object and the virtual object.

[0011] According to the second aspect of the present invention, the situation in real space can be conveniently compared with the assumed state. [Brief description of the drawings]

[0012] [Figure 1] 1 is an overall configuration diagram showing a user U carrying an information processing terminal 1 of an embodiment, and a positioning target Y at a site X (real space). [Diagram 2] FIG. 2A shows the rear side of information processing terminal 1 (terminal main body 2 to which position acquisition terminal 3 is attached), and FIG. 2B shows its front side. [Diagram 3] FIG. 2 is a functional block diagram showing a hardware configuration of an information processing terminal 1. [Figure 4]2 is a functional block diagram when a computer program according to an embodiment is executed by an information processing terminal 1. FIG. [Diagram 5] 1 is a process flow for explaining an information processing method and a computer program according to an embodiment. [Figure 6] 13 is an example of the display screen 13 when the user U is about to set a positioning point P1 on the subject (upper vertex A) in the embodiment. [Figure 7] Fig. 7A is a diagram showing positioning points P1 to P8 and a three-dimensional figure Qr (an example of an imaging object) in an augmented reality space W corresponding to the site X (real space) in Fig. 1. Fig. 7B is a diagram showing an example of a virtual object Qv. [Figure 8] Fig. 8A shows an example of a web screen 200, a map screen 201, and a subject image 202 with location information, all of which have been uploaded to the cloud. Fig. 8A is an enlarged view of part B enclosed by a dashed line in Fig. 8A. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiment can be combined with each other. Also, each characteristic feature can be an invention independently.

[0014] 1. Description of the configuration of the embodiment 1-1. Overall structure As shown in FIG. 1, in the embodiment, as an example, a user U measures the position of a positioning target Y using a portable information processing terminal 1 at a site X (real space). The information processing terminal 1 is a portable terminal. In the embodiment, a position acquisition terminal 3 described later in the information processing terminal 1 is capable of communicating with at least one of an artificial satellite Z and a base station K. That is, the position acquisition terminal 3 may be configured to be capable of communicating with both the artificial satellite Z and the base station K, or may be configured to be capable of communicating with only one of the artificial satellite Z and the base station K. In addition, in the embodiment, as an example, the positioning target Y is a hexahedral concrete structure, but this is merely an example, and various objects having any size, shape, structure, etc. can be adopted as the positioning target Y.

[0015] 1-2.Hardware configuration 1-2-1.Appearance 2A and 2B show an example of an information processing terminal 1 (a terminal body 2 to which a position acquisition terminal 3 is attached). The information processing terminal 1 includes a terminal body 2 and a position acquisition terminal 3. In the embodiment, the terminal body 2 is a smartphone as an example. The terminal body 2 is attached with the position acquisition terminal 3. In the embodiment, as an example, the position acquisition terminal 3 can be attached to the back surface of the terminal body 2. And, it is preferable that the position acquisition terminal 3 is configured to be detachably connected to the terminal body 2, for example. The planar dimensions of the position acquisition terminal 3 are defined so that the parts other than the antenna 2b are substantially contained within the back surface of the terminal body 2. In the embodiment, the information processing terminal 1 is configured with the terminal body 2, which is responsible for functions such as calculations related to surveying, and the position acquisition terminal 3, which is responsible for the function of acquiring position information, being separate (independent), and for example, the position acquisition terminal 3 can be attached to a generally used smartphone (terminal body 2) by a simple connection operation to increase the functionality of the terminal, resulting in a configuration with improved convenience, versatility, and usability, but is not limited to this. The information processing terminal 1 may be a terminal in which the terminal body 2 and the position acquisition terminal 3 are integrated.

[0016] In the position acquisition terminal 3 of the embodiment, various circuit configurations are housed in a plate-shaped housing 3a. In the embodiment, as shown in FIG. 2A, an antenna 3b is connected to the top of the housing 3a, and a battery 3c is provided inside the housing 2a, and these are connected to a main circuit 3d that realizes a position acquisition function and the like. Although an embodiment in which the position acquisition terminal 3 has the battery 3c is described as an example, the present invention is not limited to this, and the position acquisition terminal 3 may not have the battery 3c and may be configured to receive power from another terminal (for example, the terminal body 2) or from an external power source.

[0017] 1-2-2.Hardware configuration functional blocks As shown in FIG. 3, the terminal main body 2 has a communication unit 10, a memory unit 11, a control unit 12, a display screen 13, a camera 14, and a LiDAR scanner 15, and these components are electrically connected inside the terminal main body 2 via a communication bus 16.

[0018] The communication unit 10 may employ, for example, a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc. The communication unit 10 may employ a configuration in which it is connected to the communication network 6 via a wireless communication means such as wireless LAN network communication, mobile communication such as 3G / LTE / 5G, Bluetooth (registered trademark) communication, etc. The communication unit 10 may also employ a configuration in which the above-mentioned wired communication means and wireless communication means are used in combination.

[0019] The control unit 12 is configured to execute processing and control related to information processing of the information processing terminal 1. The control unit 12 can be configured, for example, with a central processing unit (CPU), and in the embodiment, the control unit 12 is an example of a processor capable of executing programs related to each step of a flowchart described below. The control unit 12 realizes various functions related to the information processing terminal 1, for example, by reading out a program stored in the storage unit 11. Furthermore, the information processing of the software in the information processing terminal 1 is realized, for example, by the control unit 12 as hardware processing various programs stored in the storage unit 11.

[0020] In the embodiment, the display screen 13 is a touch panel. The touch panel is configured by integrating an input unit and a display unit. The display screen 13 is an input device that accepts an operation input from the user U, and also has a function of a display device that displays various screens. The use of a touch panel is one example, and is not limited to this. The input unit that accepts the operation input from the user U and the display unit that displays images, etc. may be configured separately. In this case, the input unit may be configured to accept input from, for example, an instruction unit (e.g., a button, a lever, etc.) provided on the information processing terminal 1, or may be configured to accept voice instructions (input) from a microphone. The display unit may be configured to employ various display devices, such as a liquid crystal display, an organic EL display, a plasma display, etc.

[0021] As shown in FIG. 2A, the terminal body 2 of the embodiment has three different cameras 14 and a LiDAR scanner 15 arranged together in one location on the back surface of the terminal body 2. Of the three cameras 14, one may be a wide-angle camera, the other may be an ultra-wide-angle camera, and the other may be a telephoto camera. The user U may use these cameras to capture the positioning target Y. The configuration of the camera 14 described here is merely an example, and is not limited to this.

[0022] 3, the position acquisition terminal 3 can communicate with the storage unit 11 and the like in the terminal body 2 via the communication unit 10 and the communication bus 16. There is no limitation on the type of connection between the position acquisition terminal 3 and the communication unit 10, and any wireless connection can be adopted. Specifically, for example, Bluetooth (registered trademark) or Wi-Fi can be adopted. In addition, it is not limited to a wireless connection, and any wired connection (for example, USB-TypeC, etc.) may be adopted.

[0023] 1-3. Software configuration (functional block diagram) As shown in the functional blocks of FIG. 4, the control unit 12 of the terminal body 2 includes a display unit 101, a positioning unit 102, a UI unit 103, an acquisition unit 104, an imaging object unit 105, a virtual object unit 106, and a comparison unit 107.

[0024] The display unit 101 executes various image processes related to the display of the display screen 13. In Fig. 4, "image 101a based on imaging data", "UI image 101b", and "UI operation image 101c" are illustrated as part of the images displayed by the display unit 101 in the embodiment. Details of each image will be described later.

[0025] The positioning unit 102 executes processing related to the positioning function of the information processing terminal 1. The positioning unit 102 can execute processing to set a positioning point P and processing to obtain subject position data Dy. The subject position data Dy is position data indicating the position of the positioning point P. In the embodiment, specific examples of the positioning point P include positioning points P1 to P8 in Fig. 7A and positioning point P9 in Figs. 8A and 8B. Specific examples of each process will be described later.

[0026] The method of implementing each functional unit in FIG. 4 by the control unit 12 is not particularly limited. In the embodiment, the terminal body 2 is a smartphone as an example. For this reason, as an example, an application program is downloaded from an external server to the terminal body 2, installed in the terminal body 2, and the control unit 12 executes the application program to implement each functional unit in FIG. 4. However, the present invention is not limited to this, and as another example, the program may be provided by being stored in a non-transitory computer-readable recording medium in advance. In addition, each functional unit in FIG. 4 is not limited to being implemented by software, but may be implemented by hardware. In the embodiment, various information and concepts that encompass this are handled, but these are represented by high and low signal values ​​or quantum bits as a binary bit collection consisting of 0 or 1, and communication and calculation can be performed by the above-mentioned software or hardware aspects.

[0027] 4, the position acquisition terminal 3 is connected to each functional unit of the terminal body 2 so as to be able to communicate with each other. The terminal position data Dt that can be acquired by the position acquisition terminal 3 may include, for example, latitude, longitude, and altitude, and may also include, for example, geoid height in addition to these. Here, the case where the terminal position data Dt uses a coordinate system such as latitude and longitude (geographic coordinate system) is described as an example, but the present invention is not limited to this, and a planar Cartesian coordinate system may also be adopted.

[0028] 1-4. Acquisition of various location data 4, the information processing terminal 1 acquires terminal position data Dt, and specifically, in the embodiment, as an example, the position acquisition terminal 3 acquires the terminal position data Dt. The terminal position data Dt is position data of the information processing terminal 1, and subject position data Dy is acquired based on this terminal position data Dt.

[0029] The terminal position data Dt can be acquired by communication with the artificial satellite Z, communication with the base station K, or SLAM (Simultaneous Localization and Mapping), and any combination of two or more of these three techniques may be used. In this case, RTK (Real Time Kinematic) may be adopted for communication between the position acquisition terminal 3 and the artificial satellite Z.

[0030] Specifically, as an example, the position acquisition terminal 3 may communicate with the artificial satellite Z (the communication method may be, for example, RTK, GNSS, CLAS, or SLAS) to acquire the terminal position data Dt. Note that GNSS is an abbreviation for Global Navigation Satellite System, CLAS is an abbreviation for Centimeter-Level Augmentation Service, and SLAS is an abbreviation for Submeter-Level Augmentation Service. As another example, a base station K capable of communicating with the position acquisition terminal 3 may be installed at the surveying site, and the position acquisition terminal 3 may acquire the terminal position data Dt from this base station K. In this case, the base station K is configured to be able to acquire base station coordinates corresponding to the position of the base station. Then, the position acquisition terminal 3 can acquire the terminal position data Dt based on the base station coordinates and distance information.

[0031] The position acquisition terminal 3 can acquire the base station coordinates from the base station K. The position acquisition terminal 3 may calculate and acquire the distance information by communicating with the base station K. Alternatively, the base station K may calculate and acquire the distance information by communicating with the position acquisition terminal 3, and the base station K may transmit the distance information to the position acquisition terminal 3, thereby causing the position acquisition terminal 3 to acquire the distance information.

[0032] Here, the distance information is based on the distance between the base station K and the position acquisition terminal 3. This distance information may be, for example, information that represents the distance itself between the base station K and the position acquisition terminal 3. Alternatively, it may be, for example, information that indicates the relative positional relationship between the base station K and the position acquisition terminal 3. This positional relationship can be grasped as, for example, the relative position (X, Y, Z) of the position acquisition terminal 3 with respect to the base station K.

[0033] In addition, when base station K acquires base station coordinates, it can adopt a communication method such as RTK, GNSS, CLAS, or SLAS.

[0034] To acquire the terminal position data Dt, for example, the coordinates of inertial positioning using an inertial sensor may be adopted, or the coordinates using SLAM may be adopted. For example, an acceleration sensor or a gyro sensor built into the terminal body 2 may be adopted as the inertial sensor used in the inertial positioning. For SLAM, the SLAM may be SLAM using the camera 14, or the SLAM using the LiDAR scanner 15 may be adopted. In SLAM, for example, the above-mentioned camera 14 or the LiDAR scanner 15 may be combined with the above-mentioned inertial sensor to acquire the coordinates.

[0035] In the embodiment, as an example, the information processing terminal 1 acquires relative position data Dr in addition to the terminal position data Dt. The relative position data Dr is data based on a relative positional relationship between the information processing terminal 1 and the subject. In the embodiment, as an example, the relative position data Dr includes a relative position and a direction. In the embodiment, as an example, the information processing terminal 1 acquires subject position data Dy based on the terminal position data Dt and the relative position data Dr.

[0036] In the embodiment, as an example, the relative position data Dr is acquired by performing a LiDAR scan using the LiDAR scanner 15. As an example, the information processing terminal 1 (terminal body 2) calculates the coordinates of each point of the point cloud by the LiDAR scan based on the terminal position data Dt, the relative position, and the orientation. As an example, the information processing terminal 1 (terminal body 2) acquires the subject position data Dy based on the coordinates of a specific point in the point cloud. Note that the relative position data Dr may be acquired using any distance measuring sensor, not limited to the LiDAR scanner 15.

[0037] The subject position data Dy obtained by the various methods described above may be stored in the memory unit 11 of the information processing terminal 1, may be displayed on the display screen 13, may be used in other processes, or may be transferred to the outside of the information processing terminal 1 via the communication unit 10, etc.

[0038] 2. Description of the Operation of the Embodiment The operation of the embodiment will be described with reference to FIGS. 5 to 7B.

[0039] 2-1. Display process S1 In the processing flow of FIG. 5, first, a display process S1 is executed, and thereby the display screen 13 illustrated in FIG. 6 is presented. FIG. 6 is an example of the display screen 13 when the user U is about to set a positioning point in the embodiment. In the display process S1 of the embodiment, the display unit 101 displays an "image 101a based on imaging data" on the display screen 13, and also superimposes a UI image 101b on the "image 101a based on imaging data." Furthermore, as an example in the embodiment, a UI operation image 101c (UI operation buttons 101c1 to 101c8) is also superimposed on the display screen 13 (see FIG. 6).

[0040] In FIG. 6, the UI operation button 101c1 is a button for setting a positioning point P. The UI operation button 101c2 is a button for opening a setting menu for various setting items. The UI operation button 101c3 is a mode switching button. The UI operation button 101c4 is a button for canceling the operation for setting a positioning point. The UI operation button 101c5 is a button for redoing the canceled operation for setting a positioning point. The UI operation button 101c6 is a button for creating a new solid object composed of positioning points. When this button is pressed, the positioning points set thereafter are treated as vertices of the new solid object. The UI operation button 101c7 is a button for hiding various UI operation buttons. The UI operation button 101c8 is a button for opening a menu for selecting a 3D model or a positioning point to be displayed on the screen. The screen in FIG. 6 is an example and is not limited thereto, and for example, some of the above operation buttons may be omitted.

[0041] The "image 101a based on the imaging data" may be a current state image or a processed image. The current state image is an image captured by the camera 14, and in this embodiment is an image of the site X. The processed image is an image obtained by processing the current state image using any processing process. The image 101a based on the imaging data is updated in real time by the user U pointing the camera 14 in any direction.

[0042] The UI image 101b is used to indicate a subject for which a positioning point P is to be set. In the embodiment, as an example, the UI image 101b includes an indication figure 101b1 and an indication point 101b2. The indication figure 101b1 extends from a predetermined position toward the indication point 101b2 within the display screen 13. In the embodiment, as shown in FIG. 6, two straight lines (indication figures 101b1) extend from the center of each of the vertical sides of the display screen 13 toward the indication point 101b2. Each straight line becomes thinner as it approaches the indication point 101b2. That is, as shown in FIG. 6, each straight line (indication figure 101b1) is thinner at the tip side (rear side) corresponding to the indication point 101b2 than at the front side. In other words, each straight line is tapered, which allows the user to grasp a sense of perspective and is highly convenient.

[0043] As a modified example, the indication figure 101b1 may move from a predetermined position toward the indication point 101b2. For example, an image may be used in which a straight line with a river-like flow toward the indication point 102b2 is dynamically presented. Alternatively, the indication figure 101b1 may be replaced with another non-linear figure (e.g., a circle, a sphere, a triangle, a pin, etc.), and this sphere image or the like may move toward the indication point 102b2. In this way, the indication figure 101b1 is not limited to a straight line, and any figure may be used.

[0044] The indication point 101b2 indicates a position for setting the positioning point P. When the user U presses the UI operation button 101c1 while the indication point 101b2 overlaps with the subject (a specific part of the subject), the position of the subject (a specific part of the subject) in the site X (real space) is acquired, as described in the next positioning process S2.

[0045] 2-2. Positioning process S2 In the positioning process S2 of the embodiment, the positioning unit 102 sets a positioning point P and acquires subject position data Dy in response to a user operation (pressing the UI operation button 101c1).

[0046] Please refer to Fig. 6 for a specific example. While viewing the display screen 13 in Fig. 6, the user U adjusts the direction of the camera 14 to align the indication point 101b2 of the UI image 101b with the desired subject (upper apex A in the example of Fig. 6). In this state, the user U performs an operation (pressing the UI operation button 101c1) to set a positioning point P on the subject (upper apex A). In the example of Fig. 6, the indication point 101b2 overlaps with the upper apex A, and when the UI operation button 101c1 is pressed in this state, the positioning point P1 is set on the subject (upper apex A), and subject position data Dy of the positioning point P1 is acquired.

[0047] Please refer to Fig. 7A. In the augmented reality space W of Fig. 7A, a positioning point P1 is set at a position corresponding to the upper apex A of the real space (site X) of Fig. 1. Here, for convenience of explanation, the positioning point P1 is illustrated in the augmented reality space W of Fig. 7A but not in Fig. 1, but the positioning point P1 has position information of the real space based on the subject position data Dy.

[0048] 6, as an example, in the embodiment, when the user U sets the positioning point P1 at the upper vertex A, the display unit 101 superimposes an image corresponding to the positioning point P1 (positioning point image) on the display screen 13. As a result, the positioning point image is AR-displayed (augmented reality display) in the augmented reality space W, and the user U can visually recognize the positioning point P1 on the display screen 13. The positioning point image can be any image having any shape and color, and may be, for example, any point image or stake image.

[0049] In the embodiment, for example, when an operation to set a positioning point P in Fig. 6 (pressing the UI operation button 101c1) is performed, an image 101a based on the imaging data at that moment is also captured, and a "subject image with location information" in which the captured image and the positioning point image are superimposed is obtained. This subject image with location information can be saved, displayed, used in other processes, or transferred. A specific example of a subject image with location information will be described later with reference to Figs. 8A and 8B.

[0050] 7A also illustrates other positioning points P2 to P8. The user U can capture the other peaks of the positioning target Y with the camera 14 and set each of the positioning points P2 to P8 on the display screen 13 in FIG. 6 with each peak as a subject.

[0051] 2-3. UI processing S3 In the embodiment, the positioning process S2 and the UI process S3 are executed in parallel. In the UI process S3, the UI image 101b is changed as follows, thereby making it possible to provide improved convenience for the user U when performing positioning.

[0052] In the UI processing S3 of the embodiment, as described below, it is preferable that the UI unit 103 changes the UI image 101b (see FIG. 6) according to the “positioning situation in the positioning unit 102” or the “contents of the image 101a based on the imaging data.” In this UI process S3, at least one of the processes described in the following sections 2-3-1, 2-3-2, and 2-3-3 is executed. That is, one of the processes in the sections 2-3-1, 2-3-2, and 2-3-3 may be executed, two of these processes may be executed, or all of these processes may be executed.

[0053] 2-3-1. UI processing related to positioning status in the positioning unit 102 Specifically, the "positioning situation" may be, for example, "whether the distance to the subject can be measured" or "the positioning accuracy of the subject." The "ability to measure distance" may be determined based on, for example, the distance measured by the LiDAR scanner 15, or the presence or absence or degree of diffuse reflection (for example, from the surface of water, etc.). The "positioning accuracy" may include, for example, the positioning accuracy by LiDAR or the positioning accuracy associated with satellite reception sensitivity. The "positioning status" may include, for example, the "reliability of positioning" and the like, and may include, as an example, the reflection intensity of the LiDAR scanner 15 and the like. Note that "positioning accuracy" indicates, for example, how accurate the position obtained by positioning is with respect to the actual position (the degree of expected error), and "positioning reliability" indicates the degree to which the positioning result is kept stable with respect to factors such as the reflection conditions for the measurement object, external noise, weather, etc. The method by which the control unit 12 calculates the value of the positioning accuracy (the degree of error and the degree of stability of the positioning result) is not particularly limited, and any known method can be adopted.

[0054] For example, in a first case where the positioning situation is one where distance measurement is possible, the UI image 101b may be in a first mode, and in a second case where distance measurement is impossible, the UI image 101b may be in a second mode. The modes described here are different modes that can be distinguished from each other, and one example is color. For example, when the mode is color, a configuration can be adopted in which the color of the UI image 101b is, for example, green in the first case and gray in the second case. For example, in a first case where the positioning accuracy is equal to or greater than a first threshold accuracy, the UI image 101b may be in a first mode, in a second case where the positioning accuracy is equal to or less than a second threshold accuracy that is lower than the first threshold accuracy, the UI image 101b may be in a second mode, and in a third case where the positioning accuracy is smaller than the first threshold accuracy and larger than the second threshold accuracy, the UI image 101b may be in a third mode. The modes described here are different modes so that they can be distinguished from each other, and one example is color. For example, when the mode is color, a configuration can be adopted in which the color of the UI image 101b is, for example, green in the first case, gray in the second case, and yellow in the third case. Note that, in the above, the mode in which the positioning accuracy is divided into three cases, the first case to the third case, has been described, but of course, the present invention is not limited to this, and may be two cases (i.e., one threshold) or four or more cases (i.e., three or more thresholds).

[0055] In the above, the manner is changed from the viewpoint of distance measurement capability and positioning accuracy. Similarly, the manner exemplified above may be changed depending on the level of positioning reliability, etc.

[0056] In the above description, the aspect is described as color, so the color of the UI image 101b is changed, but this is merely an example. In the embodiment, any type of aspect change that can be presented to the user U on the display screen 13 can be adopted. In the embodiment, as an example, at least one of the following change aspects (f) may be changed. Changes (f): Size, thickness, length, movement speed, quantity, shape, display presence / absence, blinking presence / absence, blinking time interval, rotation angle, operation, color and transparency of UI image 101b, and sound and vibration associated with UI image 101b

[0057] 2-3-2. UI processing related to the contents of the image 101a based on the imaging data The "contents of the image 101a based on the imaging data" may specifically include, for example, the "state of the subject." The "state of the subject" may include the presence or absence or the degree of diffuse reflection by the LiDAR scanner 15. Alternatively, the "state of the subject" may be, for example, when the subject (concrete, etc.) has a crack or the like, the "point where the crack or the like is present" may be automatically detected by, for example, AI (artificial intelligence) to change the UI image 101b according to the detection result, and the coordinates of the point may be measured. As for the "contents of the image 101a based on the captured image data," any one or more of the above-mentioned variations (f) can be adopted for the UI image 101b.

[0058] 2-3-3. Processing according to the distance to the subject (i) and processing to guide the user to the subject (ii) Furthermore, the UI process S3 in the embodiment may include the following process (i) or process (ii). It is more preferable to execute both processes (i) and (ii).

[0059] In the process (i), a process of changing the UI image 101b according to the distance between the subject and the information processing terminal 1 may be executed. The distance can be acquired, for example, from the above-mentioned relative position data Dr. In this process, for example, the indication figure 101b1 (UI image 101b) of the UI image 101b may be made thicker (larger) as the user U holding the information processing terminal 1 approaches the subject (strictly speaking, the part overlapping with the indication point 101b2 on the display screen 13). Any one or more of the above-mentioned change modes (f) may be adopted for the UI image 101b. Regarding the process (i), a numerical value corresponding to the distance between the information processing terminal 1 and the subject may be displayed. This numerical value may be the distance itself (absolute value) or a relative value. The numerical value corresponding to the distance may change so that it becomes smaller as the user U holding the information processing terminal 1 gets closer to the subject, and the numerical value corresponding to the distance may change so that it becomes larger as the user U moves away from the subject. The numerical value corresponding to the distance described here may be displayed, for example, above the indication figure 101b1 or indication point 101b2 of the UI image 101b.

[0060] In the process (ii), a process may be executed in which a UI image (which may be the UI image 101b, or another UI image may be added) is changed so as to guide the information processing terminal 1 to a predetermined subject position. The guide direction may be determined, for example, by using the orientation of the relative position data Dr described above. In this process, for example, any one or more of the above-described change modes (f) may be adopted in the UI image 101b so as to present the positional relationship between the information processing terminal 1 and the predetermined subject position. For example, the blinking time interval of the UI image 101b may be shortened as the information processing terminal 1 approaches the subject position, or any navigation UI image (e.g., an arrow image, a triangle, a pin, etc.) indicating the guide direction to the subject position may be added.

[0061] In the embodiment, as an example, in the UI process S3, the UI unit 103 may change the UI image 101b stepwise or continuously in response to a change in the relative positional relationship between the information processing terminal 1 and the subject. In the embodiment, for example, any one or more of the above change modes (f) can be changed stepwise or continuously in combination with the above process (i) or process (ii).

[0062] 2-4. Acquisition process S4 Through the above processes S1 to S3, the user U can set a positioning point P for an arbitrary subject. In the embodiment, as an example, the set positioning point P (corresponding to positioning points P1 to P8 in FIG. 7A) is used to execute the acquisition process S4. In the acquisition process S4, the acquisition unit 104 of the information processing terminal 1 acquires the following (a) to (e) (specifically, automatic calculation process, etc.) using the subject position data Dy. Note that, in the embodiment, it is not necessary for all of the following (a) to (e) to be acquired in the acquisition process S4, and it is preferable that at least one of them is acquired. In the embodiment, it is assumed that all of them are acquired.

[0063] (a) Coordinates of each positioning point P (in the example of FIG. 7A, the coordinates of each positioning point P1 to P8 in real space (site X in FIG. 1)) (b) The distance between each of the positioning points P whose coordinates have been acquired (for example, the distance L1 between the positioning points P3 and P4 in the example of FIG. 7A ) (c) The area of ​​a plane figure having vertices each of the positioning points P whose coordinates have been acquired (in the example of FIG. 7A, for example, the area of ​​a quadrangle Mr consisting of the positioning points P3, P4, P7, and P8) (d) The volume or capacity of a solid figure having vertices at each of the positioning points P whose coordinates have been acquired (in the example of FIG. 7A, in the case of a solid figure Qr having vertices at positioning points P1 to P8, if the solid figure Qr is a tangible object, the volume is calculated; if the solid figure Qr is a hollow object, the volume is calculated). (e) the direction, angle, or gradient between each of the positioning points P whose coordinates were obtained; The solid figure Qr corresponds to an object to be imaged at the measurement site, and also corresponds to an imaged object, which will be described later. In addition, for the above-mentioned distance between points, the components related to distance, such as horizontal distance and elevation difference, may be displayed separately, or conversely, only the distance between points may be displayed without displaying them separately. Also, at least one of the components of distance, such as horizontal distance and elevation difference, may be displayed.

[0064] In the embodiment, as an example, an augmented reality display (AR display) is adopted in relation to the above-mentioned acquisition process S4. Specifically, for example, the display unit 101 executes the display process S1, and any image representing each of the components (positioning points, straight lines, figures, orientations, etc.) described above (a) to (e) may be superimposed (AR display) on the display screen 13 in response to a user request operation or automatically. For example, when any number of positioning points P are set, a line image connecting them, a two-dimensional figure image connecting them, or a three-dimensional figure image having them as vertices may be superimposed and displayed on the display screen 13. Each of these images may be accompanied by the numerical values ​​described above (a) to (e).

[0065] As an example, in Fig. 7A, a three-dimensional figure Qr having vertices at positioning points P1 to P8 is illustrated by a dashed line and is superimposed on the positioning target Y. Note that in Fig. 7A, the three-dimensional figure Qr and the positioning target Y are illustrated slightly shifted from each other to make them distinguishable, but this is for convenience. Ideally, it is preferable that the shape and size of the three-dimensional figure Qr completely match those of the positioning target Y.

[0066] 2-5. Image capture object processing S5 Next, in the imaging object process S5 of the embodiment, the imaging object unit 105 acquires an "imaging object." The imaging object is an object based on imaging data. The object is any digital element displayed in a digital space.

[0067] In the embodiment, as an example, a three-dimensional model representing a solid figure Qr shown in Fig. 7A is adopted as an imaging object of the imaging object unit 105. The three-dimensional model of the solid figure Qr is an example of an imaging object because the solid figure Qr is configured by setting each of the positioning points P1 to P8 in an image 101a based on imaging data.

[0068] There is no limitation on the method of acquiring the imaged object, and other acquisition methods may be used. Any object (two-dimensional object or three-dimensional object) may be generated by any means, and this arbitrary object may be adopted as the imaged object. The imaged object may be generated from a point cloud or the like, and examples of the method include three-dimensional scanning processing of the LiDAR scanner 15, SfM (Structure from Motion) processing, three-dimensional reconstruction processing, photogrammetry processing, photogrammetry processing, NeRF processing, and 3D Gaussian splatting processing. The object may be surface data or a three-dimensional model, and may be a mesh or a solid.

[0069] The information processing terminal 1 may acquire the imaging object in the imaging object unit 105 by using a three-dimensional scanning process of the information processing terminal 1 (processing using the LiDAR scanner 15).

[0070] 2-6. Virtual object processing S6 In the virtual object processing S6 of the embodiment, the virtual object unit 106 acquires a virtual object Qv (see FIG. 7B). In the information processing terminal 1 of the embodiment, the virtual object unit 106 generates a virtual object based on object data or a virtual figure set in the virtual space by the user.

[0071] The "object data" may be any digital data, and may be, for example, any two-dimensional or three-dimensional model data, or may be, for example, design data (for example, two-dimensional or three-dimensional CAD data, etc.). The terminal body 2 may read the object data from, for example, a predetermined cloud server (storage), etc. As an example, the virtual object Qv in FIG. 7B is based on the design data of the positioning target Y (for example, a concrete structure).

[0072] The "virtual figure" is any figure set by the user U. The display unit 101 may display, for example, a "setting screen for virtual figure" on the display screen 13. In this setting screen, the user U may input the virtual figure in the form of any line, surface, or solid, and may superimpose the virtual figure on the captured object. In the embodiment, it may be possible to "cut up an object by designating any number of points," and in this case, the lines and points used at the time of cutting are also included in the virtual figure.

[0073] 2-7. Comparison process S7 In the comparison process S7 of the embodiment, the comparison unit 107 acquires a comparison result between the imaging object (a solid figure Qr as an example) and the virtual object Qv. The "comparison result" may be, for example, an overlap of a plurality of objects, or a difference between a plurality of objects. The display unit 101 may display, for example, a "comparison result presentation screen" on the display screen 13. Any display method may be adopted for the difference display, and for example, the result of a Boolean operation (e.g., subtraction) of the objects may be presented, or, for example, the objects may be combined (unioned) and the non-overlapping difference parts may be visually presented (e.g., color-coded or flashing). In addition, the comparison result is not limited to a visual display in a digital space, and may be presented on the display screen 13 as character information, numerical values, or the like. The acquired comparison result may be stored, displayed, used in other processes, or transferred, or the like.

[0074] In the embodiment, as an example, before acquiring a comparison result, the comparison unit 107 aligns the positions of the imaging object (for example, a three-dimensional figure Qr) and the virtual object Qv. The digital coordinate system and the coordinate system (real coordinate system, geographic coordinate system) of the real space X are associated with each other to accurately match the position of the imaging object to the virtual object Qv.

[0075] Then, the current processing flow ends.

[0076] 3. Description of the Functions and Effects of the Embodiments As described above, according to the embodiment, when the user U performs subject positioning, the UI process S3 changes the UI image 101b, thereby making it possible to effectively assist the user U.

[0077] Furthermore, according to the embodiment, as described above in "1-4. Acquisition of Various Position Data", there is an advantage that various position data required for subject positioning can be appropriately acquired by making full use of various positioning technologies and the like.

[0078] Furthermore, according to the embodiment, as described in the acquisition process S4, various pieces of information (a) to (e) can be acquired from the subject position data Dy, and convenience for the user U at the site X is improved.

[0079] Also, according to the embodiment, as described in the UI processing S3 and the like, various improvements are made to the UI image 101b. According to the embodiment, since the UI image 101b includes the instruction figure 101b1, it is possible to provide the user U with good operability. Also, in the embodiment, it is possible to impart higher visibility, etc. to the change in the UI image 101b by adopting various change modes listed in (f) above or by imparting a gradual or continuous change. Also, in the embodiment, in the UI processing S3, it is possible to change the UI image 101b according to various conditions, etc. that are effective in supporting the user U (whether distance measurement is possible, positioning accuracy, the state of the subject, the distance between the subject and the terminal, and guidance to the subject position).

[0080] According to the embodiment, the objects can be compared with each other by the processes described in the imaging object process S5 to the comparison process S7. This makes it possible to compare the state of the site X (e.g., the three-dimensional figure Qr based on the positioning target Y in FIG. 7A) with an arbitrary virtual object Qv (e.g., design data or an arbitrary virtual figure, etc.).

[0081] 4. Variations 4-1. Switching location data acquisition devices The terminal body 2 can also acquire the terminal position data. The terminal position data of the information processing terminal 1 is not limited to the terminal position data Dt of the position acquisition terminal 3 (see FIG. 2), but may be the terminal position data of the terminal body 2.

[0082] Depending on the accuracy of the terminal position data Dt of the position acquisition terminal 3 (the accuracy of RTK in the embodiment), it may be possible to switch from the terminal position data Dt acquired by the position acquisition terminal 3 to the terminal position data acquired by the terminal main body 2, and conversely, it may be possible to switch from the terminal position data acquired by the terminal main body 2 to the terminal position data Dt acquired by the position acquisition terminal 3. Specifically, when a predetermined condition regarding the accuracy of the terminal position data of the position acquisition terminal 3 is satisfied (in this example, when the accuracy of RTK can be ensured), the terminal position data Dt acquired by the position acquisition terminal 3 is used as the terminal position data of the information processing terminal 1. Also, when a predetermined condition regarding the accuracy of the position data of the position acquisition terminal 3 is not satisfied (in this example, when the accuracy of RTK cannot be ensured), the information processing terminal acquired coordinates acquired by the terminal main body 2 may be used as the position data. Depending on the accuracy of the terminal position data Dt of the position acquisition terminal 3, the terminal position data acquired by the terminal body 2 and the terminal position data Dt acquired by the position acquisition terminal 3 may be combined.

[0083] In the above, as an example, the acquisition device (terminal body 2 and position acquisition terminal 3) of the terminal location data is switched or the terminal location data is combined depending on the accuracy, but the present invention is not limited to this. For example, the acquisition device of the terminal location data may be switched or the terminal location data may be combined depending on various conditions such as the processing speed of the terminal body 2, the remaining amount of the battery 2c, the balance between these, the user's request, etc. In other words, the acquisition device of the terminal location data may be switched or the terminal location data may be combined depending on a predetermined condition, not limited to the accuracy of the terminal location data.

[0084] 4-2. Upload function to web cloud, etc. A function may be provided to record the photograph (image 101a based on the imaging data) of the site X and the subject position (subject position data Dy) and display them on the display screen 13 by the application of the information processing terminal 1, and to synchronize the information with the server web cloud and view it on the web screen 200. The web screen 200 can be viewed on any terminal (including a personal computer, tablet, etc.) that can connect to the server web cloud. That is, the information processing method according to the embodiment may further include an acquired information display process. The acquired information display process is a process of acquiring information from a storage unit such as a server web cloud, and displaying the acquired information on the information processing terminal 1. The acquired information display process can be executed, for example, independently of and in parallel with the process of the control flow described in FIG. 5. Here, the acquired information display process displays the following acquired information (A) to (C). (A) Image 101a based on imaging data (B) Subject position data Dy of the subject in the image 101a based on the imaging data (C) Positioning point image associated with subject position data Dy of a subject in an image 101a based on imaging data A specific description will be given below. In the following specific example, an embodiment in which all of (A) to (C) are displayed will be described, but the present invention is not limited to this, and at least one of (A) to (C) will be displayed.

[0085] FIG. 8A is an example of a web screen 200. When the server is accessed, a map screen 201 is presented. When a pin icon in the map screen 201 is clicked, a pop-up screen 201a is displayed. In the pop-up screen 201a, location information (e.g., shooting position coordinates XY, altitude, coordinate system) and a thumbnail image of the subject image 202 (corresponding to the image 101a based on the imaging data) are displayed. When "Check details" at the bottom is clicked, the subject image 202 and its detailed information (not shown) are further displayed. The subject image 202 is an image to which location information (corresponding to the subject position data Dy) is attached. FIG. 8B is an enlarged view of the dashed frame B, and a positioning point image (point image) of the positioning point P9 is drawn in the subject image 202. As a result, the position information of the positioning point P9 is linked to and recorded in the subject image 202. The subject image 202 may include information 202a indicating the direction in which the subject image 202 was photographed together with the coordinate position ("north" in the example of FIG. 8A).

[0086] The appearance (e.g., color or shape) of the positioning point image of positioning point P9 may change depending on the radio wave conditions or the positioning accuracy. As an example, if the radio wave conditions or the positioning accuracy are good, the color may be green, and as the conditions deteriorate, the color may change to yellow or gray. For example, green may indicate that the radio wave conditions and the positioning accuracy are good. Also, a numerical value corresponding to the distance between the information processing terminal 1 and the subject may be displayed. This numerical value may be the distance itself (absolute value) or may be a relative value. The numerical value corresponding to the distance described here may be displayed, for example, at the bottom of the positioning point image of positioning point P9.

[0087] 4-3.Other Instead of a smartphone, a tablet terminal may be adopted as the terminal body 2. Unlike a notebook computer or the like, a smartphone or a tablet terminal is provided as an information processing terminal that is mainly held in one hand and suitable for image capture. As a normal usage mode, it is assumed that the terminal is held in one hand and the touch panel is operated with the holding hand or the other hand. By attaching the small and lightweight position acquisition terminal 3 according to the embodiment to such a terminal body 2, the positioning work can be easily performed with one hand. In particular, the position acquisition terminal 3 according to the embodiment is equipped with a battery 2c and an antenna 2b, and is a size that fits in one hand when attached to the terminal body 2, so that the user's convenience is high when moving when acquiring images. In addition, it is possible to suppress a decrease in the mobility of the user's actions at civil engineering / construction work or infrastructure management sites (e.g., railways, roads, electricity, gas, water), or disaster sites. Furthermore, the embodiments can be applied to various fields such as on-site surveys by civil engineering consultants, archaeological excavation surveys, agricultural applications, and livestock research (e.g., recording the shape and location of cow dung), in addition to civil engineering and construction sites. Furthermore, the embodiments are not limited to certain users, and are suitable for, for example, civil engineering and construction workers, infrastructure workers, researchers, land and house surveyors, media, police, and the Self-Defense Forces.

[0088] In the embodiment, all of S1 to S7 in the flow of FIG. 5 are executed, but the present invention is not limited to this. For example, it may be an information processing method (program) that executes S5-S7 of the flow. That is, in the process flow of Fig. 5, only "imaging object process S5, virtual object process S6, and comparison process S7" may be extracted and executed independently without combining with other processes (S1 to S4). This is because the imaging object can be acquired by various methods such as SfM process as described above. Note that it may be an information processing method (program) in which at least one of the flows S1 to S3 and the flow S4 is further additionally executed. Conversely, it may be an information processing method (program) that executes S1-S3 of the flow. Furthermore, it may be an information processing method (program) that additionally executes at least one of the flow S4 and the flows S5 to S7.

[0089] In the display process S1, when a subject corresponding to previously set subject position data Dy is included in the imaging range of the current display process, a positioning point image corresponding to the previously set subject position data Dy may be superimposed and displayed on the image 101a based on the imaging data. Note that the positioning point image is the same as that described in the embodiment, and is an image that is AR displayed (augmented reality display) in the augmented reality space W and is visible on the display screen 13, and any image having any shape and color can be adopted, and may be, for example, any point image or stake image. That is, in the display process S1 in the embodiment, a positioning point image corresponding to previously set subject position data Dy may be displayed. When a user captures an image of a subject in the past and sets and saves a positioning point, a positioning point image is set and saved at that positioning point. Then, when the user is currently positioning and points the camera again so that the subject is included, the previously set positioning point image is displayed on the information processing terminal 1. This allows the user to easily understand the location where the position was previously measured.

[0090] In the positioning process, the component points constituting the depicted image are regarded as the respective positioning points, and each subject position data is acquired, and a depicted image (not shown) may be displayed superimposed on the image 101a based on the imaging data. Here, the depicted image is an image depicted via an input device in the same image space as the UI image (i.e., the augmented reality space W). That is, in the positioning process in the embodiment, when a user uses an input device such as an electronic pen to draw a figure or the like in the augmented reality space W, each point (the above-mentioned constituent points) constituting the drawn figure (drawn image) becomes each positioning point. Then, subject position data Dy is set and saved for each positioning point. As a result, even if the user changes the angle of the camera, the drawn image continues to be displayed in the augmented reality space W. Needless to say, this depiction image may be displayed in a superimposed manner together with the UI image 101b described in the embodiment. In addition, since the positioning process, display process, and UI process are processes that can be performed in parallel, it is described here that in the positioning process, the depiction image is superimposed on the image 101a based on the imaging data, but the control flow can also be organized so that it is superimposed in the display process.

[0091] The function described in the section "4-2. Upload Function to Web Cloud, etc." may be realized not only by the server web cloud but also by the information processing terminal 1 alone (application of the information processing terminal 1). That is, in the application of the information processing terminal 1, when a pin icon in the map screen 201 is clicked, a pop-up screen 201a is displayed, and the pop-up screen 201a may display location information, a thumbnail image of the subject image 202, and the like. Also, the appearance of the positioning point image of the positioning point may change depending on the radio wave conditions or the positioning accuracy. Also, a numerical value corresponding to the distance between the information processing terminal 1 and the subject may be displayed. This numerical value may be the distance itself (absolute value) or may be a relative value. The numerical value corresponding to the distance described here may be displayed, for example, below the positioning point image of the positioning point P9.

[0092] 5. Notes Various embodiments are exemplified below. The embodiments shown below can be combined with each other. [Appendix 1] An information processing method executed by a portable information processing terminal, comprising: The display processing, the positioning processing, and the UI processing are provided, In the display process, an image based on the imaging data is displayed on a display screen, and a UI image is superimposed on the image based on the imaging data; the image based on the imaging data is a current image or a processed image obtained by processing the current image, the UI image is used to point out a subject for which a positioning point is to be set; In the positioning process, subject position data, which is position data of the positioning point of the subject, is acquired in response to an operation of setting the positioning point of the subject; The information processing method, wherein in the UI processing, the UI image is changed depending on a positioning situation in the positioning processing or on content of an image based on the imaging data in the display processing. [Appendix 2] 2. The information processing method according to claim 1, The portable information processing terminal acquires terminal position data, which is position data of the portable information processing terminal, by communicating with an artificial satellite, communicating with a base station, or performing SLAM, and acquires the subject position data based on the terminal position data. [Appendix 3] 5. The information processing method according to claim 2, the portable information processing terminal acquires relative position data in addition to the terminal position data, and acquires the subject position data based on the terminal position data and the relative position data; The information processing method, wherein the relative position data is data based on a relative positional relationship between the portable information processing terminal and the subject. [Appendix 4] 4. The information processing method according to claim 3, The portable information processing terminal acquires the relative position data including a relative position and a direction. [Appendix 5] 5. The information processing method according to claim 4, The portable information processing terminal includes: Obtaining the relative position data by performing a LiDAR scan; Calculating coordinates of each point of the point cloud obtained by the LiDAR scan based on the terminal position data, the relative position and the orientation; The subject position data is obtained based on the coordinates of a specific point in the point cloud. [Appendix 6] An information processing method according to any one of Supplementary Note 2 to Supplementary Note 5, The portable information processing terminal has a terminal body and a position acquisition terminal attached to the terminal body, The location acquisition terminal acquires the terminal location data, The terminal body acquires the subject position data. [Appendix 7] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 6, The method further includes an acquisition process, The information processing method, wherein the portable information processing terminal, in the acquisition process, acquires at least one of the following (a) to (e) using the subject position data acquired in the positioning process: (a) the coordinates of each of said positioning points; (b) the inter-point distance between each of the positioning points whose coordinates are obtained; (c) the area of ​​a plane figure having vertices each of the positioning points whose coordinates are acquired; (d) The volume or capacity of a three-dimensional figure having vertices each of the positioning points whose coordinates are acquired. (e) the azimuth, angle, or gradient between each of the positioning points from which the coordinates were obtained; [Appendix 8] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 7, The UI image includes an instruction graphic, the indication figure extends from a predetermined position to the indication point on the display screen, or moves from the predetermined position to the indication point, An information processing method, wherein the indication point indicates a position where the positioning point is set. [Appendix 9] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 8, The information processing method, wherein the portable information processing terminal changes at least one of the following (f) in the UI processing: (f) The size, thickness, length, movement speed, number, shape, whether or not it is displayed, whether or not it blinks, the time interval between blinks, the rotation angle, the operation, the color, and the transparency of the UI image, and the sound and vibration associated with the UI image [Appendix 10] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 9, The information processing method, wherein the portable information processing terminal changes the UI image stepwise or continuously in the UI processing in response to a change in a relative positional relationship between the portable information processing terminal and the subject. [Appendix 11] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 10, The information processing method, wherein the portable information processing terminal changes the UI image in the UI processing depending on whether or not the distance to the subject can be measured, the positioning accuracy of the subject, or the state of the subject. [Appendix 12] An information processing method according to any one of claims 1 to 11, An information processing method, wherein the UI processing executes the following (i) or (ii): (i) A process of changing the UI image according to the distance between the subject and the portable information processing terminal. (ii) A process of changing the UI image so as to guide the portable information processing terminal to a predetermined subject position. [Appendix 13] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 12, In the display process, when the subject corresponding to the set subject position data is included in the imaging range of the current display process, a positioning point image corresponding to the set subject position data is superimposed and displayed on an image based on the imaging data. [Appendix 14] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 13, In the positioning process, the object position data is acquired by using the constituent points constituting the depicted image as the respective positioning points, and the depicted image is superimposed on an image based on the imaging data and displayed; An information processing method, wherein the depicted image is an image depicted via an input device in the same image space as the UI image. [Appendix 15] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 14, The imaging object processing, the virtual object processing, and the comparison processing are further provided, The portable information processing terminal includes: In the imaging object processing, an imaging object is acquired, In the virtual object processing, a virtual object is acquired, The information processing method, wherein in the comparison process, a comparison result between the imaging object and the virtual object is obtained. [Appendix 16] 16. The information processing method according to claim 15, further comprising: The portable information processing terminal acquires the comparison result after aligning the imaging object with the virtual object in the comparison process. [Appendix 17] 17. The information processing method according to claim 15, further comprising: The portable information processing terminal acquires the imaging object using a three-dimensional scan process of the information processing terminal in the imaging object process. [Appendix 18] An information processing method according to any one of Supplementary Note 15 to Supplementary Note 17, The information processing method, wherein the portable information processing terminal generates the virtual object based on object data or based on a virtual figure set by a user in a virtual space in the virtual object processing. [Appendix 19] An information processing method according to any one of Supplementary Note 1 to Supplementary Note 18, Further comprising an acquired information display process, In the acquired information display process, at least one of the following (A) to (C) is displayed as acquired information: (A) An image based on the imaging data (B) the subject position data of the subject in the image based on the imaging data; (C) a positioning point image associated with the subject position data of the subject in an image based on the imaging data; [Appendix 20] An information processing method including an imaging object process, a virtual object process, and a comparison process, In the imaging object processing, an imaging object is acquired, The virtual object processing includes acquiring a virtual object, The information processing method, wherein in the comparison process, a comparison result between the imaging object and the virtual object is obtained. [Appendix 21] A computer program for executing the information processing method according to any one of claims 1 to 20. [Explanation of symbols]

[0093] 1: Information processing terminal 2: Device body 2a: Housing 2b: Antenna 2c: Battery 3: Location acquisition device 3a: Housing 3b: Antenna 3c: Battery 3d: Main circuit 6: Communication network 10: Communications Department 11: Storage section 12: Control section 13:Display screen 14: Camera 15: LiDAR scanner 16: Communication bus 101: Display section 101a: Image based on imaging data 101b :UI image 101b1: Pointing figure 101b2 :Indication point 101c:UI operation image 101c1~101c8: UI operation buttons 102: Positioning unit 102b2 :Indication point 103:UI section 104: Acquisition Department 105: Imaging object section 106: Virtual Objects Section 107: Comparison section 200: Web page 201: Map screen 201a: Pop-up screen 202: Subject image (subject image with location information) 202a: Information indicating the direction in which the image was taken Dr: Relative position data Dt: Terminal location data Dy: Subject position data P, P1~P8: Positioning point Qr: 3D figure (an example of an imaging object) Qv : Virtual object U :User X: On-site (real space) Y: Positioning target Z:Artificial satellite A: Upper apex K:Base station W: Augmented reality space

Claims

1. An information processing method executed by a portable information processing terminal, comprising: The display processing, the positioning processing, and the UI processing are provided, In the display process, an image based on the imaging data is displayed on a display screen, and a UI image is superimposed on the image based on the imaging data; the image based on the imaging data is a current image or a processed image obtained by processing the current image, the UI image is used to point out a subject for which a positioning point is to be set; In the positioning process, subject position data, which is position data of the positioning point of the subject, is acquired in response to an operation of setting the positioning point of the subject; In the UI processing, the UI image is changed depending on a positioning situation in the positioning processing or on content of an image based on the imaging data in the display processing.

2. 2. The information processing method according to claim 1, The portable information processing terminal acquires terminal position data, which is position data of the portable information processing terminal, by communicating with an artificial satellite, communicating with a base station, or performing SLAM, and acquires the subject position data based on the terminal position data.

3. 3. The information processing method according to claim 2, the portable information processing terminal acquires relative position data in addition to the terminal position data, and acquires the subject position data based on the terminal position data and the relative position data; The information processing method, wherein the relative position data is data based on a relative positional relationship between the portable information processing terminal and the subject.

4. 4. The information processing method according to claim 3, The portable information processing terminal acquires the relative position data including a relative position and a direction.

5. 5. The information processing method according to claim 4, The portable information processing terminal includes: Obtaining the relative position data by performing a LiDAR scan; Calculating coordinates of each point of the LiDAR scanned point cloud based on the terminal position data, the relative position, and the orientation; The subject position data is obtained based on the coordinates of a specific point in the point cloud.

6. 3. The information processing method according to claim 2, The portable information processing terminal has a terminal body and a position acquisition terminal attached to the terminal body, The location acquisition terminal acquires the terminal location data, The terminal body acquires the subject position data.

7. 2. The information processing method according to claim 1, The method further includes an acquisition process, The portable information processing terminal, in the acquisition process, acquires at least one of the following (a) to (e) using the subject position data acquired in the positioning process: (a) the coordinates of each of said positioning points; (b) the inter-point distance between each of the positioning points whose coordinates are acquired; (c) The area of ​​a plane figure having vertices each of the positioning points whose coordinates are acquired. (d) The volume or capacity of a three-dimensional figure having vertices each of the positioning points whose coordinates are acquired. (e) the azimuth, angle, or gradient between each of the positioning points whose coordinates were obtained;

8. 2. The information processing method according to claim 1, The UI image includes a pointer graphic, the indication figure extends from a predetermined position to an indication point on the display screen, or moves from the predetermined position to the indication point, An information processing method, wherein the indication point indicates a position where the positioning point is set.

9. 2. The information processing method according to claim 1, The information processing method, wherein the portable information processing terminal changes at least one of the following (f) in the UI processing: (f) the size, thickness, length, movement speed, number, shape, display presence / absence, blinking presence / absence, blinking time interval, rotation angle, operation, color and transparency of the UI image, and sound and vibration associated with the UI image;

10. 2. The information processing method according to claim 1, The information processing method, wherein the portable information processing terminal changes the UI image stepwise or continuously in the UI processing in response to a change in a relative positional relationship between the portable information processing terminal and the subject.

11. 2. The information processing method according to claim 1, The information processing method, wherein the portable information processing terminal changes the UI image in the UI processing depending on whether or not the distance to the subject can be measured, the positioning accuracy of the subject, or the state of the subject.

12. 2. The information processing method according to claim 1, The information processing method, wherein the UI processing executes the following (i) or (ii): (i) A process of changing the UI image according to a distance between the subject and the portable information processing terminal. (ii) A process of changing the UI image so as to guide the portable information processing terminal to a predetermined subject position.

13. 2. The information processing method according to claim 1, In the display process, when the subject corresponding to the set subject position data is included in the imaging range of the current display process, a positioning point image corresponding to the set subject position data is superimposed and displayed on an image based on the imaging data.

14. 2. The information processing method according to claim 1, In the positioning process, the object position data is acquired by using the constituent points constituting the depicted image as the respective positioning points, and the depicted image is superimposed on an image based on the imaging data and displayed; An information processing method, wherein the depicted image is an image depicted via an input device in the same image space as the UI image.

15. 2. The information processing method according to claim 1, The imaging object processing, the virtual object processing, and the comparison processing are further provided, The portable information processing terminal includes: In the imaging object processing, an imaging object is acquired, In the virtual object processing, a virtual object is acquired, The information processing method, wherein in the comparison process, a comparison result between the imaging object and the virtual object is obtained.

16. 16. An information processing method according to claim 15, The portable information processing terminal acquires the comparison result after aligning the imaging object with the virtual object in the comparison process.

17. 16. An information processing method according to claim 15, The information processing method, wherein the portable information processing terminal acquires the imaging object using a three-dimensional scan process of the information processing terminal in the imaging object process.

18. 16. An information processing method according to claim 15, The information processing method, wherein the portable information processing terminal generates the virtual object based on object data or based on a virtual figure set by a user in a virtual space in the virtual object processing.

19. 2. The information processing method according to claim 1, Further comprising an acquired information display process, In the acquired information display process, at least one of the following (A) to (C) is displayed as acquired information. (A) An image based on the imaging data (B) the subject position data of the subject in the image based on the imaging data; (C) a positioning point image associated with the subject position data of the subject in the image based on the imaging data;

20. A computer program for executing the information processing method according to any one of claims 1 to 19.

Images