Location calculation method and information processing system

Abstract

This invention provides a position calculation method and information processing system that enable easy on-site control of an aircraft. [Solution] The position calculation method includes the steps of: specifying a predetermined position on a captured image; acquiring a two-dimensional coordinate position corresponding to the predetermined position and at least shooting state information; calculating an angle and direction from the user terminal position to a three-dimensional coordinate position corresponding to the two-dimensional coordinate position based on the two-dimensional coordinate position and shooting state information; associating at least the user terminal position and the calculated angle and direction with three-dimensional coordinates in a virtual space, and calculating the data of the point that first touches the line when a virtual straight line is extended from the user terminal position to the calculated angle and direction as a virtual designated position corresponding to the position specified by the user; and calculating the three-dimensional coordinate position of the designated position from the virtual designated position.

Description

[Technical Field]

[0001] This invention relates to a method for calculating position and an information processing system. [Background technology]

[0002] In recent years, drones and unmanned aerial vehicles (UAVs) have become popular. Aerial vehicles (hereinafter collectively referred to as "aerial vehicles") began to be used in industry. In this context, Patent Document 1 describes a method for controlling an aircraft using a dedicated device such as a radio control system. This has been disclosed. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2018-093435 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, in the technology disclosed in Patent Document 1, operation using a dedicated device such as a transmitter is easy. Rather, skilled techniques are required to control the aircraft, especially for tasks such as inspections. .

[0005] On the other hand, there is a method where a flight path is specified in advance, and the aircraft is automatically controlled along that flight path. While there are laws in place, it is common practice to designate flight paths outside the site, and flexibility is required to adapt to the situation at the site. It lacked something.

[0006] The present invention has been made in view of such a background, and an object thereof is to provide a position calculation method, a flight control method, and an information processing system that enable easy control of a flying object at the site.

Means for Solving the Problems

[0007] The main invention of the present invention for solving the above problems is a method for calculating a position specified on the screen of a user terminal, comprising: specifying a predetermined position on a captured image acquired by a capturing unit on the screen of the user terminal; acquiring a two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position, and at least imaging angle, imaging degree, imaging orientation, and imaging state information including user terminal position information when the predetermined position is specified; calculating an angle and orientation from the three-dimensional coordinate position indicated by the user terminal position information to a three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the screen of the user terminal based on the two-dimensional coordinate position and the imaging state information; associating at least the three-dimensional coordinate position indicated by the user terminal position information, and the calculated angle and orientation with three-dimensional coordinates in the virtual space, and calculating, as a virtual specified position in the virtual space corresponding to the position specified by the user, a three-dimensional coordinate position indicated by point data of a point that first contacts the straight line when a virtual straight line is extended from the three-dimensional coordinate position indicated by the user terminal position information to the calculated angle and orientation; and calculating a three-dimensional coordinate position of a specified position specified by the user in the real space from the virtual specified position based on the correspondence relationship between the three-dimensional coordinates on the virtual space and the three-dimensional coordinates on the real space. ​​​​​​​​​​​​​​​ [Effects of the Invention]

[0008] According to the present invention, a position calculation method and information processing system that enable easy on-site control of an aircraft are provided. We can provide the service. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the configuration of a management system according to an embodiment of the present invention. [Figure 2] This block diagram shows the hardware configuration of the management server in Figure 1. [Figure 3] This block diagram shows the hardware configuration of the user terminal in Figure 1. [Figure 4] This block diagram shows the hardware configuration of the aircraft in Figure 1. [Figure 5] This block diagram shows the functions of the management server in Figure 1. [Figure 6] Figure 5 is a block diagram showing the structure of the parameter information storage unit. [Figure 7] This is a flowchart of a flight control method according to an embodiment of the present invention. [Figure 8] This figure shows an example of a description of a position calculation method according to an embodiment of the present invention. [Figure 9] This figure shows an example of a description of a position calculation method according to an embodiment of the present invention. [Figure 10] This figure shows an example of a description of a position calculation method according to an embodiment of the present invention. [Figure 11] This figure shows an example of a description of a position calculation method according to an embodiment of the present invention. [Modes for carrying out the invention]

[0010] The embodiments of the present invention will be listed and described below. A method for calculating position according to an embodiment of the present invention. The information processing system has the following configuration: [Item 1] A method for calculating a position specified on the user terminal screen, On the screen of the user terminal, specify a predetermined position on the captured image acquired by the shooting unit. The steps, The two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position, and the predetermined When specifying the location, at least the shooting angle, shooting direction, and user terminal location information are required. The steps include receiving shooting status information, Based on the two-dimensional coordinate position and the shooting status information, the user terminal position information indicates the tertiary position From the original coordinate position, the three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the user terminal screen. A step of calculating the angle and direction to, At least the three-dimensional coordinate position indicated by the user terminal location information, and the calculated angle. And the direction is mapped to three-dimensional coordinates in the virtual space, and the three-dimensional position indicated by the user terminal position information When a virtual straight line is extended from the target position to the angle and direction calculated above, the line is closest to the target position. The three-dimensional coordinate position indicated by the initial contact point data corresponds to the position specified by the user. The steps include: calculating a virtual designated location within the virtual space; Based on the correspondence between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculates the three-dimensional coordinate position of the user-specified location in real space from the virtual specified location. The steps include, A method for calculating position characterized by the following features. [Item 2] The position calculation method described in item 1, The aforementioned point data is one of the three-dimensional point cloud data in the virtual space. A method for calculating position characterized by the following features. [Item 3] A method for calculating position as described in item 1 or 2, The aforementioned point data is stored in the aircraft. A method for calculating position characterized by the following features. [Item 4] A method for calculating position as described in item 1 or 2, The aforementioned point data is stored in the user terminal. A method for calculating position characterized by the following features. [Item 5] The position calculation methods described in items 1 to 4, The aforementioned imaging unit is provided in the user terminal. A method for calculating position characterized by the following features. [Item 6] The position calculation methods described in items 1 to 4, The aforementioned imaging unit is installed on the aircraft, A method for calculating position characterized by the following features. [Item 7] An information processing system that calculates a specified position on the screen of a user terminal, The aforementioned information processing system includes an imaging unit, a designated location information acquisition unit, and an imaging status information receiving unit. Equipped with a position calculation unit, The aforementioned designated location information acquisition unit, On the screen of the user terminal, the user specifies the image captured by the camera unit. The two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position is obtained. The aforementioned shooting status information receiving unit is When the predetermined position is specified, at least the shooting angle, shooting direction, and user end Receive shooting status information including last position information, The position calculation unit, Based on the two-dimensional coordinate position and the shooting status information, the user terminal position information indicates the tertiary position From the original coordinate position, the three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the user terminal screen. Calculate the angle and direction to, At least the three-dimensional coordinate position indicated by the user terminal location information, and the calculated angle. And the direction is mapped to three-dimensional coordinates in the virtual space, and the three-dimensional position indicated by the user terminal position information When a virtual straight line is extended from the target position to the angle and direction calculated above, the line is closest to the target position. The three-dimensional coordinate position indicated by the initial contact point data corresponds to the position specified by the user. Calculated as a virtual designated position within the aforementioned virtual space, Based on the correspondence between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculates the three-dimensional coordinate position of the user-specified location in real space from the virtual specified location. do, An information processing system characterized by the following:

[0011] <Details of the embodiment> The following describes a position calculation method, an aircraft control method, and an information processing system according to embodiments of the present invention. Embodiments of this will be described. In the attached drawings, identical or similar elements are shown as identical. Or similar reference numerals and names are used, and the same or similar elements are used in the description of each embodiment. Duplication of explanations regarding this matter may be omitted. Furthermore, the features shown in each embodiment are mutually exclusive. This can be applied to other embodiments as long as it does not contradict the above.

[0012] <Structure> As shown in Figure 1, the management system in this embodiment includes a management server 1 and one It comprises a user terminal 2, one or more aircraft 4, and one or more aircraft storage devices 5. Management server 1, user terminal 2, aircraft 4, and aircraft storage device 5 are connected via a network. They are connected to each other in a way that allows them to communicate. Note that the illustrated configuration is just one example and is not limited to this. For example, it may also be a configuration in which the aircraft is carried by the user without having an aircraft storage device 5. stomach.

[0013] <Management Server 1> Figure 2 shows the hardware configuration of management server 1. Note that the configuration shown is one This is just an example, and other configurations are also possible.

[0014] As shown in the diagram, the management server 1 has multiple user terminals 2, an aircraft 4, and an aircraft storage unit. It is connected to the 5 and forms part of this system. Management Server 1 is, for example, a workstation It could be a general-purpose computer like a personal computer, or it could be a cloud computer. • It may be logically implemented through computing.

[0015] Management server 1 includes at least a processor 10, memory 11, storage 12, and transmit / receive. It includes a section 13, an input / output section 14, etc., which are electrically connected to each other via a bus 15.

[0016] The processor 10 controls the overall operation of the management server 1 and the transmission and reception of data between each element. A computing unit that performs information processing necessary for signal control, application execution, and authentication processing. For example, processor 10 is CPU (Central Processing Unit). (it) and / or GPU (Graphics Processing Unit) Yes, the program for this system is stored in storage 12 and loaded into memory 11. The following are performed to carry out each information processing task.

[0017] Memory 11 is DRAM (Dynamic Random Access Memory) The main memory consists of volatile memory devices such as y), and flash memory and HDD (Hard Memory includes auxiliary storage composed of non-volatile storage devices such as disc drives. 11 is used as the work area of ​​processor 10, and also when management server 1 starts up. The BIOS (Basic Input / Output System) is executed. It also stores various setting information, etc.

[0018] Storage 12 stores various programs such as application programs. Even if a database containing the data used for processing is built on storage 12, stomach.

[0019] The transmitting / receiving unit 13 connects the management server 1 to the network and the blockchain network. Connect. Note that the transmitting / receiving unit 13 uses Bluetooth (registered trademark) and BLE (Blu Even if it has a short-range communication interface (etooth Low Energy), stomach.

[0020] The input / output section 14 includes information input devices such as keyboards and mice, and output devices such as displays. It is a device.

[0021] Bus 15 is connected in common to each of the above elements, for example, to address signals, data signals and each It transmits a type of control signal.

[0022] <User Terminal 2> The user terminal 2 shown in Figure 3 also includes a processor 20, memory 21, storage 22, It includes a transmitting / receiving unit 23, an input / output unit 24, an imaging unit 26, an imaging status information acquisition unit 27, etc. They are electrically connected to each other via bus 25. The function of each element is as described above with the management server 1. Since a similar configuration is possible, a detailed explanation of this configuration will be omitted.

[0023] The imaging unit 26 is, for example, a visible light camera or an infrared camera, and captures images (for example, still images). Acquire images, videos, etc.

[0024] <Flying object 4> Figure 4 is a block diagram showing the hardware configuration of the aircraft 4. Flight controller 41 is one of the programmable processors (for example, a central processing unit (CPU)). It can have the above processors.

[0025] Furthermore, the flight controller 41 has a memory 411, and access to this memory It is possible. Memory 411 is used to perform one or more steps on the flight controller. It stores executable logic, code, and / or program instructions. The flight controller 41 uses inertial sensors (accelerometer, gyroscope), GP This may include sensors 412 such as S sensors and proximity sensors (e.g., lidar).

[0026] Memory 411 is used for isolation of, for example, SD cards and random access memory (RAM). It may include a suitable medium or an external storage device. Acquired from the imaging unit / sensors 42 The data may be directly transmitted to and stored in memory 411. For example, in the imaging unit, etc. The captured still images and video data may be recorded in the internal memory or external memory, but Not limited to, but also from the shooting unit / sensor 42 or built-in memory via the network NW, The data may also be recorded on the management server 1, user terminal 2, or aircraft storage device 5. Part 42 is mounted on the aircraft 4 via the gimbal 43.

[0027] The flight controller 41 is configured to control the state of the aircraft (not shown) It includes modules. For example, the control module has 6 degrees of freedom (translational motion x, y and z, and rotational motion θ x , θ y and θ z The spatial arrangement, speed, and / Alternatively, to adjust the acceleration, use ESC44 (Electric Speed ​​Control) The propulsion mechanism of the aircraft (motor 45, etc.) is controlled via the roller. Battery 4 The motor 45, powered from 8, rotates the propeller 46, generating lift for the aircraft. The control module can control one or more of the states of the mounted components and sensors. Cut.

[0028] The flight controller 41 controls one or more external devices (for example, a transmitter / receiver (transmitter / controller)). )49, transmitting and / or data from terminals, display devices, or other remote controls It is possible to communicate with the transceiver 47 which is configured to receive. The transceiver 49 is wired communication Alternatively, any suitable means of communication, such as wireless communication, may be used.

[0029] For example, the transmitting / receiving unit 47 is a local area network (LAN), wide area network Network (WAN), infrared, wireless, Wi-Fi, point-to-point (P2P) network You can use one or more of the following: work, telecommunications networks, cloud communication, etc. Cut.

[0030] The transmitting / receiving unit 47 receives data acquired by the sensors 42 and data generated by the flight controller 41. Processing results, predetermined control data, user commands from a terminal or remote controller, etc. You can send and / or receive one or more of these.

[0031] The sensors 42 in this embodiment include inertial sensors (accelerometer, gyroscope), GPS sensors, proximity sensors (e.g., LiDAR), or vision / image sensors (e.g.) For example, it could include a camera.

[0032] <Features implemented in each configuration> Figure 5 shows the management server 1, user terminal 2, and aircraft 4 implemented in this embodiment. This is a block diagram illustrating the functionality.

[0033] First, as a function to be implemented in management server 1, processor 10 will handle flight missions. It is equipped with a generation unit 110, and the storage 12 is a flight path information storage unit 122, flight path It is equipped with a memory unit 124, and the transmitting / receiving unit 13 is equipped with a communication unit 130. The flight mission generation unit 110 includes a flight path generation unit 112. The storage unit 12 is It also has a memory unit that stores the information necessary to perform the flight mission. Also, information about flight conditions (e.g., flight speed, waypoint spacing, etc.) is often used. Also, a memory unit (not shown) that stores virtual space information (for example, three-dimensional point cloud data) as described later. They may each have one of these.

[0034] The flight mission generation unit 110 generates flight missions. The system includes at least waypoint information (e.g., latitude and longitude information and flight altitude information). This includes information such as flight paths. Setting a flight path is, for example, as described later, about the object Three-dimensional point cloud model information, the aircraft's current position, the flight mission start position, and the target object The flight path generation unit 112 automatically calculates and sets waypoints by acquiring distances and other information. You may also do this. The generated flight path information is stored in the flight path information storage unit (not shown). It may also be stored in memory. Furthermore, flight missions are tasks specified by the user, for example. For example, it may include imaging, inspection, security, etc., from the flight mission starting position. The aircraft may be controlled to initiate the above operations. Detailed settings may be registered in advance on management server 1, etc.

[0035] Furthermore, the flight path may, for example, hover near the user without having an aircraft storage device 5. The current position is determined by the location where the aircraft is waiting or the location where the user has carried the aircraft. It could also be configured so that the user retrieves the aircraft at the starting position or the end position of the flight. Furthermore, information about the aircraft storage device 5 managed by the management server 1 (for example, location information and storage status) Based on status information, hangar aircraft information, etc., the flight start position, intermediate stops, or flight end position are determined. The configuration may also be one in which the flight path is generated including the position of the selected aircraft storage device 5.

[0036] The flight path information storage unit 122 stores the flight path information of the aircraft generated by the flight mission generation unit 110. It stores flight path information. The flight log storage unit 124 stores, for example, flight mission information. Information acquired by aircraft 4 along the flight path set in (for example, from takeoff to landing) It stores information about the locations it passed through before reaching land, as well as still images, moving images, audio, and other information.

[0037] The communications unit 130 communicates with the user terminal 2, the aircraft 4, and the aircraft storage device 5. Unit 130 also functions as a reception desk that accepts flight requests from user terminal 2.

[0038] Next, as a function to be implemented in the user terminal 2, the processor 20 is a unit for acquiring shooting status information. The device includes a designated location information acquisition unit 240, and the storage 22 stores shooting status information. It is equipped with a unit 222, and the transmitting / receiving unit 23 is equipped with a communication unit 230. Also, storage 22 is a storage unit (not shown) that stores the designated location information acquired by the designated location information acquisition unit 240. It may also have

[0039] The shooting status information acquisition unit 220 uses, for example, GPS, gyro sensor, barometric pressure sensor, and temperature sensor. The system stores information about sensors such as the camera and the terminal's camera unit 26 (for example, information about the field of view). From the memory unit (which may be part of memory 21 or storage 22), the captured image is taken. The data acquired includes the user's device's field of view, shooting angle, shooting direction, and user device position. Information (such as latitude, longitude, and altitude) is acquired as shooting status information. The altitude information included in the user terminal location information is calculated based on the aforementioned barometric pressure sensor and temperature sensor. This could be advanced information provided by the user, or advanced information set by the user, for example, Based on the height information provided, or the average height information corresponding to the user's gender, the expected user The altitude information may be a value that is offset vertically by a predetermined height depending on the terminal's position.

[0040] The designated location information acquisition unit 240 acquires the location on the screen specified by the user (for example, a predetermined location). The origin is set as an xy coordinate position with the screen width as the x-axis and the screen height as the y-axis. The specified location information is obtained from the user. The way the user specifies the location on the screen is, for example, You can specify by touching the touch display with your finger, or manipulate objects such as pointers. You can specify a location, or move user terminal 2 to a specific position on the screen (for example, the center). This can be achieved by specifying the combination of the following and executing the decision action:

[0041] As shown in Figure 6, the shooting status information storage unit 222 is the shooting angle of view information storage unit 2221, and the shooting Angle information storage unit 2222, shooting direction information storage unit 2223, user terminal location information storage unit 22 It includes at least 24 and stores the relevant shooting state information for each.

[0042] The communications unit 230 communicates with the management server 1, the aircraft 4, and the aircraft storage device 5.

[0043] Next, as a function to be implemented in the aircraft 4, the processor 413 is the shooting status information receiving unit 4 It is equipped with 15 and a position calculation unit 417, and the memory 411 is a virtual space information storage unit 421 The transmitting / receiving unit 47 is equipped with a communication unit 470. The memory 411 is also equipped with a memory unit 411. The system further includes a storage unit (not shown) that stores the shooting status information received by the status information receiving unit 415. It's fine if you do that.

[0044] The shooting status information receiving unit 415 receives the shooting status information acquired by the shooting status information acquisition unit 220 of the user terminal 2. Shadow status information is received via the communication unit 470.

[0045] The position calculation unit 417 calculates at least the shooting status information of the user terminal 2, the specified position information, and the virtual sky. The virtual space information (e.g., three-dimensional point cloud data) stored in the inter-information 421, and the current position of the flying object 4. Based on location information (e.g., obtained from GPS), the user can specify on the screen of user terminal 2. This function calculates whether the specified location is in virtual space or real space. Detailed calculation method The law will be discussed later.

[0046] The virtual space information storage unit 421 stores, for example, three-dimensional point cloud data within the range that can be captured from the user terminal 2. It stores virtual space information including data. Three-dimensional point cloud data is, for example, LIDAR (Light The imaging unit can measure detection and ranging, as well as depth. This is three-dimensional coordinate data in a virtual space acquired by sensors, which was obtained by flying the device beforehand. Alternatively, data can be retrieved from, for example, management server 1 via the network. You may come. Also, the three-dimensional coordinates of the flying object 4 in real space and the three-dimensional coordinates in virtual space should be obtained in advance. The targets may be associated with each other and stored in the virtual space information storage unit 421.

[0047] In this embodiment, the functions of the processor 413 on the aircraft 4 include position calculation. The explanation is based on an example that includes an outlet 417, but it is not limited to this example; for example, user terminal 2 Processors such as processor 20, processor 10 of management server 1, and processor of aircraft storage device 5. It is designed to have a built-in function that reads the information necessary for position calculation into the relevant processor and performs the calculation. This may be done by assigning the processing load for position calculation to the desired processor. can.

[0048] Furthermore, the position is calculated and the aircraft is controlled based on the image captured by the camera unit 26 of the user terminal 2. However, instead of this, the position is calculated and the aircraft is controlled based on the images captured by the camera unit of aircraft 4. This is also acceptable. In this case, the aircraft 4 Position calculation can be achieved by displaying the image captured by the camera unit on the screen of user terminal 2. i. When based on the image captured by the shooting unit 26 of the user terminal 2, the user can determine the shooting environment. While visually observing, the user points the camera unit 26 of the user terminal 2 towards that location and objectively and intuitively points out the position. By doing so, it is possible to control the aircraft 4, but based on the images captured by the camera unit of the aircraft 4 In this case, the positioning is subjective and intuitive, as if the user were directly controlling it. By specifying this, it becomes possible to control the aircraft.

[0049] <An example of a position calculation method> The position calculation method according to this embodiment will be described with reference to Figure 7-10. A flowchart illustrating the position calculation method according to this embodiment is shown. Figure 8-10 is shown. This figure shows an example of a description of the position calculation method according to an embodiment of the present invention. For example, the processor 413 on the aircraft 4 is equipped with a position calculation unit 417, This explanation will describe a configuration in which the position is calculated based on the image captured by the camera unit 26 of terminal 2.

[0050] First, as shown in Figure 8, the image acquired by the imaging unit 26 is displayed on the screen of the user terminal 2. A predetermined position on the captured image S is specified using a pointer P or the like (SQ101).

[0051] Next, the aircraft 4 receives information from the shooting status information receiving unit 415, and the user terminal at SQ101... When specifying a predetermined position (x and y coordinates on the screen) on screen 2 and a predetermined position Get the shooting status information (shooting angle of view, shooting angle, shooting direction, user terminal location information) (S Q102).

[0052] Next, the aircraft 4 is determined by the position calculation unit 417 based on the above-mentioned xy coordinate position and shooting status information. Therefore, from the position of the user terminal (three-dimensional coordinates), a predetermined position specified on the screen (x on the screen) The angle and direction to at least the position (three-dimensional coordinates) in virtual space, corresponding to the y-coordinate position. Calculate the position (SQ103).

[0053] More specifically, Figure 9 illustrates how to calculate the angle in the y-axis direction.

[0054] First, the deviation d from the center of the shooting unit. y When the origin of the y-coordinate on the screen is taken as the top edge, From the specified position y in the y-axis direction on the surface, by the height d determined by the resolution h Subtract the half value of and take the absolute value of the result, and the following relationship of Equation 1 holds.

Equation

[0055] Next, based on the deviation angle α from the orientation of the imaging unit h the following relationship of Equation 2 holds.

Equation

[0056] Next, based on the half value of the angle of view FOV h the following relationship of Equation 3 holds.

Equation

[0057] Here, for example, substitute the formula described in Equation 3 into the formula described in Equation 2 based on the distance distance, and organize the deviation angle α from the orientation of the imaging unit h to obtain the following relationship of Equation 4.

Equation

[0058] At this time, the deviation angle α from the orientation of the imaging unit h also satisfies the following relationship of Equation 5 and

Equation

[0059] Finally, by substituting the obtainable values ​​into the above number 4, the direction of the imaging unit in the y-axis direction can be determined. Angle of deviation from the base α h This is calculated.

[0060] Furthermore, the orientation in the x-axis direction is calculated using the same method as above. α - deviation from the base w This is calculated.

[0061] Let's return to the flowchart in Figure 7 and explain the position calculation method. Next, the flying object 4 The position calculation unit 417 calculates the position P of the user terminal 2. s and the displacement angle calculated with SQ103 α h Aligned and offset direction α w Position P of flying object 4 d By associating this with three-dimensional coordinates in a virtual space, - The location P of terminal 2 s Angle of deviation α h Aligned and offset direction α w When extending a straight line, initially The point data D (or point cloud model M) that came into contact with the object is sent to the user-specified location (or object). It is calculated as (SQ104).

[0062] More specifically, Figure 10 illustrates how to calculate the position specified by the user. To simplify the explanation, let's start by explaining it in terms of the x-axis.

[0063] The left diagram of Figure 10 shows the position P of user terminal 2. s The deviation direction α calculated using the above formulas, etc. w The relationship between the object specified on the image and the specified location is shown. The value and position P of the flying object 4 d The position calculation unit 417 of the flying object 4 calculates, for example, a three-dimensional point cloud. The data is applied within the mapped virtual space, resulting in a positional relationship as shown in the right-hand diagram of Figure 10. .

[0064] Here, at position P of user terminal 2, s Azimuth deviation α w Extend a virtual straight line in that direction, The position of the point data D that made initial contact is calculated as the position specified by the user. Similarly, Angle α h Since the same process is being carried out in parallel in the y-axis direction using this method, the result and Then, the three-dimensional coordinate position of the point data in the virtual space corresponding to the location specified by the user. The (virtual designated position) is calculated.

[0065] Finally, position P of flying object 4 d Based on this, three-dimensional coordinates in virtual space and three-dimensional coordinates in real space Based on the correspondence with the original coordinates, the user-specified position in real space is calculated (SQ1 05). In this embodiment, the flying object 4 has virtual space information, therefore Position P of row 4 d The position in real space is calculated based on this, but user terminal 2 is in virtual space If information is available, it does not necessarily mean the position P of the flying object 4. d Information regarding this is not needed. Location P of user terminal 2 s That alone may suffice.

[0066] This allows the position specified by the user on the screen of user terminal 2 to be placed in a three-dimensional position in real space. This makes it possible to calculate the target position, which can then be used to easily control various aircraft. This is the result.

[0067] <Example of aircraft control 1-1> For example, using the three-dimensional coordinates in real space of a position specified on the screen of the user terminal 2 described above The aircraft 4 may be controlled to fly to that position. It may be executed by processor 413 or the processor 20 of user terminal 2, or by management server This may be executed by processor 10 of unit 1. This will acquire three-dimensional point cloud data. If you are near the target object, you can easily specify its location on the screen of user terminal 2. It becomes possible to fly aircraft 4 up to this point.

[0068] <Example of aircraft control 1-2> In the above-mentioned aircraft control example 1-1, the aircraft 4 is flown to a position specified by the user. However, in addition to that, it also determines the shape of the object including its position and the position specified by the user (and We will perform flight missions such as inspections from a position offset by a predetermined distance from there. This is also acceptable. The shape of the object may be determined by referring to three-dimensional point cloud data, or by mounting it on the aircraft 4. The three-dimensional point cloud data of the object may be acquired on-site using the included LIDAR or similar device. For example, the user-specified location is used as the flight mission start position, and the 3D point cloud of the target object is generated. The system may also determine the flight path and execute the flight mission based on the data. These processes, for example, also record the virtual space information stored by the aircraft 4 on the management server 1. The flight path may be kept in mind and generated in the flight path generation unit 112. Additionally, the designation of a flight mission depends on the flight before the user specifies their location. All that's required is that a mission is specified.

[0069] <Examples of aircraft control 1-3> In the above-mentioned aircraft control examples 1-1 and 1-2, three point data points are used as the user-specified position. Although dimensional coordinates are used, instead, the object containing the point data in question is determined, and the object The shape of an object is determined from three-dimensional point cloud data, and for flight missions targeting that object... The flight path may be pre-generated. This allows the user to control the aircraft. Flight missions to the target object can be executed without strictly specifying the location.

[0070] <Example of aircraft control 1-4> For example, after the aircraft 4 is flown to a location specified by the user, a device capable of communicating with the aircraft 4. Communication may be performed with the aircraft. For example, the device is an aircraft storage device 5, and the aircraft 4 The acquired information is transmitted to the management server 1 etc. via the aircraft storage device 5, or the aircraft storage device Performs communication processing for storage in device 5, or replaces an aircraft in aircraft storage device 5 or They may also perform communication processing for conducting joint flight missions. Whether this action is performed depends on when the user specifies a location on the screen of user terminal 2. The system may also be configured to specify the operation.

[0071] <Example of aircraft control 2-1> For example, as described above, specified on the screen of user terminal 2 When calculating the three-dimensional coordinates of a position in real space, the user terminal position P is calculated in virtual space. s (Temporary) When an imaginary straight line is extended, if it does not touch any of the three-dimensional point cloud data, or if a predetermined If the aircraft does not make contact with the 3D point cloud data even after extending the distance, the aircraft 4 will execute a predetermined fail-safe. A fail-safe operation may be performed. Such a fail-safe operation may be, for example, on the spot or Finally, the device flies to a location specified by the user or to the location of user terminal 2 and hovers there. Landing may also be performed. Additionally, alerts may be displayed on user terminal 2 (e.g., 3D point cloud). You may display a message indicating that the data is not being accessed and that the location cannot be calculated.

[0072] <Example of aircraft control 2-2> For example, as described above, when controlling the flight of aircraft 4 to a position specified by the user, When an obstacle is detected by the imaging unit or sensor mounted on the aircraft 4, a predetermined fail-second - Action (You can stop in place and hover, or land, or immediately before) You can return to the specified position or the position of user terminal 2, or you can use the 3D point cloud data to remove obstacles. (If the shape is discernible, you may continue flying while avoiding the obstacle) or alert table The demonstration may be performed. Also, for example, the positional relationship is as shown in Figure 11. In some cases, the object itself located at the user's specified position may become an obstacle, and similarly, Fail-safe operation (especially when changing the flight path, using 3D point cloud data) (For example, generating routes that circle around the target object or routes that go above it) and alert tables. The display may be made to perform the operation. Furthermore, for example, the onboard imaging unit of the aircraft 4 may determine the color information. If the proportion of a pre-registered color is high (for example, moving in the direction of grass or tree leaves) Even if you register green in anticipation of such a situation, or register brown, which is the color of soil on the ground, Alternatively, the system may perform predetermined fail-safe actions or display alerts.

[0073] <Example of aircraft control 2-3> The control of the aforementioned aircraft 4 is primarily to make the aircraft 4 move in a straight line to a position specified by the user. We anticipate that, for example, when a user specifies a location, the system will use pre-acquired 3D point cloud data. Therefore, if the vehicle moves straight towards the position specified by the user, it will collide with an obstacle before reaching the specified position. If it is possible to determine this, the shape of the obstacle will be converted into a three-dimensional point cloud once the user has specified the location. The system may identify the target from the data and generate a flight path to avoid it. It may also display an alert (for example... You may also include statements such as "we are about to collide with an obstacle" or "we have changed our flight path."

[0074] <Example of aircraft control 3-1> In the above-described embodiment, as an example, the position is determined based on the image captured by the shooting unit 26 of the user terminal 2. The configuration for performing the calculation was explained, but instead, the position is based on the images taken by the camera unit of aircraft 4. It may also be possible to perform a position calculation. This configuration allows objects, including walls, to be specified in any direction. It is particularly effective indoors (for example, in living rooms, warehouses, factories, buildings, etc.) and during flight. From the viewpoint of the camera unit of body 4, the direction of flight of aircraft 4 can be specified subjectively and intuitively. It is possible. For example, if the aircraft 4 is flying to a location specified by the user, By changing the direction of the camera unit and allowing the user to re-specify its position, the direction of flight 4 can be changed. It may be controlled to do so.

[0075] Thus, a position calculation method and an aircraft control method that enable easy on-site control of an aircraft, This will enable us to provide a report processing system.

[0076] The embodiments described above are merely illustrative examples to facilitate understanding of the present invention and do not limit the present invention. This invention is not intended to be interpreted in that way. The present invention may be modified or improved without departing from its spirit. It is possible to do so, and it goes without saying that the present invention includes equivalents thereof. [Explanation of Symbols]

[0077] 1. Management Server 2 User terminals 4 flying objects

Claims

1. A method for calculating a position specified on the user terminal screen, On the screen of the user terminal, specify a predetermined position on the captured image acquired by the shooting unit. The steps, The two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position, and the predetermined When specifying the location, at least the shooting angle, shooting direction, and user terminal location information are required. The steps include receiving shooting status information, Based on the two-dimensional coordinate position and the shooting status information, the user terminal position information indicates the tertiary position From the original coordinate position, the three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the user terminal screen. A step of calculating the angle and direction to, At least the three-dimensional coordinate position indicated by the user terminal location information, and the calculated angle. And the direction is mapped to three-dimensional coordinates in the virtual space, and the three-dimensional position indicated by the user terminal position information When a virtual straight line is extended from the target position to the angle and direction calculated above, the line is closest to the target position. The three-dimensional coordinate position indicated by the initial contact point data corresponds to the position specified by the user. The steps include: calculating a virtual designated location within the virtual space; Based on the correspondence between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculates the three-dimensional coordinate position of the user-specified location in real space from the virtual specified location. The steps include, A method for calculating position characterized by the following features.

2. A method for calculating position according to claim 1, The aforementioned point data is one of the three-dimensional point cloud data in the virtual space. A method for calculating position characterized by the following features.

3. A position calculation method according to claim 1 or 2, The aforementioned point data is stored in the aircraft. A method for calculating position characterized by the following features.

4. A position calculation method according to claim 1 or 2, The aforementioned point data is stored in the user terminal. A method for calculating position characterized by the following features.

5. A position calculation method according to claims 1 to 4, The aforementioned imaging unit is provided in the user terminal. A method for calculating position characterized by the following features.

6. A position calculation method according to claims 1 to 4, The aforementioned imaging unit is installed on the aircraft, A method for calculating position characterized by the following features.

7. An information processing system that calculates a specified position on the screen of a user terminal, The aforementioned information processing system includes an imaging unit, a designated location information acquisition unit, and an imaging status information receiving unit. Equipped with a position calculation unit, The aforementioned designated location information acquisition unit, On the screen of the user terminal, the user specifies the image captured by the camera unit. The two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position is obtained. The aforementioned shooting status information receiving unit is When the predetermined position is specified, at least the shooting angle, shooting direction, and user end Receive shooting status information including last position information, The position calculation unit, Based on the two-dimensional coordinate position and the shooting status information, the user terminal position information indicates the tertiary position From the original coordinate position, the three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the user terminal screen. Calculate the angle and direction to, At least the three-dimensional coordinate position indicated by the user terminal location information, and the calculated angle. And the direction is mapped to three-dimensional coordinates in the virtual space, and the three-dimensional position indicated by the user terminal position information When a virtual straight line is extended from the target position to the angle and direction calculated above, the line is closest to the target position. The three-dimensional coordinate position indicated by the initial contact point data corresponds to the position specified by the user. Calculated as a virtual designated position within the aforementioned virtual space, Based on the correspondence between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculates the three-dimensional coordinate position of the user-specified location in real space from the virtual specified location. do, An information processing system characterized by the following:

Images