Position calculation method and information processing system

Abstract

To provide a position calculation method capable of easily controlling a flying body on a site, and an information processing system.SOLUTION: A position calculation method includes the steps of: designating a prescribed position on a photographic image; acquiring a two-dimensional coordinate position corresponding to the prescribed position and at least photographing state information; calculating an angle and an orientation to a three-dimensional coordinate position corresponding to the two-dimensional coordinate position from a user terminal position on the basis of the two-dimensional coordinate position and the photographing state information; associating at least the user terminal position and the calculated angle and orientation with three-dimensional coordinates in a virtual space, and calculating point data that first comes into contact with a straight line in extending a virtual straight line to the calculated angle and orientation from the user terminal position as a virtual designation position corresponding to a position designated by the user; and calculating a three-dimensional coordinate position of the designation position from the virtual designation position.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a position calculation method and an information processing system. [Background technology]

[0002] In recent years, drones and unmanned aerial vehicles (UAVs) have become increasingly popular. Air vehicles such as the 100mW F-100mW ... In this context, Patent Document 1 describes a method for controlling an aircraft using a dedicated device such as a radio. has been disclosed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-093435 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in the above Patent Document 1 does not allow easy operation using a dedicated device such as a remote control. However, skilled techniques are required to control the aircraft, especially for inspection and other operations. .

[0005] On the other hand, there are methods that specify a flight path in advance and automatically control the aircraft along that flight path. Although there are laws in place, flight routes are usually designated off-site, and there is little flexibility in adapting to local conditions. It was lacking in

[0006] The present invention has been made in view of the above background, and provides a method for easily controlling an aircraft on-site. The object of the present invention is to provide a position calculation method, a flight control method, and an information processing system that can [Means for solving the problem]

[0007] The main invention of the present invention for solving the above problem is to provide a method for displaying a specified position on a screen of a user terminal. A method for calculating a position of a user terminal, the method comprising: displaying a position of a user terminal on a screen of a photographed image acquired by a photographing unit; and specifying a predetermined position on the screen of the user terminal corresponding to the predetermined position. and at least the imaging angle, imaging angle acquiring photographing state information including a photographing angle, a photographing direction, and a user terminal position information; A three-dimensional coordinate position indicated by the user terminal position information based on the original coordinate position and the shooting state information. from the two-dimensional coordinate position on the screen of the user terminal, the angle to the three-dimensional coordinate position and and calculating a three-dimensional coordinate position indicated by at least the user terminal position information. and associating the calculated angle and direction with three-dimensional coordinates in the virtual space, A virtual straight line is drawn from the three-dimensional coordinate position indicated by the user terminal position information to the calculated angle and direction. When the line is extended, the three-dimensional coordinate position indicated by the point data of the first contact with the line is determined as the three-dimensional coordinate position of the user. calculating a virtual designated position in said virtual space corresponding to the position designated by the user; Based on the correspondence between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculate the three-dimensional coordinate position of the specified position specified by the user in the real space from the virtual specified position. and a step of outputting the position calculation result. Effect of the Invention

[0008] According to the present invention, a position calculation method and an information processing system that can easily control an aircraft on-site are provided. It is possible to provide a [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing a configuration of a management system according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram showing a hardware configuration of the management server of FIG. 1. [Diagram 3] 2 is a block diagram showing a hardware configuration of the user terminal of FIG. 1. [Figure 4] FIG. 2 is a block diagram showing the hardware configuration of the aircraft in FIG. 1. [Diagram 5] FIG. 2 is a block diagram showing the functions of the management server of FIG. 1. [Figure 6] 6 is a block diagram showing a structure of a parameter information storage unit in FIG. 5. [Figure 7] 4 is a flowchart of a flight control method according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram illustrating an example of a position calculation method according to an embodiment of the present invention. [Figure 9] FIG. 1 is a diagram illustrating an example of a position calculation method according to an embodiment of the present invention. [Figure 10] FIG. 1 is a diagram illustrating an example of a position calculation method according to an embodiment of the present invention. [Figure 11] FIG. 1 is a diagram illustrating an example of a position calculation method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The contents of the embodiments of the present invention will be listed and described below. The information processing system has the following configuration. [Item 1] A method for calculating a position designated on a screen of a user terminal, comprising the steps of: A predetermined position on the captured image acquired by the image capturing unit is designated on the screen of the user terminal. and A two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position, and At least the shooting angle, shooting angle, shooting direction, and user terminal position information when the position is specified receiving shooting state information including: Based on the two-dimensional coordinate position and the shooting state information, the three-dimensional coordinate indicated by the user terminal position information is A three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the screen of the user terminal from the original coordinate position. calculating an angle and orientation to At least the three-dimensional coordinate position indicated by the user terminal position information and the calculated angle and the direction are associated with three-dimensional coordinates in a virtual space, and the three-dimensional coordinates indicated by the user terminal position information are When a virtual straight line is extended from the target position to the calculated angle and direction, The three-dimensional coordinate position indicated by the first contact point data corresponds to the position designated by the user. calculating a virtual designated position in the virtual space; Based on a correspondence relationship between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculate the 3D coordinate position of the specified position in real space from the virtual specified position and A position calculation method comprising: [Item 2] 2. The position calculation method according to item 1, The point data is one of three-dimensional point cloud data in a virtual space. A position calculation method comprising: [Item 3] 3. The position calculation method according to item 1 or 2, The point data is stored in the aircraft. A position calculation method comprising: [Item 4] 3. The position calculation method according to item 1 or 2, The point data is stored in a user terminal. A position calculation method comprising: [Item 5] A position calculation method according to any one of items 1 to 4, The photographing unit is provided in a user terminal. A position calculation method comprising: [Item 6] A position calculation method according to any one of items 1 to 4, The photographing unit is provided on an aircraft. A position calculation method comprising: [Item 7] An information processing system for calculating a position designated on a screen of a user terminal, The information processing system includes an image capturing unit, a designated position information acquiring unit, an image capturing state information receiving unit, A position calculation unit is provided, The designated position information acquisition unit On the screen of the user terminal, a user-specified image on the captured image acquired by the image capturing unit is displayed. acquiring a two-dimensional coordinate position on a screen of the user terminal corresponding to the predetermined position; The photographing state information receiving unit At least the photographing angle of view, the photographing angle, the photographing direction, the user end, Receive shooting status information including end location information; The position calculation unit is Based on the two-dimensional coordinate position and the shooting state information, the three-dimensional coordinate indicated by the user terminal position information is A three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the screen of the user terminal from the original coordinate position. Calculate the angle and direction to At least the three-dimensional coordinate position indicated by the user terminal position information and the calculated angle and the direction are associated with three-dimensional coordinates in a virtual space, and the three-dimensional coordinates indicated by the user terminal position information are When a virtual straight line is extended from the target position to the calculated angle and direction, The three-dimensional coordinate position indicated by the first contact point data corresponds to the position designated by the user. Calculate the specified virtual position in the virtual space; Based on a correspondence relationship between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculate the 3D coordinate position of the specified position in real space from the virtual specified position do, An information processing system comprising:

[0011] <Details of the embodiment> The following describes a position calculation method, an aircraft control method, and an information processing system according to the embodiments of the present invention. In the accompanying drawings, the same or similar elements are denoted by the same or similar reference characters. In the description of each embodiment, the same or similar elements are designated by the same or similar reference numerals and names. In addition, the features shown in each embodiment may be mutually exclusive. The above is also applicable to other embodiments as long as it does not contradict the above.

[0012] <Configuration> As shown in FIG. 1, the management system according to the present embodiment includes a management server 1 and one or more The system includes a user terminal 2 as shown in FIG. 1, one or more flying objects 4, and one or more flying object storage devices 5. The management server 1, the user terminal 2, the flying object 4, and the flying object storage device 5 are connected to each other via a network. The illustrated configuration is merely an example, and is not intended to be limiting. For example, the flying object may be configured to be carried by a user without having the flying object storage device 5. stomach.

[0013] <Administration Server 1> FIG. 2 is a diagram showing the hardware configuration of the management server 1. Note that the illustrated configuration is This is merely an example, and other configurations may be used.

[0014] As shown in the figure, the management server 1 is connected to a plurality of user terminals 2, an aircraft 4, and an aircraft storage device. The management server 1 is connected to a workstation 5 and constitutes a part of the present system. The computer may be a general-purpose computer such as a personal computer or a cloud-based · may be logically realized by computing

[0015] The management server 1 includes at least a processor 10, a memory 11, a storage 12, a transmission / reception device, and a These are electrically connected to each other via a bus 15.

[0016] The processor 10 controls the operation of the entire management server 1 and transmits and receives data between the various elements. A computing device that controls communication, and processes information required for application execution and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit). it) and / or GPU (Graphics Processing Unit) A program for the present system stored in the storage 12 and deployed in the memory 11 etc. to carry out each information processing.

[0017] Memory 11 is a Dynamic Random Access Memory (DRAM). The main memory is made up of volatile storage devices such as flash memory and HDD (Hard Disk Drive). and auxiliary storage configured with non-volatile storage devices such as a disk drive. 11 is used as a work area for the processor 10 and is also used when the management server 1 is started. BIOS (Basic Input / Output System), which runs on It also stores various setting information, etc.

[0018] The storage 12 stores various programs such as application programs. A database storing data used in the processing may be constructed in the storage 12. stomach.

[0019] The transmission / reception unit 13 connects the management server 1 to the network and the blockchain network. The transmitter / receiver 13 is compatible with Bluetooth (registered trademark) and BLE (Blu-ray Disc). It may be equipped with a short-range communication interface such as EtherNet / IP (Ethos Low Energy). stomach.

[0020] The input / output unit 14 is a device for inputting information such as a keyboard and a mouse, and a device for outputting information such as a display. It is equipment.

[0021] A bus 15 is commonly connected to each of the above elements, and transmits, for example, address signals, data signals, and The control signal is transmitted.

[0022] <User device 2> The user terminal 2 shown in FIG. 3 also includes a processor 20, a memory 21, a storage 22, The device includes a transmitting / receiving unit 23, an input / output unit 24, an image capturing unit 26, and an image capturing state information acquiring unit 27. The components are electrically connected to each other through a bus 25. The functions of each component are the same as those of the management server 1 described above. Since a similar configuration is possible, a detailed description of the similar configuration will be omitted.

[0023] The image capturing unit 26 is, for example, a visible light camera or an infrared camera, and captures a captured image (for example, a still image The system acquires images, video images, etc.

[0024] <Flying object 4> 4 is a block diagram showing the hardware configuration of the flying object 4. Flight controller 41 is a programmable processor (e.g., a central processing unit (CPU)) The device may have one or more processors.

[0025] The flight controller 41 also has a memory 411. The memory 411 can be used to access the flight controller to perform one or more steps. The device stores logic, code, and / or program instructions that are executable by the device. The flight controller 41 also includes an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, The sensors 412 may include an S sensor, a proximity sensor (e.g., a lidar), and the like.

[0026] The memory 411 may be a separate memory such as an SD card or a random access memory (RAM). The image data acquired from the imaging unit / sensors 42 may include a recording medium or an external storage device. The data may be directly transmitted to and stored in the memory 411. For example, The captured still and video data may be recorded in the built-in memory or external memory. At least one of the above is transmitted from the imaging unit / sensor 42 or the built-in memory via the network NW. The photograph may be recorded in either the management server 1, the user terminal 2, or the flying object storage device 5. The unit 42 is mounted on the aircraft 4 via a gimbal 43 .

[0027] The flight controller 41 is a control device (not shown) configured to control the state of the aircraft. For example, the control module may have six degrees of freedom (translational x, y and z, and rotational motion θ x , θ y and θ z ) the spatial arrangement, speed, and / or Or use ESC44 (Electric Speed ​​Control) to adjust the acceleration. The propulsion mechanism (motor 45, etc.) of the aircraft is controlled via the battery 4 A propeller 46 is rotated by a motor 45 supplied with power from 8, thereby generating lift for the aircraft. The control module can control one or more of the following: the status of the mounted part and the sensors. Cut.

[0028] The flight controller 41 may be connected to one or more external devices (e.g., a transceiver (transmitter / receiver) ) 49, a terminal, display device, or other remote control) to transmit data and / or The transceiver 49 can communicate with the transceiver 47 configured to receive the Any suitable communication means may be used, such as wireless communication or the like.

[0029] For example, the transceiver 47 may be connected to a local area network (LAN), a wide area network (WLAN), Network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network One or more of the following can be used: Cut.

[0030] The transmitter / receiver 47 transmits data acquired by the sensors 42 and data generated by the flight controller 41. The processing results, predetermined control data, user commands from a terminal or a remote controller, etc. The device may transmit and / or receive one or more of these.

[0031] The sensors 42 according to the present embodiment include inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (e.g., LiDAR), or vision / imaging sensors (e.g., For example, a camera.

[0032] <Functions implemented in each configuration> FIG. 5 shows the configuration of the management server 1, the user terminal 2, and the flying object 4 according to the present embodiment. FIG. 2 is a block diagram illustrating functions.

[0033] First, as a function implemented in the management server 1, the processor 10 executes flight missions. The storage 12 includes a flight route information storage unit 122, a flight log The transmission / reception unit 13 includes a communication unit 130. The flight mission generation unit 110 includes a flight path generation unit 112. and a storage unit for storing information required for executing a flight mission. For example, information about flight conditions (e.g., flight speed, waypoint spacing, etc.) and a storage unit (not shown) for storing virtual space information (e.g., three-dimensional point cloud data, etc.) described later. Each of the

[0034] The flight mission generation unit 110 generates a flight mission. The option includes at least waypoint information (e.g., latitude and longitude information and flight altitude information). The flight route is information including the flight route of the target object, for example, as described later. 3D point cloud model information, the aircraft's current position, the flight mission start position, and the target object The flight route generation unit 112 automatically calculates and sets waypoints by acquiring the distance, etc. The information about the generated flight route may be stored in a flight route information storage unit (not shown). The flight mission may be stored in the storage device 100. Also, the flight mission may be, for example, a task designated by a user ( For example, imaging, inspection, security, etc.) from the flight mission start position The flying object may be controlled to start the above-mentioned operation. Detailed settings for this purpose may be registered in advance in the management server 1 or the like.

[0035] In addition, the flight path may be, for example, when the aircraft does not have a flying object storage device 5 and hovers near the user. The current position is the position where the drone is waiting or where the user carried the drone. Alternatively, the user may retrieve the aircraft at the flight start position or the flight end position. The information of the flying object storage device 5 managed by the management server 1 (for example, location information and storage status) Based on the flight status information, aircraft storage information, etc., the flight start position, intermediate stopover point, or flight end position The flight path may be generated by including the position of the selected flying object storage device 5.

[0036] The flight route information storage unit 122 stores the flight route information of the flying object generated by the flight mission generation unit 110. The flight log storage unit 124 stores flight route information, for example, The information acquired by the flying object 4 on the flight route set by the It stores information about the locations passed on the way to land, still images, video images, audio, and other information.

[0037] The communication unit 130 communicates with the user terminal 2, the flying object 4, and the flying object storage device 5. The device 130 also functions as a reception unit that receives flight requests from the user terminal 2.

[0038] Next, as a function implemented in the user terminal 2, the processor 20 includes a shooting state information acquisition unit 220, a designated position information acquisition unit 240, and the storage 22 is a storage unit for storing shooting status information. The transmission / reception unit 23 includes a communication unit 230. 22 is a storage unit (not shown) that stores the designated position information acquired by the designated position information acquisition unit 240. It may further comprise:

[0039] The photographing state information acquisition unit 220 includes, for example, a GPS, a gyro sensor, an air pressure sensor, a temperature sensor, The sensor information such as the camera sensor, and the information regarding the imaging unit 26 of the terminal (for example, imaging angle information, etc.) are stored. The captured image is read from a storage unit (which may be a part of the memory 21 or the storage 22) that has been stored in the memory unit. Information on the user's device's field of view, shooting angle, shooting direction, and user device position at the time of acquisition Information (e.g., latitude and longitude information and altitude information) is acquired as shooting status information. The altitude information included in the user terminal location information is calculated based on the above-mentioned air pressure sensor and temperature sensor. The altitude information may be the altitude information provided by the user or the altitude information set by the user. The height information of the user or the average height information according to the user's gender is used to determine the expected user's height. The altitude information may be a value that is offset in the vertical direction by a predetermined height depending on the terminal position.

[0040] The designated position information acquisition unit 240 acquires information on a position on the screen designated by a user (for example, a predetermined position The x- and y-coordinate positions are set with the origin as the width of the screen and the height of the screen as the x-axis and y-axis, respectively. The method for the user to specify a position on the screen is, for example, You can use your finger to touch the touch screen to specify a point or to manipulate objects such as the pointer. Or, the user can move the user terminal 2 to a specific position on the screen (e.g., the center ) and execute the decision action.

[0041] As shown in FIG. 6, the shooting state information storage unit 222 includes a shooting view angle information storage unit 2221, a shooting Angle information storage unit 2222, shooting orientation information storage unit 2223, user terminal position information storage unit 22 24, each of which stores the target shooting condition information.

[0042] The communication unit 230 communicates with the management server 1, the flying object 4, and the flying object storage device 5.

[0043] Next, as a function implemented in the flying object 4, the processor 413 includes a shooting status information receiving unit 4 15 and a position calculation unit 417. The memory 411 includes a virtual space information storage unit 421. The transmitting / receiving unit 47 includes a communication unit 470. The imaging device further includes a storage unit (not shown) for storing the imaging status information received by the status information receiving unit 415. It may be possible.

[0044] The image capture state information receiving unit 415 receives the image capture state information acquired by the image capture state information acquiring unit 220 of the user terminal 2. Shadow state information is received via the communication unit 470.

[0045] The position calculation unit 417 calculates at least the shooting state information of the user terminal 2, the designated position information, the virtual space The virtual space information (e.g., three-dimensional point cloud data) stored in the time information 421, the current position of the aircraft 4, Based on the location information (obtained from a GPS, for example), the user specifies the location on the screen of the user terminal 2. The specified position is calculated as being in either virtual space or real space. The law will be discussed later.

[0046] The virtual space information storage unit 421 stores, for example, three-dimensional point cloud data of a range that can be photographed from the user terminal 2. The 3D point cloud data is stored in the virtual space including the LIDAR (Light Detection and Ranging) sensor data. t Detection and Ranging) and Depth measurement It is three-dimensional coordinate data in a virtual space acquired by a drone or a sensor, and is acquired by flying in advance. Or, the necessary data is acquired from, for example, the management server 1 via the network. In addition, the three-dimensional coordinates of the flying object 4 in the real space and the three-dimensional coordinates in the virtual space may be determined in advance. The landmarks may be associated with each other and stored in the virtual space information storage unit 421.

[0047] In this embodiment, the processor 413 on the aircraft 4 performs position calculations. The above description is based on an example in which the user terminal 2 includes the output unit 417. The processor 20, the processor 10 of the management server 1, the processor of the aircraft storage device 5, etc. The function is to read the information required for position calculation into the relevant processor and perform the calculation. This allows the processing load for position calculation to be borne by a desired processor. can.

[0048] In addition, the position is calculated and the flying object is controlled based on the image captured by the image capture unit 26 of the user terminal 2. However, instead of this, the position is calculated and the flying object is controlled based on the image captured by the imaging unit of the flying object 4. At this time, the flying object 4 may be captured in the same manner as the image captured by the image capturing unit 26 of the user terminal 2. The position calculation may be realized by displaying the captured image of the image capturing unit on the screen of the user terminal 2. When the image is based on the image captured by the image capturing unit 26 of the user terminal 2, the user may While visually checking, the user points the image capturing unit 26 of the user terminal 2 at the location and objectively and intuitively specifies the location. It is possible to control the flying object 4 by setting the When set to , the position is subjectively and intuitively determined, as if the user were moving it directly. By specifying this, it becomes possible to control the aircraft.

[0049] <Example of position calculation method> The position calculation method according to the present embodiment will be described with reference to Figs. 7-10. 8-10 are flowcharts illustrating the position calculation method according to the present embodiment. FIG. 1 is a diagram illustrating an example of a position calculation method according to an embodiment of the present invention. For example, the processor 413 on the aircraft 4 is provided with a position calculation unit 417 as a function thereof, The following describes a configuration in which position calculation is performed based on an image captured by the image capture unit 26 of the user terminal 2.

[0050] First, as shown in FIG. 8, the image captured by the image capturing unit 26 is displayed on the screen of the user terminal 2. A given position on the captured image S is designated with the pointer P or the like (SQ101).

[0051] Next, the flying object 4 receives the image capture status information from the user terminal SQ101 via the image capture status information receiving unit 415. When a specified position (x and y coordinate positions on the screen) and a specified position are specified on the screen in 2 Obtain the shooting status information (shooting angle of view, shooting angle, shooting direction, user terminal position information) of Q102).

[0052] Next, the aircraft 4 calculates the position based on the above-mentioned xy coordinate position and the shooting state information by the position calculation unit 417. Based on this, the position of the user's terminal (three-dimensional coordinates) is converted to a specified position on the screen (x y coordinate position) to the position (three-dimensional coordinate) in the virtual space. Calculate the rank (SQ103).

[0053] More specifically, a method for calculating the angle in the y-axis direction will be illustrated with reference to FIG.

[0054] First, the deviation from the center of the image capture area d y If the origin of the y coordinate on the screen is the top edge, From the specified position y in the y-axis direction on the surface, the height d is determined by the resolution. h The absolute value of the value minus half of The relationship in the following equation 1 holds true.

number

[0055] Next, the deviation angle α from the direction of the imaging unit h Based on this, the relationship in Equation 2 below holds.

number

[0056] Next, the FOV h Based on the half value of , the relationship of the following equation 3 holds.

number

[0057] Here, for example, the formula 3 is substituted into the formula 2 based on the distance, Deviation angle α from the direction of the imaging unit h When the above is rearranged, the relationship in Equation 4 below holds.

number

[0058] At this time, the deviation angle α h Also satisfies the relationship in the following formula 5. do.

number

[0059] Finally, by substituting the obtainable values ​​into Equation 4, the orientation of the imaging part in the y-axis direction is calculated. Deviation angle α from the h is calculated.

[0060] The direction in the x-axis direction is also calculated in the same way. Deviation from the angle α w is calculated.

[0061] Returning to the flowchart of FIG. 7, the position calculation method will be described. Next, the flying object 4 The position calculation unit 417 calculates the position P s and the deviation angle calculated using SQ103 α h As well as deviation direction α w , the position P of the flying object 4 d The user's position is then mapped to three-dimensional coordinates in the virtual space. Position P of user terminal 2 s Deviation angle α h As well as deviation direction α w When a straight line is extended to The point data D (or point cloud model M) that touches the target object is stored at the position (or object) specified by the user. (SQ104).

[0062] More specifically, a method for calculating a position designated by a user will be illustrated with reference to FIG. For simplicity, the x-axis direction will be described first.

[0063] In the left diagram of FIG. 10, the position P s and the deviation azimuth α calculated using the above formula w , the relationship between the object specified on the image and the specified position is shown. and the position P of the aircraft 4 d The position calculation unit 417 of the flying object 4 calculates, for example, a three-dimensional point cloud The data is applied to the mapped virtual space, resulting in the positional relationship shown in the right figure of Figure 10. .

[0064] Here, the position P of the user terminal 2 s Deviation direction α w Extend an imaginary straight line in the direction of The position of the first contact point data D is calculated as the position specified by the user. Angle α h The same thing is done in parallel in the y-axis direction using The three-dimensional coordinate position of the point data in the virtual space corresponding to the position specified by the user is then calculated. (Virtual designated position) is calculated.

[0065] Finally, the position of the flying object 4, P d Based on this, the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space are Calculate the position specified by the user in real space based on the correspondence with the original coordinates (SQ1 In this embodiment, the flying object 4 has virtual space information, so Position P of row 4 d The position in the real space is calculated based on the user terminal 2 in the virtual space. If the aircraft 4 has the position P d There is no need for information about Position P of user terminal 2 s alone may be sufficient.

[0066] As a result, the position designated by the user on the screen of the user terminal 2 is converted into a three-dimensional coordinate in real space. This makes it possible to calculate the target position, which can be used to easily control various flying objects. It becomes.

[0067] <Example of control of flying object 1-1> For example, the above-mentioned three-dimensional coordinates in the real space of a position specified on the screen of the user terminal 2 are used. The control may be performed by controlling the flying object 4 to fly to the position. The process may be executed by the processor 413 or the processor 20 of the user terminal 2, or may be executed by the management server. This may be executed by the processor 10 of the server 1. If the target object is nearby, the user can easily locate it by simply specifying it on the screen of the user terminal 2. It will be possible to fly the flying object 4 up to this point.

[0068] <Example of control of flying object 1-2> In the above-mentioned control example 1-1 of the flying object, the flying object 4 is flown to a position specified by the user. In addition, the system determines the shape of the object including the position and finds the position specified by the user (and (or a position offset by a specified distance from there) to carry out flight missions such as inspections. The shape of the object may be determined by referring to three-dimensional point cloud data or by measuring the shape of the object mounted on the aircraft 4. Alternatively, three-dimensional point cloud data of the object may be acquired on the spot using a LIDAR or the like mounted on the vehicle. For example, the position specified by the user is set as the flight mission start position, and the 3D point cloud of the target object is A flight route may be determined based on the data, and a flight mission may be executed. These processes are also performed by, for example, the management server 1, which records the virtual space information stored in the flying object 4. The flight path may be stored in the storage unit 112 and the flight path may be generated by the flight path generation unit 112. In addition, the flight mission specification is performed by the user before specifying the location. The mission should be specified.

[0069] <Example of control of flying object 1-3> In the above-mentioned control examples 1-1 and 1-2 of the aircraft, the position specified by the user is the triplet of point data. Instead, the object containing the point data is determined, and the object is Identifying the shape of an object from 3D point cloud data and preparing a flight mission for that object The flight path of the flying object may be generated in advance. This allows the user to control the flying object in a position. It is possible to execute a flight mission to a target object without specifying the exact location.

[0070] <Example of control of flying object 1-4> For example, a device capable of communicating with the flying object 4 after flying the flying object 4 to a position designated by a user For example, the device may be an air vehicle storage device 5, and may communicate with the air vehicle 4. The information acquired by the above is transmitted to the management server 1, etc. via the aircraft storage device 5, The communication process is performed to store the aircraft in the storage device 5, and the aircraft is replaced or replaced by the aircraft in the storage device 5. They may also carry out communication processing for carrying out a flight mission together. When the user specifies a position on the screen of the user terminal 2, the user terminal 2 The operation may also be specified in the above.

[0071] <Control Example 2-1 of Flying Object> For example, as described above, When calculating the three-dimensional coordinates of the position in the real space, the user terminal position P s From When the imaginary straight line is extended, it does not contact any of the 3D point cloud data, or If the flying object 4 does not contact the three-dimensional point cloud data even after extending the distance, the flying object 4 will The fail-safe operation may be, for example, an on-the-spot or Finally, the device flies to the location specified by the user or the location of the user terminal 2 and hovers. In addition, an alert may be displayed on the user terminal 2 (for example, a three-dimensional point cloud The device may also display a message indicating that the data is not in contact with the device and that the location cannot be calculated.

[0072] <Example of control of flying object 2-2> For example, as described above, when the flight of the aircraft 4 is being controlled to a position designated by the user, When an obstacle is detected by the camera or sensor mounted on the aircraft 4, a predetermined fail-safe is set. -free operation (stopping in place and hovering, landing, or or the position of the user terminal 2, or If the shape of the obstacle is discernible, you may continue flying to avoid the obstacle) or an alert display In addition, for example, the positional relationship shown in FIG. In this case, the object itself at the position specified by the user may be an obstacle, so Fail-safe operation (especially when changing flight paths) using 3D point cloud data , generating routes around objects or over the sky) and alert tables Furthermore, for example, color information may be determined by a photographing unit mounted on the aircraft 4. If the ratio of pre-registered colors is high (for example, moving in the direction of grass or leaves), Even if you register the color green in case of a fire, or the color brown, which is the color of the ground, In addition, a predetermined fail-safe operation or an alert display may be performed even if the abnormality occurs.

[0073] <Example of control of flying object 2-3> The control of the flying object 4 described above is mainly to make the flying object 4 fly straight to a position specified by the user. For example, when a user specifies a location, the previously acquired 3D point cloud data is used. Therefore, if the vehicle moves straight to the position specified by the user, it will collide with an obstacle before the specified position. If it is possible to determine that the obstacle is located, the shape of the obstacle is calculated based on the 3D point cloud at the stage when the user specifies the position. It is also possible to determine the cause of the problem based on the data and generate a flight route to avoid it. Also, an alert display (for example, For example, a message may be sent to the aircraft indicating that the aircraft is about to collide with an obstacle or that the flight path has been changed.

[0074] <Example of control of flying object 3-1> In the above embodiment, as an example, the position is determined based on the image captured by the image capture unit 26 of the user terminal 2. Instead of this, the position is calculated based on the image captured by the camera of the flying object 4. This configuration allows objects, including walls, to be specified in any direction. It is particularly effective indoors (e.g., in rooms, warehouses, factories, buildings, etc.) and flies The direction of travel of the flying object 4 can be specified subjectively and intuitively from the viewpoint of the camera unit of the flying object 4. It is possible. For example, while flying to a position specified by the user, By changing the direction of the camera and re-specifying the position, the user can change the direction of travel of the flying object 4. The control may be performed so that the

[0075] In this way, the position calculation method and the aircraft control method, which allow for easy on-site control of the aircraft, It is possible to provide an information processing system.

[0076] The above-described embodiment is merely an example for facilitating understanding of the present invention, and does not limit the present invention. The present invention is not intended to be interpreted as being based on the principles of the present invention. It goes without saying that the present invention includes equivalents thereof. [Explanation of symbols]

[0077] 1 Management Server 2. User terminal 4. Flying Objects

Claims

1. A method for calculating a position designated on a screen of a user terminal, comprising the steps of: A predetermined position on the captured image acquired by the image capturing unit is designated on the screen of the user terminal. and A two-dimensional coordinate position on the screen of the user terminal corresponding to the predetermined position, and At least the shooting angle, shooting angle, shooting direction, and user terminal position information when the position is specified receiving shooting state information including: Based on the two-dimensional coordinate position and the shooting state information, the three-dimensional coordinate indicated by the user terminal position information is A three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the screen of the user terminal from the original coordinate position. calculating an angle and orientation to At least the three-dimensional coordinate position indicated by the user terminal position information and the calculated angle and the direction are associated with three-dimensional coordinates in a virtual space, and the three-dimensional coordinates indicated by the user terminal position information are When a virtual straight line is extended from the target position to the calculated angle and direction, The three-dimensional coordinate position indicated by the first contact point data corresponds to the position designated by the user. calculating a virtual designated position in the virtual space; Based on a correspondence relationship between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculate the 3D coordinate position of the specified position in real space from the virtual specified position and A position calculation method comprising:

2. 2. The position calculation method according to claim 1, The point data is one of three-dimensional point cloud data in a virtual space. A position calculation method comprising:

3. 3. The position calculation method according to claim 1, further comprising: The point data is stored in the aircraft. A position calculation method comprising:

4. 3. The position calculation method according to claim 1, further comprising: The point data is stored in a user terminal. A position calculation method comprising:

5. 5. A position calculation method according to claim 1, further comprising the steps of: The photographing unit is provided in a user terminal. A position calculation method comprising:

6. 5. A position calculation method according to claim 1, further comprising the steps of: The photographing unit is provided on an aircraft. A position calculation method comprising:

7. An information processing system for calculating a position designated on a screen of a user terminal, The information processing system includes an image capturing unit, a designated position information acquiring unit, an image capturing state information receiving unit, A position calculation unit is provided, The designated position information acquisition unit On the screen of the user terminal, a user-specified image on the captured image acquired by the image capturing unit is displayed. acquiring a two-dimensional coordinate position on a screen of the user terminal corresponding to the predetermined position; The photographing state information receiving unit is At least the photographing angle of view, the photographing angle, the photographing direction, the user end, Receive shooting status information including end location information; The position calculation unit is Based on the two-dimensional coordinate position and the shooting state information, the three-dimensional coordinate indicated by the user terminal position information is A three-dimensional coordinate position corresponding to the two-dimensional coordinate position on the screen of the user terminal from the original coordinate position. Calculate the angle and direction to At least the three-dimensional coordinate position indicated by the user terminal position information and the calculated angle and the direction are associated with three-dimensional coordinates in a virtual space, and the three-dimensional coordinates indicated by the user terminal position information are When a virtual straight line is extended from the target position to the calculated angle and direction, The three-dimensional coordinate position indicated by the first contact point data corresponds to the position designated by the user. Calculate the specified virtual position in the virtual space; Based on a correspondence relationship between the three-dimensional coordinates in the virtual space and the three-dimensional coordinates in the real space, Calculate the 3D coordinate position of the specified position in real space from the virtual specified position do, An information processing system comprising:

Images