Method for displaying information concerning control of flying body

Abstract

To optimize the display of information including multiple images transmitted from multiple flying bodies.SOLUTION: A method for displaying information, including images provided from multiple flying bodies, includes receiving images transmitted from the multiple flying bodies, generating an image display area corresponding to each of the multiple flying bodies, in which the received images are displayed, and controlling to display an image corresponding to a flying body selected by a user in a first image display area of the image display area corresponding to each of the multiple flying bodies.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for displaying information relating to the control of an aircraft. [Background technology]

[0002] In recent years, drones and unmanned aerial vehicles (UAVs) have been used for various purposes. annunciated aerial vehicles (hereinafter, simply referred to as "aircraft" There are various services available using this service.

[0003] When an aircraft flies, the aircraft data is transmitted from the aircraft to a remote ground station. The signal is received by the ground station and the obtained data is displayed on the monitor screen of a computer installed at the ground station. For example, technology that displays vehicle information, map information, aircraft control panels, and aircraft-mounted camera control panels is For example, it is disclosed in Patent Document 1.

[0004] In addition, when operating multiple drones, the flight data of each drone can be switched on one screen. A technique for displaying images by switching or dividing them is disclosed in, for example, Patent Document 2. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] WO2006-098469 publication [Patent Document 2] Japanese Patent Application Publication No. 6-312700 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology disclosed in the above patent document focuses on the operation of a single or multiple flying objects. This allows the optimization of the monitor screen display of necessary information on a remote device. The system aims to display multiple image information transmitted from multiple aircraft and optimize the image display. No technology has yet been disclosed that can achieve this.

[0007] Therefore, the present invention provides a method for optimizing the display of information including multiple images transmitted from multiple flying vehicles. The purpose is to achieve the following. [Means for solving the problem]

[0008] One embodiment of the present invention is an information display device that displays information including images provided by a plurality of air vehicles. A display method, comprising receiving images transmitted from a plurality of flying objects and displaying the received images. generating image display areas corresponding to each of the plurality of air vehicles; The image corresponding to the flying object selected by the user is displayed on the screen of the image display device. The image is controlled so as to be displayed in a first image display area of ​​the display area. Effect of the Invention

[0009] According to the present invention, the display of information including a plurality of images transmitted from a plurality of flying objects is optimized. It can be achieved. [Brief description of the drawings]

[0010] [Figure 1] 1 is a system configuration diagram of a management terminal and an aircraft according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram showing a system configuration including a management terminal and a user terminal according to a first embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of a management terminal according to the first embodiment of the present invention. [Figure 4]1 is a functional block diagram of an aircraft according to a first embodiment of the present invention. [Diagram 5] 3 is a functional block diagram of a control unit and a storage unit in a management terminal according to the first embodiment of the present invention. FIG. [Figure 6] FIG. 2 is a flow chart showing information display according to the first embodiment of the present invention. [Figure 7] 4 is an example of a user interface displayed on a user terminal according to the first embodiment of the present invention. [Figure 8] 5 is another example of a user interface displayed on the user terminal according to the first embodiment of the present invention. [Figure 9] 10 is an example of a user interface displayed on a user terminal according to a second embodiment of the present invention. [Figure 10] FIG. 13 is a functional block diagram of a control unit and a storage unit in a management terminal according to a third embodiment of the present invention. [Figure 11] FIG. 11 is a flowchart showing information display according to the third embodiment of the present invention. [Figure 12] 1 is a diagram illustrating an example of a screen display of a system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The contents of the embodiments of the present invention will be listed and explained. It has the following configuration. [Item 1] An information display method for displaying information including images provided from a plurality of flying objects, comprising: receiving images transmitted from the plurality of air vehicles; an image display area for displaying the received image, Generate an image display area corresponding to An image corresponding to the flying object selected by the user is displayed corresponding to each of the plurality of flying objects. The first image display area Control how information is displayed. [Item 2] The information display method according to claim 1, The first image display area is an image display area corresponding to each of the other multiple flying objects. A method of displaying information, [Item 3] The information display method according to claim 1, generating a second image area displaying predetermined information related to the predetermined air vehicle; How to show. [Item 4] The information display method according to claim 3, The predetermined information includes the flight path, position information, flight position information, and flight altitude of the corresponding flying object. , flight speed, and battery remaining capacity. [Item 5] The information display method according to claim 1, A method of displaying information that provides a control panel area for controlling the selected flying object. [Item 6] The information display method according to claim 1, An information display method that records images transmitted from the specified flying object. [Item 7] The information display method according to claim 1, The displayed image is transmitted to multiple users. [Item 8] The information display method according to claim 7, receiving a message input from each of the plurality of users; generating a message display area for displaying the received message;

[0012] (First embodiment) The following describes a control method, a flying object, and a flight program according to a first embodiment of the present invention. The following description will be given with reference to the drawings. FIG. 1 shows a first embodiment of the present invention. FIG. 2 is a system configuration diagram consisting of a management terminal and an aircraft.

[0013] As shown in FIG. 1, a management terminal 1 receives a plurality of Connect to aircraft 2A, 2B, 2G, etc. (hereinafter, for convenience, any one of these aircraft will be simply referred to as An example of a network is a local area network (LAN). ), Wide Area Network (WAN), Infrared, Wireless, WiFi, Point-to-Point These include peer-to-peer (P2P) networks, telecommunications networks, and cloud communications. Here, the flying object 2 may, for example, fly autonomously or be remotely controlled by a user terminal described later. An example of such an object is an unmanned aerial vehicle (e.g., a drone) that can be operated and controlled. Predetermined information, for example, still images and video data captured by a camera or the like of the flying object 2, is transmitted to the management terminal 1. In addition to the data, certain control data, such as flight position information (latitude, longitude, altitude, etc.), flight history, Route, battery usage / remaining, flight speed, flight time, acceleration, inclination, and other device operations Status and other information is sent automatically or upon request from the user.

[0014] FIG. 2 shows a system including a management terminal and a user terminal according to a first embodiment of the present invention. As shown in FIG. 2, a management terminal 1 is connected to a network such as the Internet. The terminal is connected to a plurality of user terminals 3A, 3B, 3G, etc. via the Any user terminal is simply called "user terminal 3"). As an example of a network, Area network (LAN), wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) networks, telecommunications networks, Here, the user terminal 3 may be, for example, a workstation or The computer may be a general-purpose computer such as a personal computer, a smartphone, The user terminal 3 may be a portable device terminal such as a tablet. The display device receives predetermined information including image and video data transmitted from the management terminal 1. is displayed using display software such as a built-in browser.

[0015] FIG. 3 is a diagram showing the hardware configuration of the management server 1. This is merely an example, and other configurations may be used.

[0016] As shown in the figure, a management server 1 is connected to a database (not shown) and manages the system. The management server 1 is, for example, a workstation or a personal computer. It can be a general-purpose computer such as It may be logically realized as follows.

[0017] The management server 1 includes at least a control unit 10, a memory 11, a storage 12, a transmission / reception unit 1 3, an input / output unit 14, etc., which are electrically connected to each other via a bus 15.

[0018] The control unit 10 controls the operation of the entire management server 1 and controls data transmission and reception between each element. It is a computing device that performs information processing required for control, application execution, and authentication processing. For example, the processor 10 is a CPU (Central Processing Unit). ) and executes a program stored in the storage 12 and deployed in the memory 11 to Carry out information processing.

[0019] Memory 11 is a Dynamic Random Access Memory (DRAM). The main memory is made up of volatile storage devices such as flash memory and HDD (Hard Disk Drive). and auxiliary storage configured with non-volatile storage devices such as a disk drive. 11 is used as a work area for the processor 10 and is also used when the management server 1 is started. BIOS (Basic Input / Output System), which runs on the Stores various setting information, etc.

[0020] The storage 12 stores various programs such as application programs. A database storing data used in the processing may be constructed in the storage 12. stomach.

[0021] The transmission / reception unit 13 connects the management server 1 to the network and the blockchain network. The transmitter / receiver 13 is compatible with Bluetooth (registered trademark) and BLE (Blu-ray Disc). It may be equipped with a short-range communication interface (e.g., Ethernet Low Energy). .

[0022] The input / output unit 14 is a device for inputting information such as a keyboard and a mouse, and a device for outputting information such as a display. It is equipment.

[0023] A bus 15 is commonly connected to each of the above elements, and transmits, for example, address signals, data signals, and The control signal is transmitted.

[0024] FIG. 4 is a functional block diagram of an aircraft according to a first embodiment of the present invention. The functional block diagram below is stored in a single device (aircraft) for ease of explanation. Although this is described as a concept, for example, some of the functions may be implemented by external devices or It may be logically constructed by utilizing cloud computing technology.

[0025] The flight controller 21 includes a programmable processor (e.g., a central processing unit A processor may have one or more processors, such as a processor unit (CPU).

[0026] The flight controller 21 has a memory 22 and is accessible to the memory. The memory 22 stores instructions that the flight controller 21 executes to perform one or more steps. The memory may store logic, code, and / or program instructions that enable the device to perform a program.

[0027] The memory 22 may be a separable memory such as an SD card or a random access memory (RAM). The information acquired from cameras and sensors27 may include a portable medium or an external storage device. The data may be directly transmitted to and stored in memory. For example, Still images and video data are recorded in the built-in memory or external memory. The camera is mounted on the aircraft. It is installed via cable 28.

[0028] The flight controller 21 is a control module configured to control the state of the air vehicle. For example, the control module contains six degrees of freedom (translational motion x, y, and z, as well as rotational motion θ x , θ y and θ z ) the spatial arrangement, speed, and / or acceleration of the aircraft In order to adjust the speed, the propulsion mechanism (motor 30, etc.) of the aircraft is controlled via the ESC 29. The motor rotates the propeller 31 to generate lift for the flying object. The controller can control one or more of the following: the mounted part, the state of the sensors.

[0029] The flight controller 21 may communicate with one or more external devices, such as a transceiver 25 ( and / or transmit data from a remote control (such as a remote control, terminal, display, or other remote controller) The transceiver 25 can communicate with a transceiver 26 configured to receive the Any suitable communication means may be used, such as wireless communication or the like.

[0030] For example, the transceiver 26 may be connected to a local area network (LAN), a wide area network (WLAN), Network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network One or more of the following can be used: Cut.

[0031] The transmitter / receiver 26 transmits data acquired by the sensors 27 and The processing results, predetermined control data, user commands from a terminal or a remote controller, etc. The device may transmit and / or receive one or more of these.

[0032] The sensors 27 according to this embodiment include inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (e.g., LiDAR), or vision / imaging sensors (e.g., For example, a camera.

[0033] FIG. 5 shows the functions of the control unit and the storage unit of the management terminal according to the first embodiment of the present invention. FIG. 6 is a functional block diagram of a management terminal according to the first embodiment of the present invention. This information display process is a flowchart of the information display process executed in the storage unit. This can be realized by executing a program for displaying information stored in the controller.

[0034] As shown in FIG. 6, first, the information receiving unit 41 of the control unit 10 receives image and video data from the flying object 2. The control unit 10 receives related information such as predetermined control data and / or predetermined data (S101). The above information is stored in the image storage section 51 or the information storage section 52 of the memory section 11.

[0035] Next, the display control unit 44 of the control unit 10 displays the UI template stored in the template storage unit 53. According to the template, the layout of the display area is set in advance and common elements are arranged By arranging the image, video data, text, and other information received from the aircraft 2, the user can A user interface (UI) screen file for displaying on the display device of the user terminal 3 is Generate (S102).

[0036] FIG. 7 shows a user interface displayed on a user terminal according to the first embodiment of the present invention. The user interface screen 60 shows an example of a user interface transmitted from each flying object 2. The display device has a plurality of display areas 61 and 62A-62E for displaying the acquired image and video data. Each display area is provided to display image and video data of the corresponding aircraft. For example, The display area 61 displays image and video data transmitted from the drone 1, and the display area 62 displays image and video data transmitted from the drone 1. A displays image and video data transmitted from drone 2. Similarly, display area 62B displays image and video data transmitted from the drone 3. Here, the display area 61 is The area is used to display the flying object selected by the user. The display area 61 is characterized by having a larger display area than the other display areas 62A-62E. For example, in FIG. 7, the display area 61 is configured to display an aircraft selected by the user via the aircraft selection unit 64. The image and video data received from the selected drone 1 is displayed. Users can now more easily view image and video data received from selected aircraft. The aircraft selection unit 64 can select other drones 2-6 by using a pull-down menu. The aircraft operation panel 63 is arranged so as to overlay the display area 61. Controlling the flight of an aircraft (e.g., controlling the flight direction of the aircraft) in response to requests from the user And control of the camera installed in the aircraft (for example, switching between still and video shooting, zooming in and out) The flight control of the aircraft is performed by the The control program stored in the aircraft control program storage section 57 of the memory section 11 is This is realized by executing the program in the aircraft control unit program execution unit 48 of the unit 10.

[0037] The user interface screen 60 also displays the GPS information transmitted from the currently selected aircraft. A map display area 65 displays the location information and flight route on a map. Information on the control data (e.g., latitude, longitude, altitude, direction, speed, battery level, etc.) The user interface screen 60 further includes a control information display area 66 for displaying control information. is for recording video data transmitted from the selected aircraft and displayed in the display area 61. The user interface screen 60 may also have a recording button 67. A share button 68 for starting a chat program for sharing is provided on the user terminal of the user. Each user can view the shared user interface screen 60. While doing so, you can also enter comments and receive feedback from other users through the chat interface. The chat program stores the chat messages in the memory unit 11. The chat program stored in the chat program storage unit 55 is executed by the control unit 10. The program execution unit 47 executes the program. In the user interface screen, a control panel 6 for the aircraft is displayed in a display area 61. In this case, the original user who shared the user interface screen can control the The user (host user) may assign the authority to operate the operation panel 63 to a specific other user. A selection button (not shown) for setting the number of times may be provided on the user interface screen 60. It is also possible.

[0038] Returning to FIG. 6, the determination unit 43 of the control unit 10 determines whether the management terminal 1 receives a plurality of flight It is determined whether a request to select a desired flying object from among the flying objects is received (S103). As shown in the figure, for example, the user terminal 3 selects an aircraft from the aircraft selection section 64 of the user interface screen 60. Select the desired aircraft by using the pull-down menu. If the result is YES, The control unit 45 displays the image and video data transmitted from the flying object selected by the user terminal 3 on the display area. For example, as shown in FIG. 8, the user terminal 3 displays the drone When a request is made to select 2, image and video data sent from drone 2 is displayed in the display area. 61. Drone 1, which was previously selected, will now be displayed as Drone 2's image. The video data is displayed in the display area 62A where the video data was displayed. By providing an interface screen, image and video data provided by multiple aircraft can be You can display large images and video data of the aircraft you want to focus on while checking the overview. This allows the user to see a more easily viewable display. A table that displays image and video data of the selected aircraft, and a control panel that allows camera operation. By providing the display area, a more user-friendly user interface can be provided. This can be done.

[0039] As can be seen by comparing FIG. 7 and FIG. 8, the map display area 65 includes the current display area 6 The drone 1 shown on the screen is displayed in the center (d1), and the other drones (For example, drone 2:d2) may be displayed on the same screen. Icons that can indicate the direction of loan progress (e.g., asymmetric shapes relative to the direction of progress) The flight history (flight route) may also be displayed. Alternatively, information for avoiding a collision may be presented by displaying the following message (without displaying the message):

[0040] In addition, in the above explanation, each drone is autonomous (having a preset or stored The premise was that the aircraft would fly based on a flight control program, but this form was also Including the above, flight control and camera operation can be performed as follows, for example. i) Flight control: Local operator / Camera operation: Local operator ii) Flight control: Local operator / Camera operation: Operation panel of this system iii) Flight control: Operation panel for this system / Camera operation: Local operator iv) Flight control: Operation panel of this system / Camera operation: Operation panel of this system R In the above cases iii) and iv), flight control is performed using the operation panel of this system. When operating the remote control, make sure that the local operator's control radio is not accepted. However, in an emergency, it is preferable to operate the device by pressing a certain button. It is more preferable to accept a forced interrupt operation from a local operator.

[0041] (Second embodiment) FIG. 9 shows a user interface displayed on a user terminal according to a second embodiment of the present invention. 1 is an example of a screen for selecting a desired flying object. When the image and video data sent from the currently selected aircraft is In the specified display area (see Figs. 7 and 8) where image and video data of another aircraft was displayed Instead of being displayed in the display area 61), the image and video data of the aircraft selected above will be The display area that was previously displayed is expanded. For example, in FIG. When the user terminal 3 selects the drone 2 in the aircraft selection unit 64, The display area 62A, in which the image and video data of drone 2 was displayed, is enlarged and Accordingly, the display area 61 of the image and video data of the adjacent drone 1 is reduced in size, and the drone The display position of the display area 62B for the image / movie data of No. 3 is changed.

[0042] (Third embodiment) FIG. 10 shows a control unit and a storage unit of a management terminal according to a third embodiment of the present invention. FIG. 11 is a functional block diagram of an information processing system according to a third embodiment of the present invention. FIG.

[0043] The functional block diagram of the control unit 10 and the storage unit 11 of the management terminal 1 shown in FIG. The functional block diagram of the control unit 10 and the storage unit 11 in the first embodiment shown in FIG. However, as a feature of this embodiment, the control unit 10 searches for a search object in the storage unit 11. The object detector 49 detects a predetermined object stored in the object storage unit 58. The following are some of the things that can be done:

[0044] As shown in FIG. 11, the information receiving unit 41 of the control unit 10 receives image and video data from the flying object 2. The first reception section 201 receives information such as the data and / or related information such as predetermined control data (S201). The control unit 10 stores the above information in the image storage unit 51 or the information storage unit 52 of the storage unit 11. do.

[0045] Next, the display control unit 44 of the control unit 10 displays the UI template stored in the template storage unit 53. According to the template, the layout of the display area is set in advance and common elements are arranged By arranging the image, video data, text, and other information received from the aircraft 2, the user can A user interface (UI) screen file for displaying on the display device of the user terminal 3 is Generate (S202).

[0046] Next, the determination unit 43 of the control unit 10 determines whether the management terminal 1 is one of the multiple flying objects. 2 determines whether a predetermined object is detected (S203). When the aircraft is searching for a person lost in a mountainous area, each aircraft transmits an image to the management terminal 1. When the video data is transmitted, the management terminal 1 searches the search object stored in the search object storage unit 58. Search image data related to the search target and search image and video data received from the aircraft It is determined whether the subject is included. If the determination result is YES, the display control unit 45 The image and video data of the aircraft that detected the search target person are displayed in the display area 61 in FIG. The user interface screen as in this embodiment is provided (S204). By providing this service, it is possible to check the image and video data provided by multiple aircraft at a bird's-eye view, By displaying the image and video data of the line where a specific object is detected in a large size, It can call attention to.

[0047] In the above-described embodiment, the display area displays camera images of multiple flying objects. However, for example, images from fixed cameras and online teleconferencing, etc. Images captured by communication software through the network, ground vehicles and water It can also display images captured by a moving object equipped with a camera, such as a medium-sized moving object. In this case, the map display area 65 (see FIG. 7) shows the images of each camera (moving body, fixed camera, electronic conference In addition, the location of the terminal (such as a conference terminal) can be displayed from multiple locations using the operation panel. When remote control is performed, the terminals that have the initiative to perform the operation must be able to recognize each other. It is preferable to display a message for switching the initiative together with the message. .

[0048] The aircraft of the present invention is expected to be used as an industrial aircraft for investigations, surveying, observation, etc. In addition, the flying object of the present invention can be used in the aircraft-related industry such as multicopters and drones. The present invention relates to the use of these vehicles and the flight of the vehicles. This can contribute to improving safety.

[0049] The above-described embodiment is merely an example for facilitating understanding of the present invention, and does not limit the present invention. The present invention is not intended to be interpreted as being based on the principles of the present invention. It goes without saying that the present invention includes equivalents thereof. [Explanation of symbols]

[0050] 1. Management terminal 2A~2G Aircraft 3A~3G User Terminal

Claims

1. An information display method for displaying information including images provided from a plurality of flying objects, comprising: receiving images transmitted from the plurality of air vehicles; an image display area for displaying the received image, Generate an image display area corresponding to An image corresponding to the flying object selected by the user is displayed corresponding to each of the plurality of flying objects. and controlling the image display area so that the image is displayed in a first image display area among the image display areas.

2. 2. The information display method according to claim 1, The first image display area includes an image display area for displaying an image corresponding to each of the other plurality of flying objects. A method of displaying information that is larger than an area.

3. 2. The information display method according to claim 1, generating a second image area displaying predetermined information related to the predetermined air vehicle; How to show.

4. 4. The information display method according to claim 3, The predetermined information includes the flight path, position information, flight position information, and flight altitude of the corresponding flying object. , flight speed, and battery remaining capacity.

5. 2. The information display method according to claim 1, A method of displaying information that provides a control panel area for controlling the selected flying object.

6. 2. The information display method according to claim 1, An information display method that records images transmitted from the specified flying object.

7. 2. The information display method according to claim 1, The displayed image is transmitted to multiple users.

8. The information display method according to claim 7, receiving a message input from each of the plurality of users; generating a message display area for displaying the received message;

Images