Method and device for displaying information provided by flight vehicles

Abstract

The present invention relates to a method for displaying information provided by flight vehicles, comprising the steps of: dividing a screen into a plurality of individual areas so as to determine a layout; determining the relative positions of the flight vehicles; matching the flight vehicles to the individual areas according to the relative positions of the flight vehicles; displaying, in the matched individual areas, information provided by the flight vehicles; detecting changes in the relative positions of the flight vehicles; and re-matching the flight vehicles to the individual areas according to the changes.

Description

Method and device for displaying information provided by an aircraft

[0001] The present invention relates to a method and device for displaying information provided by an aircraft.

[0002] Specifically, the present invention relates to an interface that displays information provided by an aircraft.

[0003]

[0004] The use of unmanned aerial vehicles (UAVs) has changed and expanded significantly in recent years. This trend is driven by technological advancements, cost reductions, and increasing applicability across various industries.

[0005] Aerial vehicles are also being used in firefighting. When a fire breaks out in a building, an aerial vehicle can photograph the building before firefighters enter, providing current information. Aerial vehicles can easily photograph rooftops and upper floors, providing information about areas otherwise invisible to firefighters.

[0006] Aerial vehicles are also used in the defense sector. When conducting reconnaissance of an operational area, they can provide images of the area before soldiers enter. Aerial vehicles can travel faster than humans and provide images taken from a higher vantage point.

[0007] In general, architectural structures such as bridges and buildings develop defects due to natural collapse or artificial damage over the course of their lifespan, and if the defects progress further, deformation or collapse of the entire structure may occur.

[0008] At this point, multiple aircraft can be operated to photograph the front of the structure and detect defects. Each aircraft captures the structure from different angles, heights, and positions, requiring the pilot or observer to simultaneously monitor multiple screens and identify key information from each image. Monitoring the images transmitted simultaneously from multiple aircraft can be extremely complex.

[0009]

[0010] The present invention seeks to provide a method for displaying information of an aircraft.

[0011] The present invention seeks to provide a method for displaying images received from a plurality of aircraft.

[0012] Specifically, the present invention aims to propose a method for displaying images received from an aircraft by assigning each area of ​​a screen to an aircraft.

[0013]

[0014] A method for displaying information provided by aircraft according to one aspect comprises: a step of dividing a screen into a plurality of individual areas to determine a layout; a step of determining relative positions of the aircraft; a step of matching the aircraft with the individual areas according to the relative positions of the aircraft; a step of displaying information provided by the aircraft in the matched individual areas; a step of detecting a change in the relative positions of the aircraft; and a step of re-matching the aircraft with the individual areas according to the change.

[0015] According to another aspect, a device for displaying information provided by aircrafts comprises: a memory storing at least one instruction; at least one processor executing the at least one instruction; and at least one display device; wherein the processor determines a layout by dividing a screen into a plurality of individual areas, determines relative positions of the aircrafts, matches the aircrafts with the individual areas according to the relative positions of the aircrafts, displays information provided by the aircrafts in the matched individual areas on the display device, detects a change in the relative positions of the aircrafts, and rematches the aircrafts with the individual areas according to the change.

[0016]

[0017] The present invention can change in real time the position at which images received from aircraft are displayed according to the positions of the aircraft.

[0018] Users can intuitively check which aircraft the image displayed on the screen was received from.

[0019] Users can check the relative positions and flight directions of multiple aircraft.

[0020]

[0021] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0022] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0023] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0024] FIG. 4 is a flowchart illustrating a method for displaying information provided by an aircraft according to one embodiment.

[0025] FIG. 5 is a drawing for explaining the layout of a screen according to one embodiment.

[0026] FIG. 6 is a drawing for explaining a method for determining the relative positions of areas according to one embodiment.

[0027] FIG. 7 is a drawing for explaining a method for determining the relative positions of aircraft according to one embodiment.

[0028] FIG. 8 is a drawing for explaining matching aircraft and areas according to one embodiment.

[0029] FIG. 9 is a drawing for explaining a case where the relative positions of aircraft are changed according to one embodiment.

[0030] FIG. 10 is a drawing for explaining a method for displaying information provided by an aircraft according to one embodiment.

[0031] FIG. 11 is a drawing for explaining a method for displaying information provided by an aircraft according to one embodiment.

[0032] FIG. 12 is a flowchart illustrating a method of matching aircraft to individual areas according to one embodiment.

[0033] FIG. 13 is a diagram illustrating a method for determining aircraft to be matched to individual areas among a plurality of aircraft according to one embodiment.

[0034] FIG. 14 is a diagram illustrating a method for displaying aircraft matched to individual areas on a minimap according to one embodiment.

[0035] FIG. 15 is a flowchart illustrating a method for displaying information provided by an aircraft based on a user's selection according to one embodiment.

[0036] FIG. 16 is a diagram illustrating a method for displaying information provided by an aircraft based on a user's selection according to one embodiment.

[0037] Figure 17 is a drawing for explaining the layout of a screen according to one embodiment.

[0038] Fig. 18 is a flowchart illustrating a method for determining the layout of a screen according to one embodiment.

[0039] FIG. 19 is a drawing for explaining a method for determining a layout when aircraft are arranged vertically according to one embodiment.

[0040] FIG. 20 is a drawing for explaining a method for determining a layout when aircraft according to one embodiment are arranged in a horizontal direction.

[0041] FIG. 21 is a drawing illustrating an example of a device for displaying information provided by an aircraft according to one aspect.

[0042]

[0043] The terms and words used in this specification and claims should not be interpreted based on their general or dictionary meanings. In accordance with the principle that inventors can define the concepts of terms and words to best describe their inventions, they should be interpreted in a way that is consistent with the technical concept of the present invention. Furthermore, the embodiments described in this specification and the configurations depicted in the drawings are merely examples of how the present invention can be realized and do not fully represent the technical concept of the present invention. Therefore, it should be understood that various equivalents, modifications, and applicable examples may exist as of the time of filing.

[0044] The terms first, second, A, B, etc. used in this specification and claims may be used to describe various components, but the components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a component could be referred to as a second component, and similarly, the second component could also be referred to as a component. The term "and / or" includes any combination of multiple related listed items or any one of multiple related listed items.

[0045] The terminology used in this specification and claims is for the purpose of describing specific embodiments only and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "comprise" or "have" in this application do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0046] The above-described purposes, features and advantages will become clearer through the following detailed description with reference to the attached drawings, so that a person having ordinary skill in the art to which the invention pertains can easily practice the technical idea of ​​the invention.

[0047] Furthermore, when describing the invention, detailed descriptions of known technologies related to the invention will be omitted if they are deemed to unnecessarily obscure the gist of the invention. Below, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0048] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0049] The aircraft (10) may include any aircraft capable of flight, including a drone, UAV (Unmanned Aerial Vehicle), UAM (Unmanned Aerial Mobility), aircraft, helicopter, etc.

[0050] The aircraft (10) can fly alone or with multiple aircraft capable of collaborating. Furthermore, the aircraft (10) can also collaborate with other types of devices, such as vehicles and robots. Furthermore, the aircraft (10) can automatically fly around the target object (20) along a predetermined flight path, or can manually fly around the target object (20) under user control.

[0051] The aircraft (10) can capture images of the object (20) while flying around the object (20). For example, the captured images of the object (20) can be used to determine whether the object (20) has a defect (e.g., breakdown, damage, etc.). The user can detect, recognize, and / or identify the object (20) through the aircraft (10).

[0052] As an example, the aircraft (10) can photograph the body or blades of a wind turbine. Specifically, the aircraft (10) can fly around the wind turbine to photograph the body or blades. For example, the aircraft (10) can take off from a station at a starting point and fly to the nose of the wind turbine. Then, the aircraft (10) can start photographing from the nose of the wind turbine and can photograph the blades while flying autonomously (or manually) along the blades. Typically, a wind turbine can include three blades, and the aircraft (10) can photograph the three blades to acquire images. At this time, the images photographed by the aircraft (10) can be utilized for inspection of the blades.

[0053] As another example, the aircraft (10) can photograph the surface of a large building. For example, the aircraft (10) can acquire images by photographing the surface while flying around the perimeter of the large building. In this case, the images captured by the aircraft (10) can be utilized for inspection of the surface of the large building.

[0054] As another example, the aircraft (10) can photograph various structures within a military facility. For example, a military facility may include barbed wire fences, buildings, and exterior walls, and the aircraft (10) can fly over various points within the military facility and photograph the surfaces of the structures. In this case, the images captured by the aircraft (10) can be utilized for inspection of the military facility.

[0055] Although wind turbines, bridges, large buildings, and military facilities are depicted as objects (20) in FIG. 1, the objects are not limited thereto. In other words, any structure having a shape may be applied to the objects (20) without limitation. For example, the objects (20) may be structures in the industrial field. The objects (20) may be structures for power generation (e.g., wind turbines, thermal power plants, hydroelectric power plants, nuclear power plants, solar power plants, etc.), large buildings (e.g., factories, exhibition halls, stadiums, etc.), bridges, dams, power lines, roads, etc., but are not limited thereto. As another example, the objects (20) may be structures that must be detected or information collected in the security and military fields (e.g., barbed wire fences, ammunition depots, exterior walls, etc.). As another example, anything that is difficult for a user to inspect entirely with the naked eye, dangerous, or requires a lot of manpower and cost for inspection may be applied to the objects (20).

[0056] At this time, in the present embodiment, the aircraft (10) can be operated as a group consisting of multiple aircraft (10) to monitor the target object (20) in real time. Accordingly, the control system operating the aircraft group can provide the user with real-time images acquired from the aircraft (10).

[0057] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0058] Referring to FIG. 2, the server (30), the controller (40), and the station (50) can independently or jointly control the aircraft (10). For example, the server (30), the controller (40), and the station (50) can control the operation (e.g., movement, rotation, etc.) of the aircraft (10) or control the filming of the aircraft (10).

[0059] The aircraft (10) includes at least one camera, and can capture images of an object (20) using the camera. For example, the camera can be installed at a location advantageous for capturing images during flight of the aircraft (10) (e.g., an area not obscured by a propeller, etc., at the front or lower front of the aircraft).

[0060] For example, the aircraft (10) can fly using a global navigation satellite system (GNSS) and / or an inertial navigation system (INS).

[0061] For example, the aircraft (10) can transmit and receive data with a server (30), a controller (40), and / or a station (50). In addition, the controller (40) and the server (30), the server (30) and the station (50), and the station (50) and the controller (40) can transmit and receive data with each other.

[0062] Here, the data may include data required to control the flight of the aircraft (10), data on a flight image of the aircraft (10), data on an image taken of an object (20) by the aircraft (10), etc.

[0063] A flight image represents the field of view of an aircraft (10) when the aircraft (10) is flying. For example, the flight image may be a dynamic image acquired in real time, but is not limited thereto.

[0064] The image captured by the aircraft (10) of the target object (20) refers to an image captured by the aircraft (10) while flying around the target object (20). At this time, the image captured by the aircraft (10) of the target object (20) can be used as an image to check for defects in the target object (20). In this case, the image captured by the aircraft (10) of the target object (20) can be referred to as an inspection image.

[0065] For example, the flight image may be an image with a relatively low resolution (or a low Ground Sampling Distance (GSD)) compared to the inspection image, and the inspection image may be an image with a relatively high resolution (or a high GSD) compared to the flight image. Meanwhile, depending on the type of aircraft (10), the cameras that generate the flight image and the inspection image may be the same or different.

[0066] A user can control the aircraft (10) using the interface of the controller (40). For example, the controller (40) can generate a control signal based on user input received through the interface and transmit the control signal to the aircraft (10). The controller (40) can transmit the control signal to the aircraft (10) via wireless communication. The control signal may be a signal that controls the flight, attitude, navigation, etc. of the aircraft.

[0067] The aircraft (10) can control the motor to rotate the propeller according to the control signal received from the controller (40). The aircraft (10) can move, rotate, etc. by changing the speed and / or attitude, etc. by the rotation of the propeller. Here, the attitude of the aircraft can be expressed as pitch (Y), roll (X), yaw (Z), etc. In addition, the aircraft can perform photographing of the target object (20), etc. according to the control signal received from the controller (40).

[0068] The controller (40) may further include a display device, and the user may check the flight image and / or inspection image of the aircraft (10) through the display device.

[0069] The controller (40) may be a device on which an application for controlling an aircraft (10) is installed. For example, the device on which the application is installed may be a variety of portable devices such as a smartphone, tablet, smart pad, laptop, or wearable device.

[0070] The server (30) or station (50) can control the aircraft (10) by directly transmitting a control signal to the aircraft. In addition, the aircraft (10) can transmit a flight image and / or an inspection image to the server (30), controller (40), or station (50).

[0071] The aircraft (10), server (30), controller (40), and station (50) can each analyze the inspection image. For example, the analysis of the inspection image may be to determine whether there is a defect in the target object (20) in the inspection image through an algorithm such as machine learning or deep learning. The aircraft (10) may directly determine whether there is a defect in the target object (20), or may transmit the inspection image to the server (30), controller (40), or station (50). The server (30), controller (40), or station (50) may analyze the inspection image received from the aircraft (10) to determine whether there is a defect in the target object (20).

[0072] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0073] Referring to FIG. 3, the aircraft (10) may include a sensor (110), a camera (120), a memory (130), a driving device (140), a communication device (150), and a processor (160). However, the components of the aircraft (10) are not limited to those illustrated in FIG. 3. In other words, the aircraft (10) may include at least one more component in addition to the components illustrated in FIG. 3, or at least one of the components illustrated in FIG. 3 may be excluded.

[0074] The sensor (110) detects various information necessary for the operation of the aircraft (10) (e.g., flight, photography, etc.), such as the aircraft (10) itself, the surrounding environment of the aircraft (10), identification of the target (20), and confirmation of the distance between the aircraft (10) and the target (20). The sensor (110) may include, but is not limited to, a gyro sensor, a barometer, an ultrasonic sensor, a magnetic sensor, an acceleration sensor, a proximity sensor, a lidar, a radar, and / or a GPS sensor.

[0075] For example, a gyro sensor and / or an acceleration sensor can measure the three-axis angular velocity of the aircraft (10). A barometer can measure pressure changes and / or air pressure in the atmosphere around the aircraft (10). Since air pressure varies with altitude, the aircraft (10) can also measure the altitude of the aircraft (10) using the barometer. An ultrasonic sensor can measure the distance between the aircraft (10) and the ground or an object (20). A magnetic sensor is a type of terrestrial magnetism sensor (compass sensor) and can detect geomagnetic information.

[0076] For example, a proximity sensor can measure the proximity state of an object (20) to an aircraft (10), the distance between the aircraft (10) and the object (20), and can include an ultrasonic sensor that can measure the distance to the object (20) from a signal reflected from the object (20) by outputting ultrasonic waves. A GPS sensor can calculate the current coordinates (x, y, z) of the aircraft (10) using GPS signals.

[0077] The sensor (110) may include an attitude and heading reference system (AHSR). For example, the attitude and heading reference system may include an inertial sensor or an inertial measurement unit (IMU). For example, the attitude and heading reference system may include a gyro sensor, an acceleration sensor, and a magnetic sensor, and may fuse sensor values ​​to output attitude values ​​(*?*φ, θ, ψ) of the aircraft (10). Here, the attitude values ​​( ) can be an angle based on three-dimensional coordinates (x-axis coordinate, y-axis coordinate, z-axis coordinate) according to GPS coordinates.

[0078] The camera (120) can capture images of the object (20) according to instructions from the processor (160). For example, the aircraft (10) can include at least one camera, and can include a low-resolution camera and / or a high-resolution camera.

[0079] The camera (120) can be combined with a gimbal whose angle can be adjusted. Accordingly, the shooting angle of the camera (120) can be adjusted by the gimbal.

[0080] The memory (130) may include any non-transitory computer-readable recording medium. As an example, the memory (130) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (130) may store an operating system (OS) and at least one program code.

[0081] These software components may be loaded from a computer-readable recording medium separate from the memory (130). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (130) via a communication device (150) that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory (160) based on a computer program (e.g., a computer program for the processor (160) to perform the operations described below with reference to FIGS. 3 to 12) that is installed by files provided by developers or a file distribution system that distributes installation files of applications via the communication device (150).

[0082] The driving device (140) controls the driving of the motor at a speed and direction according to instructions from the processor (160), and accordingly, the rotational speed and direction of the propeller connected to the motor can be controlled. For example, the driving device (140) may include a motor and a propeller.

[0083] The communication device (150) performs data communication between the aircraft (10) and an external device. For example, the communication device (150) may communicate with the controller (40), server (30), and / or station (50) using various communication methods such as infrared communication, RF (Radio Frequency) communication, Wi-Fi communication, ZigBee communication, Bluetooth communication, laser communication, UWB (Ultra-Wideband) communication, LTE, 5G, 6G, and wireless LAN. However, the communication method employed in the communication device (150) is not limited to the above-described method.

[0084] The processor (160) can process commands of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the commands can be provided from memory (130) or an external device (e.g., a server (30), a controller (40), a station (50), etc.). In addition, the processor (160) can generally control the operations of other components included in the aircraft (10).

[0085] For example, the functions performed by each module included in the processor (160) may be performed by one processor or by separate processors. The processor (160) may perform calculations or data processing related to control and / or communication of at least one other component of the aircraft (10).

[0086] The processor (160) may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor (160) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some environments, the processor (160) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and the like. For example, the processor (160) may also refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination of configurations.

[0087] Meanwhile, the operation of the processor (160) described above with reference to FIG. 3 may be implemented by a separate device. In this case, the separate device (hereinafter referred to as an “image merging device”) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and another external device. When the image merging device is included in at least one of the server (30), the controller (40), the station (50), and another external device, the aircraft (10) may capture static images and dynamic images, and transmit the captured images to the image merging device.

[0088] At this time, in this embodiment, the control system can control a group of aircrafts (10) composed of multiple aircrafts, or display images provided by the group of aircrafts. The control system can be any system including a device capable of processing data, such as a server (30), a controller (40), a station (50), or a command control center. The control system includes a processor (160) that processes data, and the processor (160) displays information provided by the aircraft (10) through a display device. Hereinafter, a method in which the processor (160) processes data and controls the display device to display information of the aircraft (10) will be described.

[0089] The processor (160) needs to accurately identify which aircraft each image received in real time from a plurality of aircraft (10) is from and provide the image to the user. Unlike when the control system displays images captured by a fixed CCTV, in the case of images captured by moving aircraft (10), the positions of the aircraft (10) can change in real time. Therefore, when multiple images are displayed to the user, it is difficult for the user to identify which aircraft (10) captured the image.

[0090] FIG. 4 is a flowchart illustrating a method for displaying information provided by an aircraft according to one embodiment.

[0091] Referring to FIG. 4, the processor (160) can display information provided by the aircraft (10) through a display device.

[0092] At step 410, the processor (160) divides the screen into multiple areas and determines the layout. The layout refers to the method of dividing the screen to display information provided by multiple aircraft (10). The layout may vary depending on the number of aircraft (10), the location of the aircraft (10), the information to be displayed, etc. The processor (160) divides the screen into two or more areas and determines what information to display in each area.

[0093] The processor (160) may determine the layout based on the number of aircraft (10), the screen size, and / or the screen orientation. For example, the processor (160) may determine the layout by dividing the screen into as many sections as the number of aircraft (10). Alternatively, the processor (160) may determine the layout to include an area for displaying a mini-map. Alternatively, the processor (160) may determine the layout based on the ratio of the horizontal and vertical lengths of the screen.

[0094] In this embodiment, a screen may be a physical area or interface that visually displays information to a user on an electronic device such as a computer, mobile phone, or tablet. The screen refers to the area on a display device where images are actually displayed.

[0095] At step 420, the processor (160) determines the relative positions of the aircraft (10). The processor (160) can determine the positions of the aircraft (10) in real time. Since the relative positions of the aircraft (10) may change while the aircraft (10) are flying, the processor (160) determines the relative positions of the aircraft (10) in real time.

[0096] The processor (160) can determine the relative positions of the aircraft (10) based on the east, west, south, and north. The processor (160) can determine the relative east, west, south, and north of the aircraft (10) based on the center points of the aircraft (10). The center points of the aircraft (10) may be intersections of straight lines connecting the aircraft (10). Alternatively, the center points may be the target object (20) being photographed by the aircraft (10). Alternatively, the center points may be the average value of the GPS coordinates of the aircraft (10). Alternatively, the center points may be target points designated by the user.

[0097] The processor (160) can receive GPS coordinates of aircraft (10) and determine the relative positions of the aircraft (10) using the GPS coordinates.

[0098] For example, the processor (160) may preferentially determine aircraft (10) located on the left (west) and aircraft (10) located on the right (east) based on the north-south direction. The processor (160) may determine aircraft (10) located relatively above (north) and aircraft (10) located relatively below (south) among aircraft (10) located on the left (west), and may determine aircraft (10) located relatively above (north) and aircraft (10) located relatively below (south) among aircraft (10) located on the right (east).

[0099] As another example, the processor (160) may preferentially determine an aircraft (10) located above and an aircraft (10) located below (south) based on the east-west direction, and among the aircrafts (10) located above (north), determine an aircraft (10) located relatively to the left (west) and an aircraft (10) located relatively to the right (east), and among the aircrafts (10) located below (south), determine an aircraft (10) located relatively to the left (west) and an aircraft (10) located relatively to the right (east).

[0100] The processor (160) can determine the relative positions of the aircraft (10) based on the user's line of sight direction. When the user is using the controller (40), the processor (160) of the controller (40) calculates the user's line of sight direction based on the direction of the controller (40). The processor (160) determines the relative positions of the aircraft (10) based on the calculated line of sight direction. The direction of the controller (40) can be measured using a sensor included in the controller (40), such as a gyro sensor.

[0101] At step 430, the processor (160) matches (or assigns) aircrafts (10) and areas based on their relative positions. Matching aircrafts (10) and areas may mean that the processor (160) determines an area corresponding to each aircraft (10). In other words, matching aircrafts (10) and areas may mean that the processor (160) determines in which area information provided by each aircraft (10) will be displayed.

[0102] The processor (160) determines the relative positions of the aircraft (10) based on the layout. The processor (160) divides the screen into a plurality of areas. The processor (160) can match the relative positions of the plurality of areas with the relative positions of the aircraft (10). For example, the processor (160) divides the screen into quadrants to determine the relative positions of the areas, and divides the screen into quadrants based on the center points of the aircraft (10) to determine the relative positions of the aircraft (10).

[0103] In step 440, the processor (160) displays information provided by the aircraft (10) in the matched area. The information provided by the aircraft (10) may include images captured by the aircraft (10), analysis results for the images, the location of the aircraft (10), the attitude of the aircraft (10), the field of view of the aircraft (10) and / or the angle of the camera (120), etc. The images include both static and dynamic images. The processor (160) may receive GPS coordinates from the aircraft (10), confirm the location of the aircraft (10), and display the location of the aircraft (10) on the mini-map. The processor (160) may display the direction in which the aircraft (10) is shooting (or the field of view of the aircraft (10)) on the mini-map. In addition, the processor (160) may also distinguish and display aircraft (10) that provide images and aircraft (10) that do not provide images.

[0104] At step 450, the processor (160) detects a change in the relative positions of the aircrafts (10). The processor (160) can receive the GPS coordinates of the aircrafts (10) in real time and confirm the relative positions of the aircrafts (10). For example, an aircraft (10) located in the first quadrant and an aircraft (10) located in the second quadrant may switch positions, and the processor (160) can detect the change in the positions of the aircrafts (10).

[0105] At step 460, the processor (160) re-associates the aircraft (10) and the areas according to the change. The processor (160) re-associates the aircraft (10) and the areas whenever there is a change in the relative positions of the aircraft (10). Re-associating the aircraft (10) and the areas may mean changing the areas that display the information provided by the aircraft (10).

[0106] FIG. 5 is a drawing for explaining the layout of a screen according to one embodiment.

[0107] Referring to FIG. 5, the processor (160) can divide the screen into five areas. The processor (160) can divide the screen into a control area (500) and individual areas (510 to 540).

[0108] The processor (160) can display a minimap in the control area (500) and display the locations of aircraft (10) on the minimap. The minimap refers to a reduced version of the map and indicates the area in which aircraft (10) are flying.

[0109] The processor (160) can divide the right side into two areas (510, 540) and the left side into two areas (520, 530). Images received from aircraft (10) can be displayed in the first to fourth areas (510 to 540).

[0110] The processor (160) can match (or assign) each of the aircraft (10) to each of the first to fourth areas (510 to 540). For example, the processor (160) can match a first aircraft (10) to a first area (510), a second aircraft (10) to a second area (520), a third aircraft (10) to a third area (530), and a fourth aircraft (10) to a fourth area (540). One aircraft (10) can be matched to one area.

[0111] If the positions of the first aircraft (10) and the second aircraft (10) change, the processor (160) matches the second aircraft to the first area (510) and matches the first aircraft to the second area (520).

[0112] Although FIG. 5 illustrates an example in which the control area (500) is located in the center of the screen, the control area (500) may be positioned in various locations, such as the right, left, top, or bottom of the screen. The processor (160) may determine the location of the control area (500) based on the number of aircraft (10) or the locations of the aircraft (10).

[0113] FIG. 6 is a drawing for explaining a method for determining the relative positions of areas according to one embodiment.

[0114] Referring to FIG. 6, the processor (160) can divide the first to fourth regions (510 to 540) into four quadrants based on the X-axis and the Y-axis. The X-axis and the Y-axis can be set in a perpendicular direction to each other with the center of the screen as the origin. The X-axis is an axis that divides the first and fourth regions (510, 540) from the second and third regions (520, 530), and the Y-axis is an axis that divides the first and second regions (510, 520) from the third and fourth regions (530, 540).

[0115] The first area (510) is located in the first quadrant, the second area (520) is located in the second quadrant, the third area (530) is located in the third quadrant, and the fourth area (540) is located in the fourth quadrant.

[0116] In one example, the processor (160) can display a minimap in the control area (500) by setting the Y-axis direction to the north direction. The user can also change the Y-axis direction by making a gesture to rotate the minimap.

[0117] In another example, the processor (160) may display a minimap in the control area (500) by setting the Y-axis direction to the user's line of sight. The processor (160) may calculate the user's line of sight direction through a sensor included in the controller (40). By setting the user's line of sight direction to the Y-axis direction by the processor (160), the user may check the relative positions of the aircraft (10) based on the user's line of sight direction.

[0118] FIG. 7 is a drawing for explaining a method for determining the relative positions of aircraft according to one embodiment.

[0119] Referring to FIG. 7, the processor (160) can determine the quadrants in which the first to fourth aircrafts (710 to 740) are located. The first aircraft (710) is located in the first quadrant, the second aircraft (720) is located in the second quadrant, the third aircraft (730) is located in the third quadrant, and the fourth aircraft (740) is located in the fourth quadrant. Accordingly, the processor (160) matches the first aircraft (710) with the first area (510), matches the second aircraft (720) with the second area (520), matches the third aircraft (730) with the third area (530), and matches the fourth aircraft (740) with the fourth area (540).

[0120] The processor (160) can determine the relative positions of the first to fourth aircrafts (710 to 740) based on the east, west, south, and north. The processor (160) can distinguish the east, west, south, and north using the center point (700) of the first to fourth aircrafts (710 to 740) as the origin. The center point (700) may be the intersection of lines connecting the first to fourth aircrafts (710 to 740), or may be the median (or average) of the GPS coordinates of the first to fourth aircrafts (710 to 740). The north direction may be the direction of the user's gaze.

[0121] Since locations on a map are generally represented according to a geographic coordinate system, on a map displayed in the true north direction, the horizontal axis is an imaginary straight line having the same latitude as the center point (700), and the vertical axis is an imaginary straight line having the same longitude as the center point (700).

[0122] Accordingly, the first aircraft (710) is located in the northeast direction, the second aircraft (720) is located in the northwest direction, the third aircraft (730) is located in the southwest direction, and the fourth aircraft (740) is located in the southeast direction.

[0123] In one example, the processor (160) can preferentially determine aircrafts (720, 730) located on the left side and aircrafts (710, 740) located on the right side based on the north-south direction. Since the relative positions of aircrafts (710 to 740) are determined based on the north-south direction, the left and right sides can be distinguished through the longitudes of the aircrafts (710 to 740). Among the GPS coordinates of aircrafts (710 to 740), aircrafts (720, 730) with relatively small longitude values ​​and aircrafts (710, 740) with relatively large longitude values ​​can be distinguished. The processor (160) can determine a relatively upper-positioned aircraft (720) and a relatively lower-positioned aircraft (730) among aircrafts (720, 730) located on the left, and can determine a relatively upper-positioned aircraft (710) and a relatively lower-positioned aircraft (740) among aircrafts (710, 740) located on the right.

[0124] As another example, the processor (160) may preferentially determine aircrafts (710, 720) located above and aircrafts (730, 740) located below based on the east-west direction, determine an aircraft (720) located relatively to the left and an aircraft (710) located relatively to the right among the aircrafts (710, 720) located above, and determine an aircraft (730) located relatively to the left and an aircraft (740) located relatively to the right among the aircrafts (730, 740) located below.

[0125] FIG. 8 is a diagram illustrating matching aircraft and areas according to one embodiment. Referring to FIG. 8, the processor (160) can match aircraft (710 to 740) with areas (510 to 540).

[0126] The processor (160) determines the relative positions of the aircrafts (710 to 740) and the areas (510 to 540) in the manner described in FIGS. 6 and 7, and matches the aircrafts (710 to 740) with the areas (510 to 540). The first aircraft (710) is matched to the first area (510), the second aircraft (720) is matched to the second area (520), the third aircraft (730) is matched to the third area (530), and the fourth aircraft (740) is matched to the fourth area (540).

[0127] As a result, the aircrafts (710 to 740) displayed on the minimap can be matched to the closest areas on the screen. Accordingly, the user can intuitively determine which aircraft the images displayed in the areas (510 to 540) are from.

[0128] FIG. 9 is a diagram illustrating a case where the relative positions of aircraft have changed according to one embodiment. Referring to FIG. 9, the processor (160) detects a change in the relative positions of aircraft (710 to 740) and re-matches the aircraft (710 to 740) with the areas (510 to 540) according to the change.

[0129] FIG. 9 illustrates an example where the relative positions of a first aircraft (710) and a second aircraft (720) have changed. The first aircraft (710) has moved from the first quadrant to the second quadrant, and the second aircraft (720) has moved from the second quadrant to the first quadrant. Accordingly, the processor (160) detects that the relative positions of the first aircraft (710) and the second aircraft (720) have changed, matches the first aircraft (710) with the second area (520), and matches the second aircraft (720) with the first area (510).

[0130] When the first and second aircraft (710, 720) and the areas (510, 520) are rematched, the processor (160) displays the information provided by the first aircraft (710) in the second area (520) and displays the information provided by the second aircraft (720) in the first area (510).

[0131] FIG. 10 is a diagram illustrating a method for displaying information provided by an aircraft according to one embodiment. Referring to FIG. 10, the processor (160) can display information provided by aircraft (710 to 740) on a minimap.

[0132] In FIG. 10, an example of a processor (160) displaying the positions and fields of view of aircraft (710 to 740) on a mini-map is described. The mini-map may be displayed in a control area (500). The processor (160) receives GPS coordinates from aircraft (710 to 740) and displays the positions of the aircraft (710 to 740) on the mini-map. Additionally, the processor (160) may display a diagram representing the fields of view of the aircraft (710 to 740). For example, the processor (160) may display the field of view (711) of the first aircraft (10).

[0133] FIG. 11 is a drawing for explaining a method for displaying information provided by an aircraft according to one embodiment.

[0134] FIG. 11 illustrates an example in which an aircraft (10) is matched to an individual area (1100) and the processor (160) displays information received from the matched aircraft (10). The processor (160) displays an image received from the aircraft (10) in the individual area (1100) and can additionally display information about the aircraft (10).

[0135] The individual region (1100) includes four sub-regions (1110 to 1140). The individual region (1100) may be one of the first to fourth regions (510 to 540).

[0136] The first partial area (1110) is an area that indicates which aircraft (10) the individual area (1100) is matched to. The processor (160) can display a name, number, figure, etc. representing the aircraft (10) in the first partial area (1110). If the matched aircraft (10) is changed, the processor (160) displays a name, number, figure, etc. representing the changed aircraft (10) in the first partial area (1110).

[0137] The second sub-area (1120) is an area that displays the control mode of the aircraft (10) matched to the individual area (1100). For example, the control mode includes automatic (AUTO), manual (MANUAL), return to home, etc. Automatic indicates a command for the aircraft (10) to fly autonomously, manual indicates a command for the user to control the aircraft (10), and return to home indicates a command for the aircraft (10) to return to a designated location.

[0138] The third section area (1130) is an area that displays the remaining battery capacity of the aircraft (10) matched to the individual area (1100).

[0139] The fourth sub-area (1140) is an area that represents objects identified by the aircraft (10) that are matched to the individual area (1100). Fig. 11 shows an example in which the aircraft (10) identifies two people and one house, and the processor (160) displays the type and number of objects in the fourth sub-area (1140).

[0140] FIG. 12 is a flowchart illustrating a method for matching aircraft to individual areas according to one embodiment. Referring to FIG. 12, the processor (160) can determine aircraft (10) for which information is to be displayed in individual areas when the number of aircraft (10) is greater than the number of individual areas.

[0141] At step 1210, the processor (160) determines the number of aircraft (10) providing information. The processor (160) determines whether it is connected to the aircraft (10) through communication and whether the aircraft (10) provide images and information.

[0142] At step 1220, the processor (160) determines the number of individual regions displaying images received from the aircraft (10). The processor (160) can perform screen layout and determine the number of individual regions included in the screen. One individual region displays an image received from one aircraft (10).

[0143] At step 1230, if the number of aircraft (10) is greater than the number of individual areas, the processor (160) determines aircraft to be matched based on the distance between the aircraft (10) and the target point or object (20). The target point can be designated by the user, and the processor (160) can calculate the distance to the target point or object (20) using the GPS coordinates of the aircraft (10). The processor (160) can determine aircraft (10) to be matched in order of the shortest calculated distance.

[0144] For example, if there are seven aircraft (10) and the number of individual areas is three, the processor (160) must select three aircraft (10). The processor (160) can determine the three aircraft (10) closest to the target point as the aircraft (10) to be matched.

[0145] At step 1240, the processor (160) matches the determined aircraft (10) with individual areas. As described in FIGS. 6 to 8 , the processor (160) matches the aircraft (10) with individual areas based on the relative positions of the determined aircraft (10) and the relative positions of the individual areas. The relative positions may be determined based on a target point or object (20).

[0146] FIG. 13 is a diagram illustrating a method for determining aircraft to be matched to individual areas among a plurality of aircraft according to one embodiment. Referring to FIG. 13 , the processor (160) may calculate the distance between the plurality of aircraft (1310 to 1360) and the target object (1300) to determine aircraft to be matched to individual areas. FIG. 13 illustrates a case where six aircraft (1310 to 1360) provide information to the processor (160).

[0147] When dividing the screen to determine the layout, the processor (160) can determine the number of individual areas. In a case such as FIG. 13, six aircraft (1310 to 1360) provide information to the processor (160), but the number of individual areas may be less than six. In this case, since there are not enough individual areas to match the six aircraft (1310 to 1360), the processor (160) must determine the number of aircraft from among the six aircraft (1310 to 1360) equal to the number of individual areas.

[0148] If the number of individual areas is four, the processor (160) must select four aircraft from among six aircraft (1310 to 1360). The processor (160) can select the four aircraft closest to the target (1300). Accordingly, the processor (160) can select the four aircraft (1310, 1330, 1350, 1360) adjacent to the target (1300).

[0149] The processor (160) can calculate the distance between the target (1300) and the aircrafts (1310 to 1360) using GPS coordinates. The positions of the target (1300) and the aircrafts (1310 to 1360) can be expressed as two-dimensional coordinates, latitude and longitude. The position of the target (1300) can be expressed as the coordinates of the center of the target (1300).

[0150] FIG. 14 is a diagram illustrating a method for displaying aircrafts matched with individual areas on a minimap according to one embodiment. Referring to FIG. 14, the processor (160) can distinguish aircrafts (1410) matched with individual areas and aircrafts (1420) not matched with individual areas and display them on the minimap (1400).

[0151] The processor (160) can distinguish aircraft (1410, 1420) using color. The processor (160) can display aircraft (1410) that are matched with individual areas in black, and display aircraft (1420) that are not matched with individual areas in gray (or translucent).

[0152] As another example, the processor (160) may indicate aircraft (1410) that are matched with individual areas with lines, and aircraft (1420) that are not matched with individual areas with dotted lines.

[0153] The processor (160) can additionally display a field of view (1411) of the aircraft (1410) matched with the individual area. Since the image provided by the aircraft (1410) matched with the individual area is displayed in the individual area, the user needs to check which direction the image received from the aircraft (1410) is shooting. Accordingly, the processor (160) can display the field of view (1411) of the aircraft (1410) on the minimap (1400) to provide the user with the direction in which the aircraft (1410) is shooting.

[0154] FIG. 15 is a flowchart illustrating a method for displaying information provided by an aircraft based on a user's selection according to one embodiment. Referring to FIG. 15 , the processor (160) can determine individual areas in which to display information provided by the aircraft (10) based on the user's selection.

[0155] At step 1510, the processor (160) displays aircraft (10) in the control area and displays information provided by the aircraft (10) in individual areas. The processor (160) may display a mini-map in the control area and display the locations of the aircraft (10) on the mini-map. The processor (160) may display the aircraft (10) as icons and change the shape, color, size, etc. of the icons depending on the status of the aircraft (10).

[0156] At step 1520, the processor (160) identifies an aircraft (10) selected by the user from among the aircrafts (10) displayed in the control area. The user can select the aircraft (10) by touching, clicking, dragging and dropping, or other methods.

[0157] At step 1530, the processor (160) identifies an individual area selected by the user among the individual areas. The user can select an individual area by touching, clicking, dragging and dropping, or other methods.

[0158] At step 1540, the processor (160) displays information provided by the selected aircraft (10) in the selected individual area. If information provided by another aircraft (10) was already being displayed in the individual area selected by the user, the processor (160) displays information provided by the selected aircraft (10) instead of the other aircraft (10). Accordingly, the user can check information about the selected aircraft (10) through the individual area.

[0159] FIG. 16 is a diagram illustrating a method for displaying information provided by an aircraft based on a user's selection according to one embodiment.

[0160] Information provided by the first aircraft (1621) is displayed in an individual area (1612), and information provided by the second aircraft (1622) is not displayed in an individual area.

[0161] The user can move the second aircraft (1622) to the individual area (1612) by dragging and dropping. The processor (160) identifies that the second aircraft (1622) has been selected by the user and that the individual area (1612) has been selected by the user. The processor (160) displays information provided by the second aircraft (1622) in the individual area (1612). The processor (160) displays the second aircraft (1622) in the control area (1611) by changing its appearance from translucent to opaque, and displays the field of view of the second aircraft (1622). The processor (160) displays the first aircraft (1621) by changing its appearance from opaque to translucent, and does not display the field of view of the first aircraft (1621).

[0162] FIG. 17 is a drawing for explaining the layout of a screen according to one embodiment. Referring to FIG. 17, the processor (160) can display a control area (1700) by overlapping individual areas (1710).

[0163] The processor (160) can display individual areas (1710) on the screen and overlap the control area (1700) with the individual areas (1710). The control area (1700) can be displayed in the center of the screen and can be displayed semi-transparently. The processor (160) can adjust the transparency of the control area (1700) and change the position of the control area (1700) according to a user's command.

[0164] Fig. 18 is a flowchart illustrating a method for determining the layout of a screen according to one embodiment.

[0165] At step 1810, the processor (160) receives the positions of the aircraft.

[0166]

[0167] At step 1820, the directionality of the aircraft (10) is determined using the positions of the aircraft (10). The directionality may refer to a horizontal direction or a vertical direction. In other words, the directionality may refer to whether the aircraft (10) are arranged horizontally or vertically. The processor (160) may calculate the directionality of the aircraft (10) based on the east, west, south, north, or the user's viewing direction.

[0168] The processor (10) can receive the GPS coordinates of the aircrafts (10) and determine the direction by comparing the latitude difference and the longitude difference of the aircrafts (10). The processor (160) can compare the largest value among the longitude differences of the aircrafts (10) with the largest value among the latitude differences of the aircrafts (10). In other words, when there are multiple aircrafts (10), the processor (160) can compare the width and height of a rectangle that includes all of the aircrafts (10). If the width of the rectangle is longer than the height, the processor (160) can determine that the aircrafts (10) are arranged in the width direction. Conversely, if the width of the rectangle is shorter than the height, the processor (160) can determine that the aircrafts (10) are arranged in the height direction.

[0169] At step 1830, the processor (160) sets the layout based on the orientation. If the aircraft (10) are arranged horizontally, the processor (160) arranges the control area and individual areas horizontally. Conversely, if the aircraft (10) are arranged vertically, the processor (160) arranges the control area and individual areas vertically.

[0170] FIG. 19 is a drawing for explaining a method for determining a layout when aircraft are arranged vertically according to one embodiment.

[0171] The processor (160) can generate a rectangle (1940) including two or more aircraft (1931, 1932) and calculate directionality by comparing the horizontal length and vertical length of the rectangle (1940). Since the horizontal length of the rectangle (1940) is shorter than the vertical length, the processor (160) can generate a vertical layout.

[0172] The processor (160) divides the screen into three areas (1910, 1921, 1922). The processor (160) sets the control area (1910) and the individual areas (1921, 1922) in a vertical direction. The control area (1910) may be a long rectangle in the vertical direction. The first individual area (1921) and the second individual area (1922) are arranged vertically. The processor (160) matches the first aircraft (1931) to the first individual area (1921) and matches the second aircraft (1932) to the second individual area (1922).

[0173] In FIG. 19, the control area (1910) is placed on the right and the individual areas (1921, 1922) are placed on the left, but the control area (1910) may be placed on the left and the individual areas (1921, 1922) may be placed on the right.

[0174] FIG. 20 is a drawing for explaining a method for determining a layout when aircraft according to one embodiment are arranged in a horizontal direction.

[0175] The processor (160) can generate a rectangle (2040) including two or more aircraft (2031, 2032) and calculate directionality by comparing the horizontal length and the vertical length of the rectangle (2040). Since the horizontal length of the rectangle (2040) is longer than the vertical length, the processor (160) can generate a horizontal layout.

[0176] The processor (160) divides the screen into three areas (2010, 2021, 2022). The processor (160) sets the control area (2010) and the individual areas (2021, 2022) in a horizontal direction. The control area (2010) may be a long rectangle in a horizontal direction. The processor (160) matches the first aircraft (2031) to the first individual area (2021) and matches the second aircraft (2032) to the second individual area (2022).

[0177] In FIG. 20, the control area (2010) is positioned at the bottom and the individual areas (2021, 2022) are positioned at the top, but the control area (2010) may be positioned at the left and the individual areas (2021, 2022) may be positioned at the bottom.

[0178] FIG. 21 is a drawing illustrating an example of a device for displaying information provided by an aircraft according to one aspect.

[0179] Referring to FIG. 21, a device (2100) for displaying information provided by an aircraft includes a processor (2110), a display device (2120), and a memory (2130). In addition to the components illustrated in FIG. 21, one or more components may be included or excluded.

[0180] The memory (2130) can store instructions that enable the processor (2110) to perform various operations described in this document. The processor (210) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations.

[0181] As described above with reference to FIG. 3, the device (2100) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and other external devices. Accordingly, the contents described above with reference to FIG. 3 may be equally applied to FIG. 21.

[0182] The device (2100) can receive information from the aircraft (10), and the processor (2110) can process the information and display it on the display device (2120). The device (2100) can display information provided by a plurality of aircraft (10) on the display device (2120) so that the user can intuitively check it.

Claims

1. In a method of displaying information provided by aircraft, A step of determining the layout by dividing the screen into multiple individual areas; A step of determining the relative positions of the above aircraft; A step of matching the aircrafts and the individual areas according to the relative positions of the aircrafts; A step of displaying information provided by the aircraft in the matched individual areas; A step of detecting a change in the relative position of the above aircraft; and A method comprising the step of re-matching the aircraft and the individual areas according to the above change.

2. In the first paragraph, the step of matching the individual areas is: A method characterized in that the relative positions of the above aircraft are matched with the relative positions of the above individual areas.

3. In paragraph 1, A method characterized in that the relative positions of the above aircraft are determined based on the east, west, south, and north.

4. In paragraph 1, A method characterized in that the relative positions of the above aircraft are determined based on the user's line of sight.

5. In paragraph 1, A method further comprising the step of displaying a minimap indicating the locations of the above aircraft in the control area.

6. In paragraph 5, A method characterized in that the position of the control area is determined according to the number of aircraft.

7. In paragraph 5, A method characterized in that the control area overlaps the individual areas and is displayed semi-transparently.

8. In paragraph 5, A method characterized in that the field of view of the aircraft is further displayed on the minimap.

9. In paragraph 5, A method characterized in that aircraft having a matched individual area and aircraft without a matched individual area are distinguished and displayed on the minimap.

10. In paragraph 1, A method characterized in that the information provided by the aircraft includes at least one of images captured by the aircraft, identification codes of the aircraft, remaining battery capacity of the aircraft, object information identified by the aircraft, and control commands for the aircraft.

11. In a device that displays information provided by aircraft, Memory that stores at least one instruction; At least one processor executing at least one instruction; and At least one display device; The above processor, Determine the layout by dividing the screen into multiple individual areas, Determine the relative positions of the above aircraft, Matching the aircraft and the individual areas according to the relative positions of the aircraft, Displaying the information provided by the aircraft in the above-mentioned matched individual areas on the display device, Detecting changes in the relative positions of the above aircraft, A device for re-matching the aircraft and the individual areas according to the above changes.

12. In the 11th paragraph, the processor, A device characterized in that the relative positions of the above aircraft are matched with the relative positions of the above individual areas.

13. In the 11th paragraph, the processor, A device characterized in that it determines the relative positions of the above aircraft based on the east, west, south, and north.

14. In the 11th paragraph, the processor, A device characterized in that it determines the relative positions of the above aircraft based on the user's line of sight.

15. In the 11th paragraph, the processor, A device characterized in that it displays a minimap indicating the location of the above aircraft in the control area.

16. In the 15th paragraph, the processor, A device characterized in that the location of the control area is determined based on the number of aircraft.

17. In the 15th paragraph, the processor, A device characterized in that the control area is displayed translucently by overlapping the individual areas.

18. In the 15th paragraph, the processor, A device characterized in that it further displays the field of view of the aircraft on the minimap.

19. In the 15th paragraph, the processor, A device characterized in that it displays aircraft with a matched individual area and aircraft without a matched individual area on the minimap.

20. In paragraph 11, A device characterized in that the information provided by the aircraft includes at least one of images captured by the aircraft, identification codes of the aircraft, remaining battery capacity of the aircraft, object information identified by the aircraft, and control commands for the aircraft.

Images