Aircraft navigation map updating and auxiliary navigation method and storage medium

Abstract

The invention discloses an aircraft navigation map updating and auxiliary navigation method and a storage medium, and the method comprises the steps: in the flight process of an aircraft, displaying a real-time picture and a navigation map in a user interface, the real-time picture displaying an image obtained by an airborne image sensor of the aircraft; in the flight process of the aircraft, first information of a target object in the flight environment is determined; in response to the determined first information, adjusting colors of a plurality of pixel points corresponding to the first information in a navigation map so as to display a first identifier for representing the target object in the navigation map; adjusting the colors of a plurality of pixel points corresponding to the target object in the real-time picture so as to display a second identifier for representing the target object in the real-time picture; and storing the adjusted navigation map, so that the stored navigation map contains the first information of the target object, wherein the first information comprises position information of the target object.

Description

Technical Field

[0001] This disclosure relates to the field of aircraft, and more particularly to a method and storage medium for updating navigation maps and assisting navigation for aircraft. Background Technology

[0002] Currently, users often face complex environments and multiple tasks when operating aircraft. For example, users need to pay attention to changes in weather and terrain during flight, as well as monitor flight status, adjust heading, and manage flight altitude. Therefore, users find it difficult to focus on identifying targets in the flight environment, making it easy for them to miss them. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a method for updating a navigation map of an aircraft, comprising: during the flight of the aircraft, displaying a real-time screen and a navigation map in a user interface, wherein the real-time screen displays images acquired by an image sensor onboard the aircraft; during the flight of the aircraft, determining first information about a target object in the flight environment; in response to the determined first information, adjusting the colors of a plurality of pixels corresponding to the first information in the navigation map to display a first identifier representing the target object in the navigation map; adjusting the colors of a plurality of pixels corresponding to the target object in the real-time screen to display a second identifier representing the target object in the real-time screen; and saving the adjusted navigation map such that the saved navigation map contains the first information about the target object; wherein the first information includes the location information of the target object.

[0004] This embodiment displays an identifier for the target object on both the real-time screen and the navigation map. On the one hand, this serves to remind and warn the user of the target object, providing a better interactive experience. On the other hand, it also allows the user to view and confirm the target object in two different display screens through mutual verification. Furthermore, the location information of the target object can be saved to the navigation map for easy reuse later.

[0005] Secondly, embodiments of this disclosure provide an auxiliary navigation method for an aircraft, comprising: displaying a real-time screen and a navigation map in a user interface during the flight of the aircraft, wherein the real-time screen displays images acquired by an image sensor onboard the aircraft; determining first information about a target object in the flight environment during the flight of the aircraft; adjusting the colors of a plurality of pixels corresponding to the first information in the navigation map in response to the determined first information, so as to display a first identifier for representing the target object in the navigation map; determining second information about the target object during the flight of the aircraft; and adjusting the colors of a plurality of pixels corresponding to the second information in the real-time screen in response to the determined second information, so as to display a second identifier for representing the target object in the real-time screen; wherein the first information includes location information of the target object, and the second information is of a different information type than the location information.

[0006] This embodiment displays identifiers representing target objects on both the real-time screen and the navigation map. On the one hand, this serves to remind and warn users of target objects, providing a better interactive experience. On the other hand, it also allows users to view and confirm target objects in two different display screens through mutual verification. Furthermore, since the two identifiers are determined based on different types of information, it is also beneficial to confirm the authenticity of target objects from different perspectives.

[0007] Thirdly, embodiments of this disclosure provide an auxiliary navigation method for an aircraft, comprising: during the flight of the aircraft, displaying a real-time screen and a navigation map in a user interface, wherein the navigation map displays the flight path of the aircraft and / or terrain information of the flight area of ​​the aircraft, and the real-time screen displays images acquired by an image sensor onboard the aircraft; during the flight of the aircraft, determining first information about a target object in the flight environment; in response to the determined first information, adjusting the colors of a plurality of pixels corresponding to the first information in the navigation map to display a first identifier for representing the target object in the navigation map; and adjusting the colors of a plurality of pixels corresponding to the target object in the real-time screen to display a second identifier for representing the target object in the real-time screen; wherein the first information includes the location information of the target object.

[0008] This embodiment of the disclosure displays an identifier for representing a target object on both the real-time screen and the navigation map. On the one hand, this serves to remind and warn users of the target object, providing a better viewing experience. On the other hand, it also allows users to view the target object in two different display screens in a cross-verification manner to confirm the target object's information.

[0009] Fourthly, embodiments of this disclosure provide an auxiliary navigation method for an aircraft, comprising: displaying a navigation map in a user interface during the flight of the aircraft; detecting the position information of a target object in real time; and adjusting the color of a set of pixels corresponding to the position information of the target object in the navigation map in response to the detected position information of the target object, wherein the set of pixels includes multiple pixels, and as more and more position information of the target object is detected, the number of pixels with changing color continuously accumulates from few to many.

[0010] This embodiment adjusts the color of the set of pixels corresponding to the location information of a target on the navigation map, which can mark the target and thus remind and warn the user of the target. The viewing experience is excellent. Moreover, as more and more location information of the target is detected, the number of pixels with color changes accumulates from few to many. That is, the number of pixels with color changes that the user can observe on the navigation map increases continuously with the increase of the information of the target detected in real time, making it convenient for the user to be aware of the detection and marking process of the target.

[0011] Fifthly, embodiments of this disclosure provide a computer device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, to implement the method described in any one of the first to fourth aspects.

[0012] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any one of the first to fourth aspects.

[0013] In a seventh aspect, embodiments of this disclosure provide an interactive system, including: a user interface device, a processor, a communication device, and an airborne sensor, wherein the user interface device, the communication device, and the airborne sensor are all communicatively connected to the processor, wherein the airborne sensor includes an image sensor and a distance sensor; the user interface device is used to display a real-time image and a navigation map, the real-time image displaying an image acquired by the image sensor; the processor, in response to information about a target object detected by the distance sensor in the flight environment, is used to adjust the color of multiple pixels in the navigation map to display a first identifier representing the target object in the navigation map; the processor is used to adjust the color of multiple pixels in the real-time image to display a second identifier representing the target object in the real-time image; the processor is used to save the adjusted navigation map so that the saved navigation map contains the first information; the processor sends the saved navigation map to an aircraft via the communication device for the aircraft to use for navigation.

[0014] This embodiment displays an identifier for the target object on both the real-time screen and the navigation map. On the one hand, it can remind and warn users of the target object, providing a good interactive experience. On the other hand, it also allows users to view and confirm the target object in two different display screens by mutual verification. Furthermore, the location information of the detected target object can be saved to the navigation map for easy reuse later.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to illustrate the technical solutions of this disclosure.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure.

[0018] Figure 2 This is a flowchart of a method for updating the navigation map of an aircraft according to an embodiment of this disclosure.

[0019] Figure 3 This is a schematic diagram of multiple pixels corresponding to the first information in an embodiment of this disclosure.

[0020] Figure 4 This is a schematic diagram of the user interface of an embodiment of this disclosure.

[0021] Figure 5This is a schematic diagram illustrating the process of accumulating and displaying the number of color-adjusted pixels in a navigation map from a small number to a large number, according to an embodiment of this disclosure.

[0022] Figure 6A , Figure 6B , Figure 6C and Figure 6D This is a schematic diagram illustrating the identification range of a specific identifier in an embodiment of this disclosure.

[0023] Figure 7 This is a flowchart of an aircraft auxiliary navigation method according to an embodiment of the present disclosure.

[0024] Figure 8A This is a flowchart of an auxiliary navigation method for an aircraft according to another embodiment of the present disclosure.

[0025] Figure 8B This is a schematic diagram of a user interface according to another embodiment of this disclosure.

[0026] Figure 9 This is a flowchart of an auxiliary navigation method for an aircraft according to another embodiment of the present disclosure.

[0027] Figure 10 This is a flowchart of an auxiliary navigation method for an aircraft according to yet another embodiment of the present disclosure.

[0028] Figure 11 This is a schematic diagram of an interactive system according to an embodiment of the present disclosure.

[0029] Figure 12 This is a schematic diagram of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0034] Figure 1 A schematic diagram illustrating an application scenario of an embodiment of this disclosure is shown. For example... Figure 1 As shown, this application scenario includes an aircraft 10 and a user terminal 20. The aircraft 10 can be an unmanned aerial vehicle (UAV) or a manned aircraft, and the user terminal 20 can be a smartphone, tablet, or other smart terminal, or a remote control used with the aircraft 10. An image sensor 40 can be mounted on the aircraft 10, either directly or via a gimbal 30. The user terminal 20 can control the flight of the aircraft 10, such as controlling its speed, direction, hovering, takeoff, and return-to-home maneuvers. The user terminal 20 can also control the image capture process of the image sensor 40, such as starting and stopping image capture, controlling its frame rate, and transmission frame rate. The image sensor 40 can capture images of the environment in which the aircraft 10 is located, obtaining at least one image. The captured image may include objects in the environment. The image captured by the image sensor 40 can be sent to the user terminal 20. User terminal 20 may include a user interface 201 for displaying real-time images captured by image sensor 40. Users can view the images captured by image sensor 40 on the user terminal and identify targets in the environment where aircraft 10 is located based on the images. Furthermore, the user interface 201 may also display a navigation map, facilitating users to confirm the real-time location of aircraft 10 and plan its path.

[0035] Understandable. Figure 1The application scenarios shown are merely illustrative and not intended to limit this disclosure. In other examples, the aircraft in the embodiments of this disclosure can also be a manned aircraft, and the user can control the flight process of the manned aircraft and / or the image capturing process of the image sensor recorded on the manned aircraft through the control components on the manned aircraft. The real-time images captured by the image sensor can be sent to a display device on the manned aircraft or a display device on the ground for display. For ease of description, the following uses... Figure 1 Taking the application scenario shown as an example, the solution of this embodiment of the disclosure will be illustrated.

[0036] When a user operates the aircraft 10 via the user terminal 20, they need to pay attention not only to weather and terrain changes during flight, but also to the aircraft 10's flight status, heading, and altitude. Therefore, it is difficult for the user to focus on identifying targets in the flight environment, making it easy to miss them. For example, if there are obstacles in the environment, the user may fail to detect them in time, leading to a collision.

[0037] Based on this, the present invention displays a navigation map and a real-time image acquired by the image sensor 40 in the user interface 201, and marks information related to the target object in the navigation map and the real-time image, thereby improving the user's perception of the target object, reminding and warning the user, and reducing the probability of the user missing the target object.

[0038] Please refer to Figure 2 and combined Figure 1 This disclosure provides a method for updating a navigation map for an aircraft, including:

[0039] Step S11: During the flight of the aircraft 10, a real-time screen and a navigation map are displayed in the user interface 201, wherein the real-time screen displays images acquired by the image sensor 40 on the aircraft 10.

[0040] Step S12: During the flight of the aircraft 10, determine the first information of the target objects in the flight environment;

[0041] Step S13: In response to the determined first information, adjust the color of multiple pixels corresponding to the first information in the navigation map to display a first identifier for representing the target object in the navigation map;

[0042] Step S14: Adjust the colors of multiple pixels corresponding to the target object in the live view to display a second identifier representing the target object in the live view; and

[0043] Step S15: Save the adjusted navigation map so that the saved navigation map contains the first information of the target object;

[0044] The first piece of information includes the location information of the target object.

[0045] In step S11, the user can control the flight of the aircraft 10 and control the onboard image sensor 40 of the aircraft 10 to capture images. For example, the user can... Figure 1 The user terminal 20 sends flight control commands to the aircraft 10 to control its flight process. The user can also send shooting control commands to the image sensor 40 via the user terminal, or send shooting control commands to the aircraft 10, which then forwards the commands to the image sensor 40 so that the image sensor can capture images. Images acquired by the image sensor 40 can be directly transmitted back to the user terminal 20, or transmitted back to the user terminal 20 via the aircraft 10. The user terminal can display real-time footage and a navigation map in the user interface. The real-time footage can display images acquired by the image sensor 40, as well as other information, such as the aircraft 10's direction of movement, real-time speed, real-time location (e.g., latitude, longitude, and altitude), flight time, sensor data, signal strength of communication signals, notification messages, and / or current time. The navigation map can display some or all of the following information: the aircraft 10's real-time location, the aircraft 10's flight path, terrain information of the aircraft 10's flight area, and environmental information of the aircraft 10's environment (e.g., landmarks, buildings, nearby facilities such as shops, restaurants, and gas stations, and / or road information). In some embodiments, the number of image sensors 40 is greater than one, and the real-time display can show images acquired by at least one image sensor 40. For example, the image sensors 40 include a first image sensor for acquiring FPV images, a second image sensor for acquiring fisheye panoramic images, and a third image sensor for acquiring laser liveview images. The real-time display can selectively show at least one of the FPV images, fisheye panoramic images, and laser liveview images.

[0046] In some embodiments, the navigation map and the live view can be displayed synchronously on the user interface 201. For example, the navigation map can be overlaid on a certain area of ​​the live view, or the live view can be overlaid on a certain area of ​​the navigation map, or the navigation map and the live view can be displayed in different display areas of the user interface 201 respectively.

[0047] In some embodiments, the navigation map and the live view can be switched on the user interface 201. For example, the user can switch back and forth between the navigation map and the live view to view the displayed content.

[0048] In step S12, the first information of the target object in the flight environment can be determined in real time. The target object can include, but is not limited to, obstacles or work objects. An obstacle can refer to any object that can obstruct the flight path or safe landing of the aircraft 10, including but not limited to static objects such as buildings and trees, as well as other flying objects in the air. A work object refers to the object targeted by the aircraft 10 when performing a task, such as the object inspected by the aircraft 10. Specifying semantic information can characterize the category (e.g., category of utility pole, building, tree, etc.) or state (e.g., moving or stationary state) of the target object, or characterize whether the target object is an obstacle. In addition, the target object can also be an object of a specified shape, an object whose moving speed meets a preset speed condition, an object whose moving direction meets a preset direction condition, an object whose distance from the aircraft 10 meets a preset distance condition, or an object whose size meets a preset size condition, etc. An object of a specified shape can be a pole-shaped object (e.g., utility pole, traffic signal pole, tree trunk, etc.), a line-shaped object (e.g., wire, rope, etc.), or an object of other shapes. Meeting the preset speed condition means that the moving speed is greater than a preset speed threshold. The movement direction meets the preset direction condition if the angle between the target's movement direction and the aircraft 10's movement direction is within a preset angle range. The distance to the aircraft 10 meets the preset distance condition if the distance between the target and the aircraft 10 is less than a preset distance threshold. The size meets the preset size condition if the target's size is greater than or less than a certain size threshold. Of course, this is just an example, and the above conditions can be other conditions as well. In addition to determining the target based on the above conditions, other conditions can also be used to determine the target, which will not be listed here.

[0049] In step S12, the target object may include an object with preset semantic information. The preset semantic information may be built into the system or user-defined. For example, the semantic information may be represented by point cloud semantics or by visual semantics, and may be determined or iterated through machine learning.

[0050] In some embodiments, the maximum height of the target object is greater than a first preset height. The maximum height of the target object can be the height of the highest point on the target object; for example, when the target object is a utility pole, the maximum height of the target object can be the height of the top of the utility pole. During the flight of the aircraft 10, objects above a certain height (such as airborne obstacles) typically affect the flight of the aircraft 10, while objects at lower heights do not affect the flight of the aircraft 10, or have a minor impact. Therefore, objects with a maximum height greater than the first preset height can be identified as targets. The first preset height can be determined based on the flight altitude of the aircraft 10. In some embodiments, at least a portion of the target object is located within the effective flight range of the aircraft 10.

[0051] The first information about targets in the flight environment can be detected by the ranging module onboard the aircraft 10. The ranging module includes, but is not limited to, at least one of the following: an image sensor (which may include...). Figure 1 The image sensor 40 shown may also include other image sensors, radar sensors, ultrasonic sensors, infrared ranging sensors, and TOF ranging modules. The first information may include the target object's position information, such as the target object's coordinates in a preset coordinate system, or other parameters related to the target object's position. The target object's position information may be absolute position information or relative position information between the target object and the aircraft 10. The aforementioned target object's position information may include the position information of at least one point on the target object, such as the position information of the target object's center point, or the position information of the target object's two endpoints, or the position information of multiple points on the target object's edge contour. In some embodiments, the aforementioned position information includes the distance information of the target object relative to the aircraft 10. In addition to including position information, the first information may also include other information, such as the target object's semantic information, size information, and / or shape information.

[0052] In addition, other devices (e.g., other aircraft around aircraft 10) can also detect the first information of targets in the flight environment and send the first information of the detected targets to aircraft 10.

[0053] In step S13, the colors of multiple pixels corresponding to the first information can be adjusted in the navigation map to obtain an adjusted navigation map. When the first information includes location information, the multiple pixels in the navigation map corresponding to the first information may include multiple pixels in the navigation map representing some or all of the location information included in the first information (denoted as the first pixel). For example, assuming the first information includes the coordinates of multiple points on the target object, denoted as {(x1,y1),(x2,y2),…,(xn,yn)}, then the multiple pixels in the navigation map corresponding to the first information may include the pixel representing the location information at coordinates (x1,y1), the pixel representing the location information at coordinates (x2,y2),…, and the pixel representing the location information at coordinates (xn,yn). Furthermore, the multiple pixels in the navigation map corresponding to the first information may also include pixels surrounding the first pixel (denoted as the second pixel), for example, pixels obtained after boundary expansion processing of the edge contour formed by the first pixel, and / or pixels on the border of the first pixel. Figure 3As shown, assuming the target object is a linear object, the first information may include the position information of multiple points on the linear object. Then, the multiple pixels in the navigation map corresponding to the first information may include the first pixel in the navigation map used to represent the position information of the multiple points on the linear object (as shown by the red line segment in the figure), the second pixel obtained after boundary expansion processing of the edge contour of the first pixel (as shown by the yellow line segment in the figure), and the second pixel on the border of the first pixel (as shown by the blue broken line in the figure).

[0054] By adjusting the colors of multiple pixels corresponding to the first piece of information, a first identifier representing the target object can be displayed on the navigation map. To make the first identifier more prominent and easier for users to observe, the color difference between the adjusted and unadjusted colors of the aforementioned pixels can exceed a preset color difference threshold. In addition to adjusting the pixel colors, visual features such as pixel transparency, brightness, and / or contrast can also be adjusted simultaneously, thereby displaying a first identifier with a specific color, transparency, brightness, and / or contrast on the navigation map. By adjusting the multi-dimensional visual features of the pixels, the first identifier can be further highlighted, thus providing a better reminder and warning effect.

[0055] In some embodiments, the visual features of the first identifier can be static or dynamically changing. For example, in an example where the colors of multiple pixels corresponding to the first information are adjusted to display the first identifier, the adjusted color can be a fixed single color, such as red or yellow, or a fixed combination of colors (such as a gradient), or a color that changes dynamically over time, such as cycling between red, orange, and yellow at preset time intervals, or changing from one color to another when a preset condition is met. Similarly, when other visual features are adjusted simultaneously, these other visual features can also be static or dynamically changing.

[0056] In some embodiments, the second identifier can be obtained through semantic information, such as machine learning based on the texture, outline, color, and other feature information of the target object. For example, the target object's outline can be highlighted, the target object's outline can be selected for emphasis, the surface feature points of the target object can be recolored, or the background outside the target object can be weakened, etc., as long as it can provide users with visual differentiation and recognition, no special limitations are imposed in this application.

[0057] In some embodiments, the number of targets may be greater than or equal to one. The first identifiers of different targets may be the same or different. For example, semantic information of each target can be obtained; targets with different semantic information may have different first identifiers, thus facilitating user differentiation of the semantic information of each target. Another example is obtaining distance information of each target relative to the aircraft 10, and determining the distance range between the target and the aircraft 10 based on this distance information. The first identifiers of targets within different distance ranges may be different, thus facilitating user differentiation of the distance ranges in which each target is located. Furthermore, information such as the shape or size of the target can be obtained; targets with different shapes or sizes may have different first identifiers. Alternatively, the first identifiers generated when the aircraft 10 is in different flight segments or different headings may be different. Other methods can also be used to generate the first identifiers of the targets, which will not be listed here.

[0058] In some embodiments, as the amount of first information detected increases, the number of pixels in the navigation map that are color-adjusted accumulates from a small number to a large number. For example... Figure 5 As shown, the red line segments represent the pixels whose color is adjusted in the navigation map. It can be seen that the length of the red line segment at time T1 is relatively short, indicating that the number of pixels whose color is adjusted in the navigation map at time T1 is relatively small. At time T2, following time T1, as the amount of first information detected increases, the length of the red line segment at time T2 is longer than that at time T1, indicating that the number of pixels whose color is adjusted in the navigation map at time T2 is greater than that at time T1. Similarly, at time T3, following time T2, the length of the red line segment is longer than that at time T2, indicating that the number of pixels whose color is adjusted in the navigation map at time T3 is greater than that at time T2. Through this method, users can clearly perceive the detection and marking process of the target object.

[0059] It is understood that the above methods are merely illustrative. In other examples, the number of pixels with color adjustments in the navigation map may remain constant throughout the entire target detection process. For instance, in response to the first information of target detection, a first identifier of a fixed pattern may be displayed on the navigation map, and the number of pixels occupied by this first identifier of a fixed pattern in the navigation map may remain constant throughout the entire target detection process.

[0060] In some embodiments, the first identifier displayed in the navigation map is location information of a specific target object within the navigation map. The specific target object refers to an object whose second information satisfies preset condition two and / or an object whose first information satisfies preset condition one. In some embodiments, the second information satisfying preset condition two may include preset semantic information. For example, the preset semantic information may be "obstacle." That is, if the second information of a target object includes the semantic information "obstacle," the target object can be identified as a specific target object. The first information satisfying preset condition one may include preset semantic information in addition to location information. Again, taking "obstacle" as an example, if the first information of a target object includes the semantic information "obstacle," the target object can be identified as a specific target object. It is understood that the above preset semantic information is merely illustrative; in other examples, the preset semantic information may also be other information, such as more granular information like trees, buildings, or utility poles.

[0061] In the example where the first identifier displayed in the navigation map is a first identifier used to represent a specific target object, the first identifier used to represent the specific target object can be displayed in response to the user's selection operation of a specific target object included in the target objects in the real-time view. In this embodiment, the selection can be a positive selection of a specific target object or a negative selection of other targets besides the featured target object, as long as it can achieve the operation of selecting a specific target object. In some cases, the image sensor 40 senses multiple targets, and all multiple targets are identified in the real-time view. Since the user can observe and understand each target object more intuitively in the real-time view, the user's selection operation of a specific target object included in the target objects in the real-time view selects the specific target object that the user wants to be identified, thereby causing the navigation map to display a second identifier for the specific target object.

[0062] In some embodiments, the navigation map also displays a third identifier. This third identifier can be used to represent any of the following information:

[0063] The first information detected at historical moments; by identifying the first information detected at historical moments, it helps users review the historical operation of the aircraft 10 and facilitates comparison between the first information acquired in real time and the first information detected in history.

[0064] The first information detected by the aircraft 10 when performing other segments different from the current segment; by identifying the first information detected by the aircraft 10 when performing other segments, it helps the user compare the operation of the aircraft 10 in different segments.

[0065] The first information detected by other devices besides aircraft 10; by identifying the first information detected by other devices, it is convenient for users to compare and analyze the first information from different sources.

[0066] To facilitate differentiation between different logos, the display style of the third logo can differ from that of the first logo; that is, the first and third logos can be displayed using different styles. For example, the colors, borders, backgrounds, transparency, and / or animation effects of the first and third logos can be different.

[0067] In step S14, the colors of multiple pixels corresponding to the target object can be adjusted in the real-time image. For example, in response to determining the second information of the target object, the colors of multiple pixels corresponding to the second information can be adjusted in the real-time image. The second information can be detected by sensors on the aircraft 10 (e.g., image sensor 40), or it can be detected by other devices and sent to the aircraft 10.

[0068] In some embodiments, the second information is of a different type than the first information. For example, the first information includes the location information of the target, while the second information includes the semantic information of the target. Exemplarily, the second information can be obtained from data detected by the image sensor 40, or from data detected by the distance sensor of the aircraft 10. This allows for multi-source cross-validation between different types of information, better determining the authenticity of the target and avoiding false detections or limitations caused by a single information source. Exemplarily, the distance sensor can be a millimeter-wave radar, lidar, etc.

[0069] The multiple pixels corresponding to the target object in the real-time image can include the pixel representing the target object (denoted as the third pixel), i.e., the pixel whose semantic information is consistent with that of the target object, or the pixels surrounding the third pixel (denoted as the fourth pixel), such as pixels obtained after boundary dilation processing of the edge contour formed by the third pixel, and / or pixels on the border of the third pixel. The semantic information of each pixel in the image can be determined by recognizing the image acquired by the image sensor 40, and the third and fourth pixels can be determined based on the semantic information of each pixel in the image.

[0070] By adjusting the colors of multiple pixels corresponding to the target object, a second identifier representing the target object can be displayed in the real-time image. This second identifier can be generated based on detection data from an image sensor or from a distance sensor on the aircraft 10. For example, the relative position of the target object to the aircraft 10 can be detected by the distance sensor, and based on the relative position and the mapping relationship between the navigation map and the real-time map, a second identifier can be roughly marked in the real-time image. Since the information sources of the second identifiers generated based on different types of sensors are different, the display styles of the second identifiers generated based on image sensor detection data and those generated based on distance sensor detection data can be different in the real-time image, thus facilitating user differentiation of the data source for generating the second identifier.

[0071] In the example where the second information is detected by image sensor 40, the second information detected by image sensor 40 can be used to display the second identifier in a real-time frame including the image acquired by image sensor 40. Further, the second information detected by one image sensor 40 can be used to display the second identifier in a real-time frame including the image acquired by another image sensor 40. The second information detected by one image sensor 40 can be mapped to the coordinate system of the other image sensor 40 based on the mapping relationship between the two image sensors 40, and the second identifier can be displayed in a real-time frame including the image acquired by the other image sensor 40 based on the mapped second information. For example, if image sensor 40 includes a first image sensor for acquiring a real-time visible light frame and a second image sensor for acquiring a fisheye view frame, the second information detected by the first image sensor can not only be used to display the second identifier in the real-time visible light frame, but can also be mapped to the fisheye view frame and used to display the second identifier in the fisheye view frame. The above method achieves the sharing of second information across multiple frames.

[0072] Figure 4 An exemplary schematic diagram of a user interface 201 is shown. Figure 4As shown, the real-time view is displayed in full screen in the user interface 201, and the navigation map is overlaid in the lower left corner of the real-time view. The first identifier displayed in the navigation map includes a first pixel (shown as the red line segment in the figure) representing the position information of multiple points on the target object, a second pixel obtained by boundary expansion processing of the edge contour of the first pixel (shown as the yellow line segment in the figure), and a second pixel on the border of the first pixel (shown as the blue broken line in the figure). The second identifier displayed in the real-time view includes a third pixel representing the target object (shown as the purple line segment in the figure) and a fourth pixel on the border of the third pixel (shown as the green broken line in the figure). It should be understood that the display method shown in the figure is merely illustrative and is not intended to limit this disclosure.

[0073] By displaying corresponding markers in both the navigation map and the live view, users can easily view and confirm targets in two different displays through cross-verification. For example, users can edit the first marker on the navigation map based on the second marker in the live view. These editing operations include at least one of the following: expanding the marker's range, deleting a portion of the first marker, selectively saving a portion of the first marker, or confirming the saving of the first marker. For instance, a user can expand the marker's range to provide a safety margin for the saved marker; a user can delete some incorrectly marked markers; a user can selectively save a portion of the first marker instead of all of it; and a user must manually confirm before saving. Thus, by granting users access to these controls, the saving process allows for customized editing, ensuring that the updated navigation map better meets user needs.

[0074] In some embodiments, users can determine whether the first identifier in the navigation map is accurate and complete based on the content displayed in the live feed. This is because the live feed is more intuitive, easier for users to understand, and provides an immersive experience. For example, if the navigation map displays a first identifier representing a target object, but the live feed does not display a second identifier representing that target object, the user can verify the accuracy of the first identifier displayed in the navigation map. If it is inaccurate, it indicates that the detected target object's first information was misdetected, leading to the navigation map incorrectly marking a first identifier that should not have been marked. Therefore, the user can delete the first identifier from the navigation map. Deleting the first identifier allows for the deletion of corresponding first information, thereby improving the accuracy of the first information. As another example, users can also delete or complete some pixels in the first identifier based on the second identifier in the live feed. Through these methods, the verification of the first identifier and the first information is achieved, improving their accuracy.

[0075] It should be noted that this disclosure does not restrict the execution order of steps S13 and S14. Step S13 can be executed before or after step S14, or it can be executed in parallel with step S14.

[0076] In step S15, the adjusted navigation map can be saved. The saved navigation map contains the first information of the target object and can display the first identifier. That is, the first information of the target object can be recorded in the adjusted navigation map, and the updated navigation map can be saved. The adjusted navigation map is obtained by adjusting the colors of multiple pixels corresponding to the first information.

[0077] Taking slender obstacles commonly found in aircraft flight environments, such as power lines, as an example, these obstacles are difficult to detect and identify due to their small cross-sectional area and crisscrossing patterns, severely impacting flight safety. Improving the efficiency and safety of bypassing slender obstacles has always been a major challenge in the industry. In related technologies, during the first detection, when the aircraft 10 flies along a certain heading (e.g., roughly perpendicular to the direction of the power line), the relatively large reflective surface of the detection echo makes it relatively easy to detect the position of the power line. However, during the second detection after the aircraft 10 changes its heading (e.g., roughly parallel to the direction of the power line), the relatively small reflective surface makes it difficult to observe the target, leading to the power line being missed and causing a safety incident. This embodiment addresses this by saving the adjusted navigation map. The initial information of the power line detected in the first detection can be saved in the navigation map for easy reuse later. Even if the power line is not identified in the second detection, obstacle avoidance can be performed based on the existing initial information of the power line in the navigation map, thereby reducing the chance of missed detection due to heading changes and improving flight safety.

[0078] Furthermore, when saving the adjusted navigation map, the land parcel to which the target object belongs can be determined, and the target object can be stored or decoupled from its corresponding land parcel and saved in the adjusted navigation map. Figure 1 Save together.

[0079] Furthermore, the first identifier can be an automatically generated identifier based on the first information, or it can be an identifier edited by the user. An option for a semantic object can be added, such as "user-added target object" or "automatically identified target object," making it easier for users to filter specific first information when subsequently accessing the navigation map.

[0080] In some embodiments, the saved navigation map contains first information about a specific target. The specific method for determining the specific target can be found in the foregoing embodiments and will not be repeated here. Optionally, in step S13, the first identifiers of all targets can be marked, and in step S15, the first identifiers of the specific target can be filtered out and saved, while the first identifiers of other targets are not saved. Alternatively, in step S13, only the first identifiers of the specific target can be marked, and in step S15, all the first identifiers marked in step S13 can be saved.

[0081] In some embodiments, the adjusted navigation map can be saved based on preset user actions and / or preset rules.

[0082] In an embodiment where the adjusted navigation map is saved based on preset user operations, the user operations include editing operations performed by the user on a first identifier that needs to be saved. These editing operations include any of the following: expanding the identifier range of the first identifier, deleting a portion of the first identifier, selectively saving a portion of the first identifier, or confirming the saving of the first identifier. Expanding the identifier range of the first identifier means increasing the coverage area of ​​the first identifier on the navigation map, so that more pixels are included within the identifier range, thereby improving visibility and recognizability. For example, the first identifier can be manually supplemented to obtain a more complete first identifier, or its boundaries can be appropriately expanded through operations such as horizontal stretching. Deleting a portion of the first identifier means deleting some pixels from the first identifier. For example, pixels with inaccurate labeling can be deleted. Selectively saving a portion of the first identifier means saving only a portion of the pixels on the first identifier. The confirming the saving of the first identifier can be a selection operation performed by the user on the option to confirm saving the first identifier, which can be triggered by physical buttons, virtual buttons, gesture control, or voice control.

[0083] In practical applications, there can be one or more target objects. When there are multiple target objects, user operations can include selection operations on the real-time screen and / or navigation map for target objects to be saved. This selection operation can be a forward selection operation or a reverse selection operation. A forward selection operation refers to selecting a target object to be saved. After the user performs a forward selection operation on a target object, the first identifier and first information corresponding to that target object can be saved to the navigation map. A reverse selection operation refers to selecting target objects that do not need to be saved. After the user performs a reverse selection operation on a target object, it can be determined that the first identifier and first information corresponding to that target object do not need to be saved to the navigation map, and the first identifier and first information corresponding to the unselected target objects are saved to the navigation map.

[0084] In an embodiment where the adjusted navigation map is saved based on preset rules, the preset rules include at least one of the following:

[0085] The first information within a preset range can be selectively saved. The preset range can be a range where the distance to the aircraft 10 does not exceed a preset distance threshold. Since the detection accuracy of targets that are too far away is generally low, saving only the first information within the preset range can improve the accuracy of the first information. Alternatively, the preset range can be a range of altitudes within a preset altitude interval. During flight, only objects within a specific altitude interval typically affect the flight process, while objects outside this interval do not overlap with the flight or operational coverage area of ​​the aircraft 10, thus having less impact on the flight process or operational tasks. Therefore, only the first information within a specific altitude interval can be labeled. Other preset ranges can also be determined based on actual needs; these will not be listed here.

[0086] Selectively save the first information of a target object with preset semantic information; the preset semantic information can be default semantic information or semantic information specified by the user. In some embodiments, the preset semantic information can be "obstacle," that is, if the semantic information of a target object is identified as an obstacle, the first information of that target object can be saved. Of course, in other examples, the preset semantic information can also be more specific semantic information, such as trees, buildings, utility poles, etc.

[0087] The adjusted navigation map synchronously stores other information used to represent the target object; this other information includes: timeliness information, semantic information, confidence information, real-time video information, detour information, and / or user operation information. Specifically, timeliness information represents the effective time of the first information, and may only display the first identifier on the navigation map during the time period indicated by the timeliness information, while not displaying it at other times; semantic information represents the category, state, and / or whether the target object is an obstacle; confidence information represents the credibility of the first information; real-time video information reflects the actual scene where the aircraft 10 is located; detour information may include the aircraft 10's flight altitude, flight speed, and flight direction when bypassing the target object; user operation information may include user editing operations on the first identifier, or user control operations on the flight status of the aircraft 10.

[0088] By saving the adjusted navigation map, it can be easily reused later. Specifically, the adjusted navigation map can be saved locally for use in subsequent flight missions of aircraft 10; sent to a cloud server for storage; and / or sent to the control terminals of other aircraft for storage for use in their flight missions.

[0089] In some embodiments, the heading of the same aircraft 10 may change when performing different flight missions or different flight phases. By saving the adjusted navigation map, the first information of the detected target can be reused, reducing the missed or false detection of the target due to the unfavorable detection angle caused by the different heading of the aircraft 10.

[0090] In some embodiments, the initial information of a target detected by the first aircraft is shared with a second aircraft. This reduces the second aircraft's heavy reliance on sensor hardware. Even if the second aircraft is not equipped with high-precision sensors, it can still perform targeted countermeasures based on the initial information of the target detected and stored by the first aircraft. During flight, the second aircraft can directly access the updated navigation map or load the initial identifier of the required target from the initial map by calling a control.

[0091] In examples where the navigation map also displays a third identifier, the adjusted navigation map can be saved based on the relative positional relationship between the third identifier and the first identifier, so that the saved navigation map has a specific identifier. The aforementioned specific identifier satisfies any of the following conditions:

[0092] When the outlines of the first and third identifiers intersect, the identifier range of a specific identifier includes the union of the outlines of the first and third identifiers; for example... Figure 6A As shown, assuming the outline of the first identifier is a rectangle with a width greater than its height (as shown by the red rectangle in the figure), and the outline of the third identifier is a rectangle with a height greater than its width (as shown by the blue rectangle in the figure), and the two rectangles intersect, then the specific identifier includes the outline formed by the entire area within these two rectangular outlines, resembling a cross shape. By determining the identifier range of the specific identifier as the union of the outline ranges of the first and third identifiers, the target objects represented by the first and third identifiers can be identified as a whole. This allows the aircraft 10 to perform response operations based on the whole composed of the two target objects during flight, such as obstacle avoidance, thereby reducing the possibility of the aircraft 10 colliding with another target object while avoiding one, and reducing the complexity of the control process of the aircraft 10.

[0093] When there is an inclusion relationship between the outline range of the first identifier and the outline range of the third identifier, the identifier range of a specific identifier includes the relatively larger outline range of the first identifier and the third identifier; such as Figure 6B As shown, assuming that the outline range of the first marker (as shown by the red rectangle in the figure) and the outline range of the third marker (as shown by the blue rectangle in the figure) are both rectangles, and the outline range of the third marker is contained within the outline range of the first marker, then the marking range of the specific marker is the outline range of the first marker. By determining the relatively larger outline range of the first and third markers as the marking range of the specific marker, the aircraft 10 can perform obstacle avoidance and detour operations based on the area where the larger target is located, reducing the possibility of the aircraft 10 colliding with a larger target while avoiding a smaller target, and reducing the complexity of the control process of the aircraft 10.

[0094] When the distance between the outlines of the first and third identifiers is less than a preset distance, the identifier range of a specific identifier includes the first identifier, the third identifier, and at least a portion of the area between them; such as Figure 6C As shown, assuming that the outline range of the first identifier (as shown by the red rectangle in the figure) and the outline range of the third identifier (as shown by the blue rectangle in the figure) are both rectangles, and the distance between the outline ranges of the first identifier and the third identifier is less than a preset distance, then the identifier range of a specific identifier includes the first identifier, the third identifier, and at least a portion of the area between them (as shown by the green rectangle in the figure). In this way, two identifiers that are close to each other can be treated as a whole, which makes it easier to control the aircraft 10 to perform obstacle avoidance and detour operations by treating the target objects represented by these two identifiers as a whole, thus reducing the complexity of the control process of the aircraft 10.

[0095] As time changes, the shape, size, or location of targets within the work area may change. In some embodiments, when the marking time of the third marker precedes that of the first marker, the marking range of the specific marker includes the outline range of the first marker, which is beneficial for obtaining the latest first information about the target. For example... Figure 6D As shown, assuming a third marker is marked on the navigation map at time T1 (as shown by the blue rectangle in the figure), and a first marker is marked on the navigation map at time T2 (as shown by the red rectangle in the figure), and time T2 is later than time T1, then the outline of the first marker can be determined as the marking range of a specific marker. During the flight path 10, the state of the target object may change. Therefore, determining the marking range of a specific marker based on the marker with the later marking time allows the marking range of the specific marker to better match the real-time state of the target object, improving marking accuracy.

[0096] See Figure 7 This disclosure also provides an auxiliary navigation method for an aircraft 10, the method comprising:

[0097] Step S21: During the flight of the aircraft 10, a real-time screen and a navigation map are displayed in the user interface, wherein the real-time screen displays images acquired by the image sensor 40 on the aircraft 10.

[0098] Step S22: During the flight of the aircraft 10, determine the first information of the target objects in the flight environment;

[0099] Step S23: In response to determining the first information, adjust the color of multiple pixels corresponding to the first information in the navigation map to display a first identifier for representing the target object in the navigation map;

[0100] Step S24: During the flight of aircraft 10, determine the second information of the target; and

[0101] Step S25: In response to the determined second information, adjust the color of multiple pixels corresponding to the second information in the real-time frame to display a second identifier for representing the target object in the real-time frame;

[0102] The first information includes the location information of the target object, and the second information is of a different type than the location information.

[0103] In this embodiment, identifiers representing target objects are displayed on both the real-time screen and the navigation map. This serves two purposes: firstly, it alerts and warns users of the target object, providing a better interactive experience; secondly, it allows users to verify and confirm the target object in two different displays. Furthermore, since the first identifier displayed on the navigation map and the second identifier displayed on the real-time screen are generated based on different information, it facilitates verification of the target object's authenticity from different perspectives. If both the first identifier representing a target object are displayed on the navigation map and the second identifier representing the same target object are displayed on the real-time screen, the target object's authenticity is high. If only the first identifier representing a target object is displayed on the navigation map, or only the second identifier representing the same target object is displayed on the real-time screen, the target object's authenticity is low. Users can analyze potential errors in the navigation map and / or real-time screen by comparing the display of the first and second identifiers, thereby controlling the aircraft 10 more accurately. Specific details of this embodiment are detailed in the foregoing embodiments and will not be repeated here.

[0104] See Figure 8A and Figure 8B This disclosure also provides an auxiliary navigation method for an aircraft 10, the method comprising:

[0105] Step S31: During the flight of the aircraft 10, a real-time screen and a navigation map are displayed in the user interface. The navigation map displays the flight path of the aircraft 10, and the real-time screen displays the images acquired by the image sensor 40 on the aircraft 10.

[0106] Step S32: During the flight of the aircraft 10, determine the first information of the target object in the flight environment;

[0107] Step S33: In response to determining the first information, adjust the colors of multiple pixels corresponding to the first information in the navigation map to display a first identifier for representing the target object in the navigation map; and

[0108] Step S34: Adjust the color of multiple pixels corresponding to the target object in the real-time image to display a second identifier for representing the target object in the real-time image;

[0109] The first piece of information includes the location information of the target object.

[0110] like Figure 8B As shown, this embodiment displays a first identifier representing the target object on the navigation map (as shown by the red, yellow, and blue broken lines in the figure), and a second identifier representing the target object on the real-time screen (as shown by the purple and green broken lines in the figure). This serves two purposes: firstly, it alerts and warns the user of the target object, providing a better interactive experience; secondly, it allows the user to view and confirm the target object in two different display screens through cross-verification. Furthermore, the navigation map displays the flight path of the aircraft 10 (as shown by the dotted line in the figure), facilitating the user's observation of the relative positional relationship between the target object and the flight path of the aircraft 10, thus enabling the user to adjust the flight path of the aircraft 10 in a timely manner.

[0111] In some embodiments, a directional marker (as shown by the arrow in the figure) representing the direction of movement of the aircraft 10 can also be displayed on the navigation map, and the location of the target object can be marked (as shown by the gray fan-shaped area in the figure). This makes it easier for users to determine how to adjust the heading of the aircraft 10 to avoid the target object.

[0112] Other specific details of this embodiment are detailed in the foregoing embodiments and will not be repeated here.

[0113] See Figure 9 This disclosure also provides an auxiliary navigation method for an aircraft, the method comprising:

[0114] Step S41: During the flight of the aircraft 10, a navigation map is displayed in the user interface 201;

[0115] Step S42: Real-time detection of the target object's location information; and

[0116] Step S43: In response to the real-time detected location information of the target object, adjust the color of the set of pixels corresponding to the location information of the target object in the navigation map. The set of pixels includes multiple pixels, and as more and more location information of the target object is detected, the number of pixels with changing color accumulates from few to many.

[0117] This embodiment adjusts the color of the set of pixels corresponding to the location information of a target on the navigation map, which can mark the target and thus remind and warn the user of the target. The viewing experience is excellent. Moreover, as more and more location information of the target is detected, the number of pixels with color changes accumulates from few to many. That is, the number of pixels with color changes that the user can observe on the navigation map increases continuously with the increase of the information of the target detected in real time, making it convenient for the user to be aware of the detection and marking process of the target.

[0118] The location information in this embodiment corresponds to the first information in the aforementioned embodiment. The location information of the target object can be detected by sensors on the aircraft 10 (such as image sensor 40, distance sensor, and / or other sensors). Multiple pixels in the pixel set corresponding to the location information of the target object correspond to multiple pixels corresponding to the first information in the aforementioned embodiment. As more and more location information of the target object is detected, the number of pixels with color changes accumulates from few to many. For example, at time T1, the number of pixels with color changes is N1; at time T2, the number of pixels with color changes accumulates from N1 to N1+N2; at time T3, the number of pixels with color changes accumulates from N1+N2 to N1+N2+N3; and so on. The pixel accumulation process in some embodiments is as follows... Figure 5 As shown.

[0119] This embodiment adjusts the color of the set of pixels corresponding to the location information of a target on the navigation map, thereby marking the target and serving as a reminder and warning to the user. The viewing experience is excellent. Furthermore, as more and more location information of the target is detected, the number of pixels with changing color accumulates continuously. That is, the number of pixels with changing color that the user can observe on the navigation map increases with the real-time detection of target information, making the target detection and marking process readily apparent to the user. Specific details of this embodiment are detailed in the foregoing embodiments and will not be repeated here.

[0120] See Figure 10 This disclosure also provides an auxiliary navigation method for an aircraft, the method comprising:

[0121] Step S51: During the flight of the aircraft 10, a real-time screen is displayed in the user interface 201, wherein the real-time screen displays images acquired by the image sensor 40 on the aircraft 10.

[0122] Step S52: In response to the image sensor 40 detecting information about the target object, a second identifier for characterizing the target object is displayed in a first style in the real-time image;

[0123] Step S53: In response to the information that the distance sensor on the aircraft 10 detects the target object, a second identifier for representing the target object is displayed in a second style in the real-time image; wherein the first style is different from the second style.

[0124] For example, in one scenario, if the image sensor 40 detects information about a target object, it can use visual semantics to display the target object's second identifier in a first style on the real-time screen. In another scenario, if the image sensor 40 does not identify the target object, but the distance sensor detects information about the target object, such as point cloud data including the target object, the point cloud data can be used to determine the target object's location information, facilitating its marking on the corresponding location on the navigation map. Furthermore, based on the target object's location information and the positional mapping relationship between the point cloud image and the real-time screen, the target object's second identifier can be displayed in a second style at the corresponding location on the real-time screen. The first and second styles are different to help users understand that the information sources for the marking are different. Further, the point cloud data can be used to determine point cloud semantic information, facilitating the selection of target objects with preset semantic information for marking. If both the image sensor 40 and the distance sensor detect the target object, the image sensor can also be selected as the information source for the second identifier, displaying the target object's second identifier in the first style on the real-time screen.

[0125] In this embodiment, depending on the type of sensor detecting the target object, different identifiers representing the target object are displayed in the real-time screen in different styles. This makes it easier for users to distinguish the type of sensor that detected the target object. Since different sensors have different detection accuracy and reliability, users can make more accurate decisions regarding the flight control process of the aircraft 10 based on the type of sensor that detected the target object, thereby improving the flight safety of the aircraft 10.

[0126] See Figure 11This disclosure also provides an interactive system 100, which includes a user interface device 11, a processor 12, a communication device 13, and an airborne sensor 14. The user interface device 11, the communication device 13, and the airborne sensor 14 are all communicatively connected to the processor 12. The airborne sensor 14 includes an image sensor 141 and a distance sensor 142.

[0127] User interface device 11 is used to display a real-time screen and a navigation map, wherein the real-time screen displays images acquired by image sensor 141;

[0128] The processor 12, in response to information about a target object detected by the distance sensor 142 in the flight environment, adjusts the color of multiple pixels in the navigation map to display a first identifier for representing the target object in the navigation map;

[0129] Processor 12 is used to adjust the color of multiple pixels in a real-time frame to display a second identifier for representing a target object in a real-time frame;

[0130] The processor 12 is used to save the adjusted navigation map so that the saved navigation map contains the first information;

[0131] The processor 12 sends the saved navigation map to the aircraft 10 via the communication device 13 for the aircraft 10 to use for navigation.

[0132] The user interface device 11 in this embodiment can be the user terminal 20 in the aforementioned embodiments, or it can be a device on the aircraft 10, which includes a user interface for displaying real-time images and navigation maps. The processor 12 can be a component with data processing capabilities on the aircraft 10 or the user terminal 20. The communication device 13 is used to realize communication between the processor 12 and the aircraft 10, and the communication device 13 can include, but is not limited to, a WIFI module, a satellite communication module, a Bluetooth module, a 4G communication module, and / or a 5G communication module. The image sensor 141 can include the image sensor 40 in the aforementioned embodiments, or it can include other image sensors. The distance sensor 142 includes, but is not limited to, an ultrasonic sensor, a radar sensor, and / or an infrared sensor. For specific embodiments of the method executed by the processor 12, please refer to the aforementioned method embodiments, which will not be repeated here.

[0133] This disclosure also provides a computer device, which includes at least a memory and a processor. The memory is used to store a computer program, and the processor is used to execute the computer program and, when executing the computer program, implement the method described in any of the foregoing embodiments.

[0134] Figure 12This illustration shows a more specific hardware structure diagram of a computing device provided in an embodiment of the present disclosure. The device may include: a processor 1001, a memory 1002, an input / output interface 1003, a communication interface 1004, and a bus 1005. The processor 1001, memory 1002, input / output interface 1003, and communication interface 1004 are interconnected internally via the bus 1005.

[0135] The processor 1001 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The processor 1001 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

[0136] The memory 1002 can be implemented in the form of read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this disclosure are implemented by software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001.

[0137] The input / output interface 1003 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0138] The communication interface 1004 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0139] Bus 1005 includes a pathway for transmitting information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004).

[0140] It should be noted that although the above-described device only shows the processor 1001, memory 1002, input / output interface 1003, communication interface 1004, and bus 1005, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this disclosure, and not necessarily all the components shown in the figures.

[0141] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0142] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0144] The systems, devices, modules, or units described in the above embodiments can be implemented by computer devices or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0145] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0146] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A method for updating a navigation map for an aircraft, characterized in that, include: During the flight of the aircraft, a real-time screen and a navigation map are displayed in the user interface, wherein the real-time screen displays images acquired by the onboard image sensor of the aircraft; During the flight of the aircraft, the first information about the target objects in the flight environment is determined; In response to the determined first information, the colors of a plurality of pixels corresponding to the first information are adjusted in the navigation map to display a first identifier for representing the target object in the navigation map; Adjust the colors of multiple pixels corresponding to the target object in the real-time image to display a second identifier representing the target object in the real-time image; and Save the adjusted navigation map so that the saved navigation map contains the first information of the target object; The first information includes the location information of the target object.

2. An auxiliary navigation method for an aircraft, characterized in that, include: During the flight of the aircraft, a real-time screen and a navigation map are displayed in the user interface, wherein the real-time screen displays images acquired by the onboard image sensor of the aircraft; During the flight of the aircraft, the first information about the target objects in the flight environment is determined; In response to the determined first information, the colors of a plurality of pixels corresponding to the first information are adjusted in the navigation map to display a first identifier for representing the target object in the navigation map; During the flight of the aircraft, second information about the target is determined; and In response to determining the second information, the colors of a plurality of pixels corresponding to the second information are adjusted in the real-time frame to display a second identifier for representing the target object in the real-time frame; The first information includes the location information of the target object, and the second information is of a different type than the location information.

3. An auxiliary navigation method for an aircraft, characterized in that, include: During the flight of the aircraft, a real-time video and a navigation map are displayed in the user interface. The navigation map displays the flight path of the aircraft and / or the terrain information of the flight area of ​​the aircraft. The real-time video displays images acquired by the onboard image sensors of the aircraft. During the flight of the aircraft, the first information about the target objects in the flight environment is determined; In response to the determined first information, the colors of a plurality of pixels corresponding to the first information are adjusted in the navigation map to display a first identifier for representing the target object in the navigation map; and In the real-time image, the colors of multiple pixels corresponding to the target object are adjusted to display a second identifier for representing the target object in the real-time image; The first information includes the location information of the target object.

4. The method according to claim 2 or 3, characterized in that, After adjusting the colors of multiple pixels corresponding to the first information in the navigation map, the method further includes: Save the adjusted navigation map so that the saved navigation map contains the first information of the target object.

5. The method according to claim 4, characterized in that, Saving the adjusted navigation map includes: Based on preset user actions and / or preset rules, the adjusted navigation map is saved.

6. The method according to claim 5, characterized in that, When there are multiple targets, the user operation includes: the user's selection operation of the targets to be saved in the real-time screen and / or the navigation map.

7. The method according to claim 5, characterized in that, The user operations include: the user's editing operations on the first identifier that needs to be saved.

8. The method according to claim 7, characterized in that, The editing operation includes any of the following: The operation of expanding the identification range of the first identifier. The operation of deleting part of the first identifier. The operation of selectively saving a portion of the first identifier. Confirm the confirmation operation to save the first identifier.

9. The method according to claim 5, characterized in that, The preset rules include at least one of the following: Selectively save the first information within a preset range; The first information of the target object with preset semantic information is selectively stored; The adjusted navigation map synchronously stores other information used to characterize the target object, including: timeliness information, semantic information, confidence information, real-time image information, detour information and / or user operation information.

10. The method according to claim 1 or 4, characterized in that, The process of saving the adjusted navigation map includes at least one of the following: The adjusted navigation map is saved locally for use in subsequent flight missions of the aircraft; The adjusted navigation map is sent to the cloud server for storage; The adjusted navigation map is sent to the control terminals of other aircraft and / or other aircraft for storage, so that it can be used by other aircraft for their flight missions.

11. The method according to claim 2, characterized in that, The first identifier used to characterize the target object in the navigation map includes: The navigation map displays the first identifier used to characterize a specific target, wherein the specific target refers to an object whose second information satisfies preset condition two and / or an object whose first information satisfies preset condition one.

12. The method according to claim 11, characterized in that, The first identifier used to characterize the specific target object in the navigation map includes: In response to a user's selection of a specific target object included in the target object in the real-time view, the first identifier representing the specific target object is displayed in the navigation map.

13. The method according to claim 11, characterized in that, The second information satisfies preset condition two, which includes preset semantic information; or, the first information satisfies preset condition one, which includes preset semantic information.

14. The method according to any one of claims 1 to 3, characterized in that, As the amount of the first information detected increases, the number of pixels with color adjustment in the navigation map accumulates from a small number to a large number.

15. The method according to any one of claims 1 to 3, characterized in that, The navigation map also displays a third identifier, which differs in display style from the first identifier.

16. The method according to claim 15, characterized in that, The third identifier is used to represent any of the following information: The first piece of information detected at a historical moment; The first information detected by the aircraft when it is performing a segment different from the current segment; The first information detected by other devices besides the aircraft; The location information of other objects in the flight environment is determined in real time, wherein the semantic information of the other objects is different from that of the target object.

17. The method according to claim 15, characterized in that, Also includes: Based on the relative positional relationship between the third identifier and the first identifier, the adjusted navigation map is saved so that the saved navigation map has a specific identifier.

18. The method according to claim 17, characterized in that, The specific identifier satisfies any of the following conditions: When the outline range of the first identifier and the outline range of the third identifier intersect, the identifier range of the specific identifier includes the union of the outline range of the first identifier and the outline range of the third identifier; In the case where there is an inclusion relationship between the outline range of the first identifier and the outline range of the third identifier, the identifier range of the specific identifier includes the relatively larger outline range of the first identifier and the third identifier; When the distance between the outline range of the first identifier and the outline range of the third identifier is less than a preset distance, the identifier range of the specific identifier includes the first identifier, the third identifier, and at least a portion of the area between them; In the case where the identification time of the third identifier precedes the identification time of the first identifier, the identification range of the specific identifier includes the outline range of the first identifier.

19. The method according to claim 2, characterized in that, The second information includes semantic information.

20. The method according to claim 19, characterized in that, The second information is obtained from the data detected by the image sensor, or the second information is obtained from the data detected by the distance sensor of the aircraft.

21. The method according to claim 19, characterized in that, The second identifier generated based on the image sensor detection data and the second identifier generated based on the distance sensor detection data of the aircraft are displayed differently in the real-time image.

22. The method according to any one of claims 1 to 3, characterized in that, The target object includes obstacles, work objects, or objects with preset semantic information.

23. The method according to any one of claims 1 to 3, characterized in that, The first information also includes semantic information, and / or the first information is obtained from data detected by the distance sensor of the aircraft.

24. An auxiliary navigation method for an aircraft, characterized in that, include: During the flight of the aircraft, a navigation map is displayed in the user interface; Real-time detection of the target object's location information; as well as In response to the real-time detected location information of the target object, the color of the set of pixels corresponding to the location information of the target object is adjusted in the navigation map. The set of pixels includes multiple pixels, and as more and more location information of the target object is detected, the number of pixels with changing color accumulates from few to many.

25. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the method of any one of claims 1 to 24.

26. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the steps of the method as described in any one of claims 1 to 24.

27. An interactive system, characterized in that, include: The system includes a user interface device, a processor, a communication device, and airborne sensors, all of which are communicatively connected to the processor. The airborne sensors include an image sensor and a distance sensor. The user interface device is used to display a real-time screen and a navigation map, wherein the real-time screen displays images acquired by the image sensor; The processor, in response to information about a target object detected by the distance sensor in the flight environment, adjusts the color of multiple pixels in the navigation map to display a first identifier for representing the target object in the navigation map; The processor is configured to adjust the color of multiple pixels in the real-time image to display a second identifier for representing the target object in the real-time image; The processor is used to save the adjusted navigation map so that the saved navigation map contains the first information; The processor will send the saved navigation map to the aircraft via the communication device for the aircraft's navigation.

Images