Navigation processing method and control device

Abstract

A navigation processing method, device, control equipment (102), aircraft, and system. The navigation processing method includes: displaying a received captured image (201) on a preset user interface (200) (step S301), wherein the captured image (201) is captured by a camera device configured on a mobile object (101); if a position selection operation is received on the user interface (200), determining position information of the position point selected by the position selection operation in the image (201) (step S302); and controlling the mobile object (101) to move toward a target navigation point (step S303), wherein the target navigation point is obtained based on the position information. The method can facilitate a user to intuitively select a target navigation point, and cause a mobile object (101) such as an aircraft to move toward the target navigation point. The operation is intuitive and quick, thereby improving the efficiency of navigation and the execution efficiency of tasks such as aerial photography.

Description

Technical Field

[0001] The present invention relates to the field of navigation application technology, and in particular to a navigation processing method and control device. Background Art

[0002] Aircraft, especially a remote-controlled drone, can effectively assist people in their work. Drones carrying cameras, agricultural sprayers and other equipment can excellently complete tasks such as aerial photography, disaster relief, surveying and mapping, power inspections, agricultural spraying, and patrol reconnaissance.

[0003] Generally speaking, drones can automatically plan and navigate routes. Traditional flight navigation requires users to mark waypoints on a map, and the drone then navigates to each waypoint, automatically flying and executing the corresponding mission.

[0004] In the existing technology, users can only determine the location of waypoints on the map, but map data generally has errors. The waypoint location determined by the user on the map may be a long distance away from the location of the object the user actually wants to observe, seriously affecting the accuracy of the aircraft in performing the corresponding flight mission. Summary of the Invention

[0005] The embodiments of the present invention provide a navigation processing method, apparatus, and control device, which allow a user to intuitively determine the location of an object to be observed from an image and control the movement of a mobile object such as an aircraft.

[0006] In a first aspect, an embodiment of the present invention provides a navigation processing method, including:

[0007] Displaying a received captured image on a preset user interface, wherein the captured image is captured by a camera device disposed on the moving object;

[0008] If a position selection operation is received on the user interface, determining position information of the position point selected by the position selection operation in the image;

[0009] The mobile object is controlled to move toward a target navigation point, where the target navigation point is obtained based on the position information.

[0010] In a second aspect, an embodiment of the present invention further provides a navigation processing device, including:

[0011] a display unit, configured to display a received captured image on a preset user interface, wherein the captured image is captured by a camera device disposed on the moving object;

[0012] a processing unit configured to, upon receiving a position selection operation on the user interface, determine position information of a position point selected by the position selection operation in the image;

[0013] The control unit is used to control the mobile object to move toward a target navigation point, where the target navigation point is obtained based on the position information.

[0014] In a third aspect, an embodiment of the present invention further provides a control device, the control device comprising: a memory and a processor;

[0015] The memory is used to store program instructions;

[0016] The processor calls the program instructions stored in the memory to perform the following steps:

[0017] Displaying a received captured image on a preset user interface, wherein the captured image is captured by a camera device disposed on the moving object;

[0018] If a position selection operation is received on the user interface, determining position information of the position point selected by the position selection operation in the image;

[0019] The mobile object is controlled to move toward a target navigation point, where the target navigation point is obtained based on the position information.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the navigation processing method as described in the first aspect above is implemented.

[0021] The embodiment of the present invention facilitates users to determine a location point based on the captured image to realize navigation of the moving object. The user can intuitively perform pointing navigation operations on the user interface to allow the moving object to move directly to a position where the target object can be effectively observed, thereby improving the accuracy of the mobile object in performing related observation tasks and improving the efficiency of task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of a navigation system according to an embodiment of the present invention;

[0023] Figure 2a is a schematic diagram of a user interface according to an embodiment of the present invention;

[0024] Figure 2b is a schematic diagram of another user interface according to an embodiment of the present invention;

[0025] Figure 2c is a schematic diagram of another user interface according to an embodiment of the present invention;

[0026] Figure 3 is a flowchart of a navigation processing method according to an embodiment of the present invention;

[0027] Figure 4 is a flowchart of another navigation processing method according to an embodiment of the present invention;

[0028] Figure 5 is a structural diagram of a navigation processing device according to an embodiment of the present invention;

[0029] Figure 6 It is a structural diagram of a control device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In an embodiment of the present invention, a user operation such as clicking can be used to select and specify a location point in a first-person view (FPV) image transmission screen, and the location information of the location point in the image can be calculated. By converting and calculating the location information in the image, a target navigation point is obtained, and then a moving object such as an aircraft or an unmanned vehicle is controlled to move toward the target navigation point corresponding to the location information, wherein the position of the target navigation point is determined based on the location information of the location point in the image.

[0031] The control device can be configured to control the mobile object in either a position-pointing navigation mode or a direction-pointing navigation mode, depending on the user's needs. In position-pointing navigation mode, after a user clicks a location point on the user interface of the control device, the control device determines the location information of the location point in the image on the user interface and sends the location information to the mobile object to control the mobile object to move toward the target navigation point indicated by the location information. The target navigation point is determined based on the location information, and the location of the target navigation point is the final destination of the movement.

[0032] In the direction pointing navigation mode, after the user clicks a certain location point on the user interface of the control device, the control device determines the location information of the location point in the image of the user interface, and the control device sends the location information to the mobile object to control the mobile object to move in the target motion direction indicated by the location information, wherein the target motion direction is determined based on the location information. For example, if the location point selected by the user is in the upper right relative to the center point of the image, then the mobile object such as an aircraft can be controlled to fly in the upper right direction. There is no target navigation point as the final destination of the mobile object. If the user does not interrupt the movement of the mobile object in the target motion direction, the mobile object will continue to move in the target motion direction.

[0033] Mobile objects such as aircraft and driverless cars are equipped with a camera device that captures images in real time. The mobile object transmits part or all of the captured images back to the control device. The image can be considered as a first-person perspective image of the mobile object. The control device can be equipped with a touch screen to display the image captured by the camera device. A communication connection can be established between the mobile object and the control device, and point-to-point communication can be achieved based on the communication connection. The camera device sends the captured image to the mobile object via a wired or wireless method. For example, the camera device sends the image to the mobile object via a short-range wireless transmission method such as Bluetooth or NFC, and the mobile object then forwards the image to the control device via a WiFi protocol, an SDR (software defined radio) protocol, or other custom protocols.

[0034] The control device is configured with a touch screen that displays the received image in real time. In one embodiment, the received image is displayed in a user interface. A grid icon is displayed in a portion of the image display area on the user interface. When a user clicks to select a location within the area covered by the grid icon, an augmented reality disk is generated that closely matches the selected location. The augmented reality disk serves as the location icon for the location and is displayed on the user interface. The grid icon can be used to represent the ground.

[0035] Based on the position information of the selected location point in the image, the coordinate position of the location point in the world coordinate system can be determined. The coordinate position in the world coordinate system is the specific location of the target navigation point. When calculating the target navigation point, the altitude information of the mobile object such as the aircraft, the attitude information of the gimbal mounted on the mobile object, the field of view (FOV) angle of the camera device mounted on the gimbal of the mobile object, and the position information of the mobile object can be comprehensively considered.

[0036] The control device can send the position information of the location point selected by the user in the image to the mobile object, and the mobile object can calculate the target navigation point of the location point in the world coordinate system. The mobile object can send the coordinate position corresponding to the target navigation point to the control device. After receiving the relevant information of the target navigation point, the control device issues a prompt whether to fly to the target navigation point, such as displaying a "start" icon on the user interface. If a user response operation to the prompt is detected, such as clicking the "start" icon, the mobile object is controlled to move toward the target navigation point.

[0037] In another embodiment, the mobile object does not need to send any information about the target navigation point to the control device. After sending the position information of the location point selected by the user in the image, the control device directly issues a prompt whether to fly to the target navigation point within a preset time period. If a confirmation response from the user is received, a control instruction is sent to the mobile object, and the mobile object moves to the target navigation point calculated by it according to the control instruction.

[0038] In another embodiment, after calculating the target navigation point, the mobile object may also send only a notification message to the control device to indicate whether to start moving. After receiving the notification message, the control device issues a prompt whether to fly to the target navigation point. If a confirmation response from the user is received, a control instruction is sent to the mobile object, and the mobile object moves to the calculated target navigation point according to the control instruction.

[0039] In one embodiment, after obtaining the position information of the location point selected by the user in the image, the control device can calculate the relevant position information of the target navigation point and issue a prompt whether to fly to the target navigation point. If a confirmation response from the user is received, a control instruction carrying the relevant position information of the target navigation point is sent to the mobile object to control the mobile object to move toward the target navigation point.

[0040] In one embodiment, based on the needs of a task such as observation, and based on a new image captured during the movement of a moving object, a user can again click and select a new location in the user interface displaying the new image. Based on the location information of the new location in the image, a new target navigation point can be determined, and the moving object can then be controlled to move toward the new target navigation point. In this embodiment of the present invention, the user can completely bypass joystick control to control the moving object, eliminating the need for navigation operations such as dotting on a map. Navigation can be achieved by simply pointing to a location on the image. Because the image object captured by the camera in front of the aircraft can be identified in the image, the user can determine the target navigation point based entirely on the image object, enabling more accurate monitoring of the object to be observed. For example, if an image already includes a power tower to be observed, the user can intuitively click on the tower's location within the area covered by the grid icon. After a series of calculations, the target navigation point corresponding to the location is determined, and the aircraft is automatically controlled to move toward the target navigation point, completing the observation task of the tower.

[0041] In one embodiment, taking into account the shooting performance of the camera device, such as the shooting distance and pixel size, it is possible to consider combining navigation by marking dots on the map with navigation based on the position pointing navigation mode in the image displayed on the user interface of an embodiment of the present invention, and determine the approximate location point of the object to be observed on the map. When flying within a preset distance range of the approximate location point, switch to navigation based on the position pointing navigation mode, and then more accurately determine the target navigation point for navigation of the moving object.

[0042] Figure 1 FIG. 1 shows a schematic structural diagram of a navigation system according to an embodiment of the present invention. The system includes a control device 102 and a mobile object 101. Figure 1 In the figure, the moving object 101 is represented by an aircraft. In other schematic diagrams, a movable robot, an unmanned car, or the like that can be equipped with a camera and can move based on a control device 102 such as a remote controller can also be used as the moving object 101.

[0043] The control device 102 can be a dedicated remote control with a touch screen and configured with corresponding program instructions, or it can be a smart terminal such as a smartphone, tablet computer, smart wearable device, etc. with a corresponding application app installed. The control device can also be a combination of two or more of the remote control, smartphone, tablet computer, and smart wearable device. The aircraft can be a quad-rotor, hexacopter, or other unmanned aerial vehicle (UAV), or a fixed-wing UAV. The aircraft can be equipped with a camera via a gimbal, allowing for flexible capture of images in multiple directions. A communication connection can be established between the control device 102 and the aircraft based on a WiFi protocol, an SDR protocol, or other custom protocols to exchange navigation data, image data, and other data required for the embodiments of the present invention.

[0044] The user enters the position-pointing navigation mode of this embodiment of the present invention through the connected aircraft application app on control device 102. After takeoff, the aircraft is controlled in position-pointing navigation mode within a safe altitude range, such as a safe altitude range of 0.3 meters to 6 meters, or other safe altitude ranges set based on the aircraft's mission and / or flight environment. After entering position-pointing navigation mode, the screen of control device 102 displays images returned by the aircraft, captured by the aircraft's camera.

[0045] In the embodiment of the present invention, Figure 2a 、 2b2c, the user interface 200 is described accordingly. The control device 102 displays the user interface 200. The user interface 200 at least displays the image 201 captured by the camera device and displays a grid icon 204. In this user interface, if the position pointing navigation mode of the embodiment of the present invention is not entered, the user interface 200 only displays the image captured by the camera device. Once the position pointing navigation mode is entered, the user interface 200 displays the image 201 captured by the camera device. Figure 2a The interface shown. The user can click on the grid icon 204 on the screen of the control device 102, that is, click on the area covered by the grid icon 204. The screen of the control device 102 can be a touch screen, and the user can directly click on the corresponding position in the area covered by the grid icon 204 with an object such as a finger. After the user clicks, a virtual reality disc 202 will be displayed on the user interface of the control device 102. The virtual reality disc 202 serves as a location icon to indicate the location point clicked by the user. After the location point is confirmed by clicking, a Go button 203 pops up in the control device 102. The button 203 is a trigger icon, which is used to control the aircraft to start moving towards the target navigation point corresponding to the location point after receiving the user's click operation.

[0046] When the user clicks the Go button 203, the control device 102 sends control commands to the aircraft. The aircraft executes flight control based on its own flight dynamics and arrives above the corresponding target navigation point. During the flight, the aircraft's horizontal altitude can remain constant. As the aircraft flies toward the target navigation point, it gradually approaches the virtual reality disk 202. The graphic of the virtual reality disk 202 gradually enlarges in the user interface to indicate that the distance between the aircraft and the target navigation point is getting closer.

[0047] As the aircraft heads toward the target navigation point, the user interface 200 displays a new image captured by the camera in real time. On the user interface 200, the user can continue to click other locations on the image 201 on the screen to control and change the aircraft's flight direction. When the user clicks other locations to change the flight direction, the aircraft performs a coordinated turning maneuver based on its own flight power, resulting in a smooth flight trajectory. In one embodiment, different control processes can be performed on the aircraft based on different click operations on the user interface 200. For example, if it is a short click operation, the aircraft's flight direction can be controlled so that the aircraft first flies to the intermediate position selected by the click operation and then continues to fly to the target navigation point. If it is a long press operation, the target navigation point is changed and a new target navigation point is calculated based on the position information of the position point corresponding to the long press operation in the image, and the aircraft no longer flies to the original target navigation point.

[0048] When the aircraft is flying towards the target navigation point, it can use the configured detection system to perform autonomous obstacle avoidance. When a smaller first-category obstacle is detected in the flight direction, evasive flight can be performed to directly bypass the first-category obstacle. If a larger second-category obstacle is encountered, automatic braking and hovering can be performed. At this time, the user can click on the left and right sides of the screen to perform on-site rotation of the heading angle yaw until the image object corresponding to the clicked position point is located in the center position area (target area) of the captured image. After the aircraft yaws in place, the position selection operation can continue on the area covered by the grid icon 204.

[0049] In an embodiment of the present invention, the position pointing navigation mode and the direction pointing navigation mode can be switched, and the switching methods include multiple. In one embodiment, when the user directly clicks on the sky part of the image 201 displayed on the user interface 200 to determine the position point, only the flight direction of the aircraft can be changed according to the position information of the position point in the image. For example, in the direction pointing navigation mode, if the position point in the sky part of the image is clicked directly above the center point of the image, the aircraft will fly upward, and if the position point in the sky part of the image is clicked to the upper right of the center point of the image, the aircraft will fly to the upper right; and if the user clicks on the position point in the area covered by the grid icon 204 in the user interface 200 to determine the position point, the target navigation point corresponding to the position point will be calculated, and the aircraft will be controlled to fly to the position of the target navigation point. In another embodiment, a button for the user to click may be configured and displayed on the user interface 200. After the user clicks the button, the control mode of the aircraft may be set to the position pointing navigation mode, and the aircraft will navigate based on the aforementioned target navigation point. Alternatively, after the user clicks the button, the control mode of the aircraft may be set to the direction pointing navigation mode, and the aircraft may only navigate by determining the flight direction. In yet another embodiment, if the user clicks to determine a position point on the user interface 200 and a corresponding target navigation point is calculated based on the position point, the control mode of the aircraft may be set to the position pointing navigation mode. If the corresponding target navigation point cannot be calculated based on the position point, the control mode of the aircraft may be set to the direction pointing navigation mode.

[0050] The embodiment of the present invention facilitates users to determine a target navigation point based on the captured image to realize navigation of the moving object. The user can intuitively perform pointing navigation operations on the user interface to allow the moving object to move directly to a position where the target object can be effectively observed, thereby improving the accuracy of the mobile object in performing related observation tasks and improving the efficiency of task execution.

[0051] See also Figure 3, is a flowchart of a navigation processing method according to an embodiment of the present invention, which can be implemented by the control device mentioned above. The method according to the embodiment of the present invention includes the following steps.

[0052] S301: Display the received captured image on a preset user interface, where the captured image is captured by a camera device mounted on a moving object. The user interface is a preset interface capable of displaying the image captured by the camera device, and can also monitor user operations to perform corresponding processing. For a specific user interface diagram, please refer to Figure 2a 、 2b The camera device can be mounted on the mobile object by means of a gimbal or the like, and the camera device and the mobile controller of the mobile object (eg, the flight controller of an aircraft) can be connected to each other by wired or wireless signals.

[0053] S302: If a position selection operation is received on the user interface, the position information of the position point selected by the position selection operation in the image is determined. The position selection operation may be generated by a user clicking on the user interface. User operations on the user interface, such as single-click, double-click, and long-press, may be used as position selection operations as needed. After receiving the position selection operation, the pixel position of the selected position point in the image, i.e., the position information of the selected position point in the image, is determined based on the screen location clicked by the user.

[0054] S303: Control the moving object to move toward a target navigation point, where the target navigation point is obtained based on the position information.

[0055] The control device sends the position information to the mobile object so that the mobile object moves to the target navigation point indicated by the position information. The target navigation point can also be calculated by the mobile object based on the position information sent by the control device. After receiving the operation of triggering the movement of the mobile object issued by the user on the user interface, the control device can generate a control instruction to control the mobile object to move according to the target navigation point calculated by it. In some cases, after the mobile object determines the target navigation point based on the position information sent by the control device, it can also move directly to the target navigation point.

[0056] The embodiment of the present invention facilitates users to determine a location point based on the captured image to realize navigation of the moving object. The user can intuitively perform pointing navigation operations on the user interface to allow the moving object to move directly to a position where the target object can be effectively observed, thereby improving the accuracy of the mobile object in performing related observation tasks and improving the efficiency of task execution.

[0057] See also Figure 4, is a flowchart of another navigation processing method according to an embodiment of the present invention, which can be implemented by the control device mentioned above. The method according to the embodiment of the present invention includes the following steps.

[0058] S401: Displaying a received captured image on a preset user interface, where the captured image is captured by a camera device disposed on a moving object.

[0059] S402: If a position selection operation is received on the user interface, determine position information of the position point selected by the position selection operation in the image.

[0060] On the user interface, a grid icon can be generated, which can represent the ground. The grid icon can be generated specifically according to at least one of the shooting angle of the shooting device (the posture of the gimbal), the FOV angle of the shooting device, and the height of the moving object; the grid icon is displayed over a designated area of ​​the captured image; a position selection operation is detected on the designated area covered by the grid icon, and the designated area can be the area corresponding to the ground part in the image. For example, a click operation on the area where the grid icon is located can be considered a position selection operation. In other words, only operations such as user clicks on the grid icon are considered position selection operations, and the following steps are executed. Otherwise, the following steps S403 and others are not executed. In some cases, user operations outside the grid icon can be used for other controls, such as controlling the gimbal of a moving object to rotate on the pitch axis pitch, or only controlling the current moving direction of a moving object such as an aircraft.

[0061] In one embodiment, user operations received in an area outside the grid icons of the user interface can be considered as direction selection operations. When a direction selection operation is received in an area outside the grid icons of the user interface, the position information of the location point selected by the direction selection operation in the image is determined; and the moving object is controlled to move in a target direction of movement, where the target direction of movement is determined based on the position information of the location point selected by the direction selection operation in the image. In other words, operations in an area outside the grid icons, such as a user click operation, can be considered as the user's attempt to control the direction of movement of the moving object.

[0062] S403: Generate a location icon for the location point selected by the location selection operation and display the location icon on the user interface. The location icon may be the virtual reality disc mentioned above. The location icon is attached to the grid icon displayed on the user interface. Subsequently, during the movement of the moving object, the size of the location icon is adjusted according to the distance between the moving object and the target navigation point. The size of the location icon is used to represent the distance between the moving object and the target navigation point. In an optional embodiment, the closer the moving object is to the target navigation point, the larger the size of the location icon.

[0063] S404: Displaying a trigger icon on the user interface, wherein the trigger icon is used to indicate whether to control the moving object to move toward the target navigation point; upon receiving a selection operation on the trigger icon, triggering the execution of the following S405.

[0064] S405: Control the mobile object to move toward a target navigation point, where the target navigation point is obtained based on the position information. The target navigation point is a position point in a world coordinate system determined based on the position information.

[0065] In one embodiment, the moving object is controlled to move toward the target navigation point according to the preset operating altitude information; wherein the operating altitude information includes: the acquired current altitude information of the moving object, or the received configuration altitude information. The control device may send a control instruction to the aircraft after receiving a click operation on the trigger icon, and the control instruction carries information on controlling the aircraft to move according to the preset motion altitude information; or, when the control device does not carry any information indicating the altitude in the control instruction, it may be considered that the aircraft is controlled to move according to the preset motion altitude information by default, such as flying according to the current altitude of the aircraft. The configuration altitude information refers to a safety altitude set through the user interface, or a safety altitude pre-configured by the user on the moving object.

[0066] In one embodiment, executing the step of controlling the movement of the mobile object toward the target navigation point may specifically include: detecting a flight control instruction; if the flight control instruction is a first control instruction, triggering execution of step S405; if the flight control instruction is a second control instruction, controlling the mobile object to move in a target motion direction, the target motion direction being obtained based on the position information of the position point selected by the position selection operation in the image. In other words, step S405 is executed only when the first control instruction is detected to facilitate control of the mobile object based on the target navigation point. If the second control instruction is detected, only the current motion direction of the mobile object, such as an aircraft, may be controlled. The flight control instruction may be a switching instruction, specifically generated when a user clicks a switching button on a user interface, or a mode selection instruction, specifically generating a mode selection instruction (first control instruction) for a position-pointing navigation mode when a user clicks a first button on the user interface, and generating a mode selection instruction (second control instruction) for a direction-pointing navigation mode when a user clicks a second button on the user interface.

[0067] In one embodiment, after the mobile object moves into the predetermined area of ​​the target navigation point, it hovers in the predetermined area above the target navigation point according to the operating altitude information. When the mobile object determines that the current position coordinates of the mobile object in the world coordinate system are the same as the position coordinates of the target navigation point or are within a preset distance range based on the positioning module such as the GPS module carried by itself, it can be considered that the navigation to the target navigation point has ended, and the aircraft as the mobile object needs to hover in a predetermined area above the target navigation point. The distance between each position in the predetermined area and the coordinate position of the target navigation point (such as the GPS coordinate close to the ground) is less than a preset threshold.

[0068] In one embodiment, during the movement of the mobile object, if a position update operation is detected on the user interface, the updated position information of the position point selected by the position update operation in the image is determined, and the mobile object is controlled to move to the updated navigation point, which is obtained based on the updated position information. The position update operation can be specifically determined when a predefined user operation such as a click operation or a long press operation is monitored by the user in the area covered by the grid icon in the image displayed on the user interface. Upon detecting such an operation, the control device re-determines a new target navigation point based on the position point selected by the position update. The re-determined target navigation point is the updated navigation point. Similarly, the updated navigation point can be calculated by the control device. The updated navigation point can also be calculated by the mobile object by sending the updated position information to the mobile object by the control device. After determining the updated navigation point, the mobile object no longer moves to the original target navigation point determined before receiving the position update operation. The control device can directly delete the original target navigation point or only store the original target navigation point for subsequent analysis of the movement data of the mobile object. The process of determining the re-determined target navigation point can refer to the description of the relevant steps of the target navigation point in the above embodiment.

[0069] During the movement of the mobile object, the mobile object can automatically detect obstacles in the direction of flight and perform different obstacle avoidance operations according to different obstacles. In one embodiment, the mobile object is in a hovering state when it detects a first type of obstacle, and performs obstacle avoidance movement when it detects a second type of obstacle. The obstacle avoidance movement is used to bypass the second type of obstacle in the process of moving to the target navigation point. The first type of obstacle can be a building, a mountain, or other large-sized obstacle that a mobile object such as an aircraft cannot quickly bypass. At this time, the aircraft can perform a hovering process to notify the user to perform corresponding operational controls. Other mobile objects such as movable robots stop moving to facilitate the user to perform corresponding operational controls. The second type of obstacles are some smaller obstacles that can be bypassed by calculating an obstacle avoidance route, such as obstacles such as telephone poles and small trees. The second type of obstacles do not require user operation, and the obstacle avoidance route is calculated by the mobile object such as the aircraft to automatically bypass them.

[0070] In one embodiment, during the movement of the mobile object, a lateral movement control operation on the user interface is monitored; if a lateral movement control operation is received, the mobile object is controlled to move sideways according to the monitored lateral movement control operation. The lateral movement control operation may include: a sliding operation from left to right on the user interface, a sliding operation from right to left on the user interface, a sliding operation from top to bottom on the user interface, a sliding operation from bottom to top on the user interface, a click operation on the left half plane of the center point of the user interface, a click operation on the right half plane of the center point of the user interface, a click operation on the upper half plane of the center point of the user interface, or a click operation on the lower half plane of the center point of the user interface.

[0071] Among them, it may be that when it is detected that the mobile object is in a hovering state, the monitoring of the lateral shift control operation on the user interface is triggered. The controlling of the lateral movement of the mobile object according to the monitored lateral shift control operation may include: according to the monitored lateral shift control operation, controlling the mobile object to move on a vertical plane of the flight direction before the mobile object is in the hovering state. If a mobile object such as an aircraft detects the above-mentioned first type of obstacle, it will be in a hovering state. The aircraft or other mobile object can notify the control device by sending a hovering notification message. At this time, the screen of the control device will also display the image captured by the camera device on the mobile object. The mobile object can be moved laterally by naked eye observation or test flight, so as to manually control the aircraft or other mobile object to avoid obstacles. For mobile objects such as quadcopters, lateral movement can be achieved by flying in four directions: up, down, left and right.

[0072] The control device can avoid the first type of obstacles by controlling the heading angle of a mobile object such as an aircraft, adjusting the heading angle of the mobile object to a certain angle, and then flying forward at the adjusted heading angle. In one embodiment, the heading angle of the mobile object is controlled according to the heading control operation detected on the user interface so that the mobile object flies according to the new heading angle. Specifically, a rotation control instruction can be sent to the mobile object based on the object position point indicated by the heading control operation detected on the user interface; the rotation control instruction is used to control the mobile object to rotate to the new heading angle so that the image object of the object position point is in the target area of ​​the image captured by the camera device. The control device can continue to control the heading angle of the mobile object to rotate the mobile object until the image of the new image captured by the camera device is in the center area of ​​the new image of the object position point indicated by the user in the heading control operation. That is to say, during the movement of a mobile object, if it encounters an obstacle that cannot be circumvented and is in a hovering state, when the user initiates a heading control operation by clicking on the user interface, or when the user actively initiates a heading control operation by clicking on the user interface, the control device can control the mobile object to rotate to change the heading and continue moving.

[0073] In one embodiment, during the movement of the mobile object, if a movement direction adjustment operation is detected on the user interface, a control instruction is issued to the mobile object to control the current movement direction of the mobile object; the movement direction adjustment operation includes: a sliding operation or a long press operation received on the user interface, etc., which is used to control and adjust the current movement direction of the mobile object. In other words, the movement direction adjustment operation is different from the above-mentioned position update operation. During the movement of the mobile object, if the control device receives certain agreed special operations that only adjust the direction, it will control the mobile object to change its current movement direction. However, after a specified period of time after the direction adjustment, the aircraft can automatically adjust the flight direction to continue moving towards the target navigation point, and the subsequent final destination will still be the target navigation point.

[0074] In one embodiment, if the target navigation point cannot be obtained according to the position information, the mobile object is controlled to move in the target movement direction, and the target movement direction is obtained according to the position information of the position point selected by the position selection operation in the image. That is to say, if the calculation of the target navigation point is wrong, or the user selects the sky in the position selection operation on the user interface, or the calculated distance of the target navigation point is too far, etc., only the user's position selection operation is used as the direction control operation, and the control mode of the mobile object is the direction pointing navigation mode, and the moving direction of the mobile object is controlled according to the position information of the position point selected by the position selection operation in the image. For example, if the position information is just above the center of the image, the mobile object is controlled to move upward, and if there is an upper left corner, the mobile object is controlled to move to the upper left corner.

[0075] The embodiments of the present invention facilitate users to determine a location point based on captured images to achieve navigation of a mobile object. Users can intuitively perform pointing navigation operations on the user interface, allowing the mobile object to move directly to a location where the target object can be effectively observed, thereby improving the accuracy of the mobile object in performing related observation tasks and improving the efficiency of task execution. Furthermore, during the movement process, users can also intuitively control the flight direction and course angle of the mobile object through the user interface so that the mobile object can avoid obstacles during autonomous navigation. At the same time, different operations can be intelligently obtained based on different user operations to complete different processing, more effectively meeting the user's demand for automated and intelligent control of mobile objects.

[0076] See also Figure 5 , is a schematic diagram of the structure of a navigation processing device according to an embodiment of the present invention. The device according to the embodiment of the present invention can be provided in a smart terminal or a dedicated control device capable of controlling a mobile object such as an aircraft. The device can specifically include the following units.

[0077] The display unit 501 is used to display the received captured image on a preset user interface, where the captured image is captured by a camera device configured on a mobile object; the processing unit 502 is used to determine the position information of the position point selected by the position selection operation in the image if a position selection operation is received on the user interface; the control unit 503 is used to control the mobile object to move toward the target navigation point, where the target navigation point is obtained based on the position information.

[0078] In an optional embodiment, the target navigation point is a position point in a world coordinate system determined according to the position information.

[0079] In an optional embodiment, the processing unit 502 is further configured to generate a location icon for the location point selected by the location selection operation, and display the location icon on the user interface.

[0080] In an optional embodiment, the processing unit 502 is also used to display a trigger icon on the user interface, and the trigger icon is used to indicate whether to control the moving object to move toward the target navigation point; when a selection operation on the trigger icon is received, the control of the moving object to move toward the target navigation point is triggered.

[0081] In an optional embodiment, the control unit 503 is specifically used to control the moving object to move toward the target navigation point according to preset operating altitude information; wherein the operating altitude information includes: the acquired current altitude information of the moving object, or the received configuration altitude information.

[0082] In an optional embodiment, after the mobile object moves into a predetermined area of ​​the target navigation point, it hovers in the predetermined area above the target navigation point according to the operating altitude information.

[0083] In an optional embodiment, the control unit 503 is also used to adjust the size of the location icon according to the distance between the moving object and the target navigation point during the movement of the moving object; wherein the size of the location icon is used to indicate the size of the distance between the moving object and the target navigation point.

[0084] In an optional embodiment, the control unit 503 is also used to, during the movement of the moving object, if a position update operation regarding the moving object is received, determine the updated position information of the position point updated by the position update operation in the image; and control the moving object to move toward the updated navigation point, which is obtained based on the updated position information.

[0085] In an optional embodiment, the control unit 503 is further configured to control the heading angle of the mobile object according to the heading control operation detected on the user interface, so that the mobile object flies according to the new heading angle.

[0086] In an optional embodiment, the control unit 503 is specifically used to send a rotation control instruction to the mobile object based on the object position point indicated in the heading control operation detected on the user interface; the rotation control instruction is used to control the mobile object to rotate to a new heading angle so that the image object of the object position point is within the target area of ​​the image captured by the camera device.

[0087] In an optional embodiment, during the movement of the mobile object, the mobile object is in a hovering state when a first type of obstacle is detected, and performs obstacle avoidance movement when a second type of obstacle is detected, and the obstacle avoidance movement is used to bypass the second type of obstacle during the movement toward the target navigation point.

[0088] In an optional embodiment, the control unit 503 is further configured to, during the movement of the moving object, if a movement direction adjustment operation is detected on the user interface, issue a control instruction to the moving object to control the current movement direction of the moving object.

[0089] In an optional embodiment, the processing unit 502 is further configured to generate a grid icon; overlay and display the grid icon on a designated area of ​​the captured image; and monitor and receive a location selection operation on the designated area covered by the grid icon.

[0090] In an optional embodiment, the control unit 503 is also used to determine the position information of the position point selected by the direction selection operation in the image when a direction selection operation is received in an area outside the grid icon in the user interface, and control the moving object to move in the target movement direction, and the target movement direction is determined based on the position information of the position point selected by the direction selection operation in the image. In an optional embodiment, the control unit 503 is also used to control the moving object to move in the target movement direction if the target navigation point cannot be obtained based on the position information, and the target movement direction is obtained based on the position information of the position point selected by the position selection operation in the image.

[0091] In an optional embodiment, the processing unit 502 is also used to detect a flight control instruction, and if the flight control instruction is a first control instruction, the mobile object is controlled to move toward a target navigation point; the control unit 503 is also used to control the mobile object to move toward a target motion direction if the flight control instruction is a second control instruction, and the target motion direction is obtained according to the position information of the position point selected by the position selection operation in the image.

[0092] It is understood that the various user interface operations mentioned in the embodiments of the present invention, such as the aforementioned location selection operation, the selection operation of the trigger icon, the location update operation, the heading control operation, the movement direction adjustment operation, and the like, can be pre-configured as needed. For example, they can be configured as the aforementioned long press, single click, double click, and other user operations as needed. When configuring these operations, the configuration is based on the premise that no erroneous processing will occur. For example, in a simple implementation, the same user operation will not trigger two or more different processes.

[0093] The specific implementation of each unit in the device of the embodiment of the present invention can refer to the description of the relevant steps and contents in the aforementioned embodiment, which will not be repeated here.

[0094] The embodiments of the present invention facilitate users to determine a location point based on captured images to achieve navigation of a mobile object. Users can intuitively perform pointing navigation operations on the user interface, allowing the mobile object to move directly to a location where the target object can be effectively observed, thereby improving the accuracy of the mobile object in performing related observation tasks and improving the efficiency of task execution. Furthermore, during the movement process, users can also intuitively control the flight direction and course angle of the mobile object through the user interface so that the mobile object can avoid obstacles during autonomous navigation. At the same time, different operations can be intelligently obtained based on different user operations to complete different processing, more effectively meeting the user's demand for automated and intelligent control of mobile objects.

[0095] See also Figure 6 , is a schematic diagram of the structure of a control device according to an embodiment of the present invention. The control device according to this embodiment of the present invention may be an intelligent terminal having at least communication and display functions, specifically a smart phone, tablet computer, or other intelligent terminal. The control device may include a power supply, physical buttons, and other structures as needed. The control device also includes: a communication interface 601, a user interface 602, a memory 603, and a processor 604.

[0096] The user interface 602 is primarily a module such as a touch screen, used to display a user interface to the user and also receive user touchscreen operations. The communication interface 601 may be an interface based on WiFi hotspots and / or radio frequency communication. Through this communication interface 601, the control device can exchange data with a mobile object such as an aircraft, for example, receiving images captured by a camera on the mobile object and sending control instructions to the mobile object.

[0097] The memory 603 may include a volatile memory, such as a random-access memory (RAM); the memory 603 may also include a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 603 may also include a combination of the above types of memory.

[0098] The processor 604 may be a central processing unit (CPU). The processor 604 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0099] Optionally, the memory 603 is further configured to store program instructions. The processor 604 may call the program instructions to implement the navigation processing method in the above embodiment.

[0100] In one embodiment, the memory 603 is used to store program instructions; the processor 604 calls the program instructions stored in the memory 603 to perform the following steps:

[0101] Displaying a received captured image on a preset user interface, wherein the captured image is captured by a camera device disposed on the moving object;

[0102] If a position selection operation is received on the user interface, determining position information of the position point selected by the position selection operation in the image;

[0103] The mobile object is controlled to move toward a target navigation point, where the target navigation point is obtained based on the position information.

[0104] In an optional embodiment, the target navigation point is a position point in a world coordinate system determined according to the position information.

[0105] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0106] A location icon is generated for the location point selected by the location selection operation, and the location icon is displayed on the user interface.

[0107] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0108] Displaying a trigger icon on the user interface, wherein the trigger icon is used to indicate whether to control the moving object to move toward the target navigation point;

[0109] When a selection operation on the trigger icon is received, the control of the moving object to move toward the target navigation point is triggered.

[0110] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 to specifically perform the following steps when executing the step of controlling the moving object to move toward the target navigation point:

[0111] Controlling the moving object to move toward the target navigation point according to the preset operating altitude information;

[0112] The operating altitude information includes: the acquired current altitude information of the moving object, or the received configuration altitude information.

[0113] In an optional embodiment, after the mobile object moves into a predetermined area of ​​the target navigation point, it hovers in the predetermined area above the target navigation point according to the operating altitude information.

[0114] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0115] During the movement of the mobile object, adjusting the size of the location icon according to the distance between the mobile object and the target navigation point;

[0116] The size of the location icon is used to indicate the distance between the moving object and the target navigation point.

[0117] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0118] During the movement of the mobile object, if a position update operation on the mobile object is received, determining updated position information of a position point updated by the position update operation in the image;

[0119] The mobile object is controlled to move toward an updated navigation point, where the updated navigation point is obtained according to the updated position information.

[0120] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0121] According to the heading control operation detected on the user interface, the heading angle of the mobile object is controlled so that the mobile object flies according to the new heading angle.

[0122] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 to specifically perform the following steps when executing the step of controlling the heading angle of the mobile object according to the heading control operation detected on the user interface:

[0123] sending a rotation control instruction to the mobile object according to a position point of the object indicated in the heading control operation detected on the user interface;

[0124] The rotation control instruction is used to control the mobile object to rotate to a new heading angle so that the image object of the object position point is within the target area of ​​the image captured by the camera device.

[0125] In an optional embodiment, during the movement of the mobile object, the mobile object is in a hovering state when a first type of obstacle is detected, and performs obstacle avoidance movement when a second type of obstacle is detected, and the obstacle avoidance movement is used to bypass the second type of obstacle during the movement toward the target navigation point.

[0126] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0127] During the movement of the moving object, if a movement direction adjustment operation is detected on the user interface, a control instruction is sent to the moving object to control the current movement direction of the moving object.

[0128] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0129] Generate grid icons;

[0130] Overlaying and displaying the grid icon on a designated area of ​​the captured image;

[0131] Listen for and receive a location selection operation on a designated area covered by the grid icon.

[0132] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0133] When a direction selection operation is received in an area outside the grid icon in the user interface, determining position information of a position point selected by the direction selection operation in the image;

[0134] The moving object is controlled to move in a target moving direction, where the target moving direction is determined according to position information of the position point selected by the direction selection operation in the image.

[0135] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0136] If the target navigation point cannot be obtained according to the position information, the mobile object is controlled to move in a target movement direction, where the target movement direction is obtained according to the position information of the position point selected by the position selection operation in the image.

[0137] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0138] Detect flight control commands;

[0139] If the flight control instruction is the first control instruction, controlling the mobile object to move toward the target navigation point;

[0140] In an optional embodiment, the processor 604 calls the program instructions stored in the memory 603 and is further configured to perform the following steps:

[0141] If the flight control instruction is the second control instruction, the mobile object is controlled to move in a target movement direction, where the target movement direction is obtained according to position information of the position point selected by the position selection operation in the image.

[0142] The functional modules of the control device in the embodiment of the present invention, especially the specific implementation of the processor 604, can refer to the description of the relevant steps and contents in the aforementioned embodiment, which will not be repeated here.

[0143] The embodiments of the present invention facilitate users to determine a location point based on captured images to achieve navigation of a mobile object. Users can intuitively perform pointing navigation operations on the user interface, allowing the mobile object to move directly to a location where the target object can be effectively observed, thereby improving the accuracy of the mobile object in performing related observation tasks and improving the efficiency of task execution. Furthermore, during the movement process, users can also intuitively control the flight direction and course angle of the mobile object through the user interface so that the mobile object can avoid obstacles during autonomous navigation. At the same time, different operations can be intelligently obtained based on different user operations to complete different processing, more effectively meeting the user's demand for automated and intelligent control of mobile objects.

[0144] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the navigation processing method mentioned in the above embodiment is implemented.

[0145] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0146] The above disclosure is only part of the embodiments of the present invention, which certainly cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A navigation processing method, characterized in that: include: Displaying an acquired captured image on a user interface, wherein the captured image is captured by a camera device disposed on the moving object; In response to obtaining a position selection operation on a designated area of ​​the captured image displayed on the user interface, determining relevant position information of a target navigation point corresponding to the position point selected by the position selection operation; Controlling the mobile object to move toward the target navigation point according to relevant position information of the target navigation point, wherein the relevant position information of the target navigation point is determined at least according to the position information of the mobile object and the posture information of the gimbal mounted on the mobile object; as well as A virtual reality location icon is displayed at the location point selected by the location selection operation, and the closer the distance between the moving object and the target navigation point is, the larger the size of the virtual reality location icon in the user interface is.

2. The method according to claim 1, wherein Also includes: In the process of controlling the moving object to move toward the target navigation point, in response to obtaining a position update operation on a specified area of ​​the captured image displayed on the user interface, determining updated position information of the position point selected by the position update operation in the captured image; as well as Based on the updated position information, the mobile object is controlled to move toward an updated navigation point, wherein the updated navigation point is obtained based on the updated position information.

3. The method according to claim 1, wherein Also includes: Generate grid icons; Overlaying and displaying the grid icon on a designated area of ​​the captured image; Listen for and obtain location selection operations on the specified area covered by the grid icon.

4. The method according to claim 1, wherein The target navigation point is a position point in the world coordinate system determined according to the position information.

5. The method according to claim 1, wherein The captured images include first person view (FPV) images.

6. The method according to claim 1, wherein Also includes: In the process of controlling the moving object to move toward the target navigation point, obstacles in the moving direction are detected so that the moving object can autonomously avoid the obstacles.

7. The method according to claim 1, wherein Also includes: In the process of controlling the moving object to move toward the target navigation point, in response to detecting a moving direction adjustment operation on the user interface, issuing a control instruction to the moving object to adjust the current moving direction of the moving object; as well as After the moving object moves for a specified time in the adjusted moving direction, the moving object is controlled to continue moving toward the target navigation point.

8. A navigation processing method, characterized in that: include: Displaying an acquired captured image on a user interface, wherein the captured image is captured by a camera device disposed on the moving object; In response to a position selection operation acquired on a designated area of ​​the captured image displayed on the user interface, determining position information of a position point selected by the position selection operation in the image; Controlling the mobile object to move toward a target navigation point, where the target navigation point is obtained based on the position information; In response to acquiring a direction selection operation outside a designated area of ​​the captured image displayed on the user interface, determining position information of a position point selected by the direction selection operation in the image; Controlling the moving object to move in a target movement direction, wherein the target movement direction is determined according to position information of the position point selected by the direction selection operation in the image; In the process of controlling the moving object to move toward the target navigation point, in response to detecting a moving direction adjustment operation on the user interface, issuing a control instruction to the moving object to adjust the current moving direction of the moving object; as well as After the moving object moves in the adjusted moving direction for a specified time, controlling the moving object to continue moving toward the target navigation point; The designated area includes an area corresponding to the ground portion in the captured image; and / or the area outside the designated area includes an area corresponding to the sky portion in the captured image.

9. The method according to claim 8, wherein The captured images include first person view (FPV) images.

10. The method according to claim 8, wherein Also includes: Generate grid icons; The grid icon is overlaid and displayed on a designated area of ​​the captured image.

11. The method according to claim 8, wherein The target navigation point is a position point in the world coordinate system determined according to the position information.

12. A navigation processing method, characterized in that: include: Displaying an acquired captured image on a user interface, wherein the captured image is captured by a camera device disposed on the moving object; generating a grid icon, and displaying the grid icon over a designated area of ​​the captured image; Listening for and obtaining a location selection operation on a designated area covered by the grid icon; In response to acquiring the position selection operation on the designated area covered by the grid icon, determining position information of the position point selected by the position selection operation in the image; Controlling the mobile object to move toward a target navigation point, where the target navigation point is obtained based on the position information; In the process of controlling the moving object to move toward the target navigation point, in response to detecting a moving direction adjustment operation on the user interface, issuing a control instruction to the moving object to adjust the current moving direction of the moving object; as well as After the moving object moves in the adjusted moving direction for a specified time, controlling the moving object to continue moving toward the target navigation point; The designated area includes an area corresponding to the ground portion in the captured image; and / or the area outside the designated area includes an area corresponding to the sky portion in the captured image.

13. The method according to claim 12, wherein: Also includes: In the process of controlling the moving object to move toward the target navigation point, in response to obtaining a position update operation on a specified area of ​​the captured image displayed on the user interface, determining updated position information of the position point selected by the position update operation in the captured image; as well as Based on the updated position information, the mobile object is controlled to move toward an updated navigation point, wherein the updated navigation point is obtained based on the updated position information.

14. The method according to claim 12, wherein: The target navigation point is a position point in the world coordinate system determined according to the position information.

15. A control device, characterized in that: The control device includes: a memory and a processor; The memory is used to store program instructions; The processor calls the program instructions stored in the memory to execute the steps of any one of claims 1-14.

Images