Work planning method and device of movable platform and storage medium

Abstract

The invention discloses an operation planning method and device of a movable platform and a storage medium. The method comprises the following steps: acquiring a three-dimensional model of a working area of a movable platform; controlling a motion track of the virtual movable platform in the three-dimensional model according to the detected motion control operation; and determining a plurality of target position points on the movement track according to the detected position confirmation operation, wherein the plurality of target position points are used for generating a working path of the movable platform in the working area. By simulating control over the movable platform in the real world, the movement track of the virtual movable platform in the three-dimensional model of the operation area is controlled, so that the target position point is determined, and the operation path determined in the mode is safer, more reasonable and higher in accuracy.

Description

Operation planning method, device and storage medium for movable platform Technical Field

[0001] The embodiments of the present application relate to the technical field of mobile platforms, and more specifically, to a method, device, and storage medium for planning operations on a mobile platform. Background Art

[0002] Mobile platforms are widely used in many fields, such as inspections, fruit picking, pesticide spraying, and watering, all performed using mobile platforms, both aircraft and ground platforms. Before using a mobile platform for operations, it is often necessary to plan the operation to determine the platform's position within the work area. Reasonable operation planning is essential for ensuring safe operation and achieving good results. Therefore, a solution for mobile platform operation planning is essential.

[0003] Summary of the Invention

[0004] In view of this, the present application provides an operation planning method, device and storage medium for a mobile platform.

[0005] According to a first aspect of the present application, a method for planning an operation of a movable platform is provided, the method comprising:

[0006] obtaining a three-dimensional model of the operating area of ​​the movable platform;

[0007] controlling a motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation;

[0008] A plurality of target position points on the motion trajectory are determined according to the detected position confirmation operation, and the plurality of target position points are used to generate an operation path of the movable platform in the operation area.

[0009] According to a second aspect of the present application, a device for planning an operation of a mobile platform is provided. The device includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, the following steps can be implemented:

[0010] Obtaining a three-dimensional model of the operating area of ​​the movable platform;

[0011] controlling a motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation;

[0012] A plurality of target position points on the motion trajectory are determined according to the detected position confirmation operation, and the plurality of target position points are used to generate an operation path of the movable platform in the operation area.

[0013] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the method mentioned in the first aspect is implemented.

[0014] Using the solution provided by this disclosure, when planning a mobile platform's operations, similar to controlling the movement of a mobile platform within a work area in the real world, a virtual mobile platform can be controlled to continuously move within a three-dimensional model of the work area. When the virtual mobile platform reaches a suitable position for the operation, a position confirmation operation is triggered, thereby obtaining multiple target locations for generating the mobile platform's operation path within the work area. By simulating the control of the mobile platform in the real world and controlling the continuous movement of the virtual mobile platform within the three-dimensional model of the work area, the operation path determined in this way is safer, more reasonable, and more accurate.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] FIG1 is a schematic diagram of an application scenario of an embodiment of the present application.

[0018] FIG2 is a flow chart of a method for planning an operation of a movable platform according to an embodiment of the present application.

[0019] FIG3 is a schematic diagram showing the movement of a virtual movable platform in a three-dimensional model of an operating area on an interactive interface according to an embodiment of the present application.

[0020] FIG4 is a schematic diagram of adjusting an observed image to obtain a sample image according to an embodiment of the present application.

[0021] FIG5 is a schematic diagram of editing a target location point according to an embodiment of the present application.

[0022] FIG6( a ) is a schematic diagram of a three-dimensional model from a first-person perspective according to an embodiment of the present application.

[0023] FIG6( b ) is a schematic diagram of a three-dimensional model from a top-down perspective of a map according to an embodiment of the present application.

[0024] FIG7 is a schematic diagram of displaying safety prompt information on an interactive interface according to an embodiment of the present application.

[0025] FIG8 is a schematic diagram showing a global target location point according to an embodiment of the present application.

[0026] FIG9 is a schematic diagram of the logical structure of an operation planning device for a movable platform according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] Movable platforms are widely used in many fields. For example, inspections, fruit picking, or drug spraying and watering are carried out through movable platforms such as aircraft and ground platforms. Before using the movable platform to perform the above operations, it is usually necessary to plan the operation of the movable platform and determine the various target positions when the movable platform is operating. For example, taking the inspection of power equipment as an example, it is necessary to pre-plan the target position of the mobile platform when taking pictures of each power equipment to ensure that the target image containing the power equipment can be captured at the target position. Furthermore, the operating parameters of the load carried by the movable platform, such as the direction and magnification of the shooting device, can also be recorded at the target position to make the target image captured at the target position more accurate.

[0029] In related technologies, when planning operations on a mobile platform, some require users to manually control the mobile platform directly at the work site. For example, the user manually adjusts the position of the mobile platform based on the user's observations or the operation results returned by the mobile platform, and records the adjustment to the appropriate position so that the mobile platform can be controlled according to the planned position during subsequent operations. This method requires users to manually control the mobile platform at the work site, which requires high user control skills, is labor-intensive, has low work efficiency, and also poses safety risks.

[0030] In order to facilitate users to plan operations for the movable platform, some technologies also provide software for users to plan operations. For example, the operation planning software can display a three-dimensional model of the operation area in an interactive interface, and the user can determine the position of the movable platform during operation by operating on the interactive interface displaying the three-dimensional model of the operation area. For example, the user can move or rotate the three-dimensional model of the operation area in the three-dimensional model, perform a click operation at a position near the operation object to determine the target operation object, and then calculate the position during operation based on relevant operation distance and other information. Although this method does not require the user to manually operate the movable platform on site, it is not intuitive and requires the user to repeatedly adjust to obtain the operation position, which is particularly cumbersome.

[0031] However, when using current operation planning software for operation planning, each target location point on the operation path is planned independently, and there is no connection between different target location points. When planning different target location points, it is also impossible to know the passability and safety of the operation path connecting these target location points, such as whether the planning of the operation path is safe and reasonable, whether there are obstacles on the operation path, whether the operation path can guarantee the operation effect, etc., which may lead to the final planned operation path being not safe enough or not reasonable enough.

[0032] Based on this, an embodiment of the present application provides a method for planning operations of a movable platform. When planning operations of a movable platform, similar to controlling the movement of a movable platform in an operation area in the real world, a virtual movable platform can be controlled to continuously move in a three-dimensional model of the operation area, and when the virtual movable platform moves to a position suitable for the operation, a position confirmation operation is triggered, thereby obtaining multiple target position points for generating an operation path of the movable platform in the operation area. By simulating the control of the movable platform in the real world, the continuous movement of the virtual movable platform in the three-dimensional model of the operation area is controlled. The operation path determined in this way will be safer, more reasonable, and more accurate.

[0033] The job planning method provided in the embodiment of the present application can be executed by an APP or web application running on the terminal. For example, the APP or web application can be a job path planning software; or the method can also be executed by a cloud server or server cluster; or some processing steps of the method are executed by the cloud server or server cluster, and the step processing steps are executed by the APP or web application on the terminal. The specific settings can be flexibly made based on actual needs, and the embodiment of the present application does not impose any restrictions.

[0034] Exemplarily, the method can be performed by any device with sufficient performance to support the acquisition and display of a 3D model. For example, in some scenarios, as shown in Figure 1, a cloud server can capture images of the work area captured by a mobile platform (using an aircraft as an example in the figure) and then pre-build a 3D model of the mobile platform's work area based on these images. A client on a terminal (which can be an app or web application) can then retrieve this 3D model from the cloud server to plan the work path based on this 3D model. Of course, in some scenarios, if the terminal's performance is sufficient to build a 3D model, both 3D model construction and work path planning can be performed on the terminal. The terminal can be a mobile phone, tablet, computer, remote control, or other device. For example, if the terminal is a computer, the 3D model of the work area can be built using the computer's software or webpage. Furthermore, the 3D model can be displayed on a display screen. Optionally, the work path planning can be implemented using a mouse, keyboard, or other means. For example, if the terminal is a remote control, the 3D model of the work area can be built using an app on the remote control. Furthermore, the 3D model can be displayed through the remote control's UI, and the work path planning can be implemented based on the UI and the remote control's joystick.

[0035] The movable platform of the embodiment of the present application can be a movable platform for performing operations on a target object. The movable platform includes a power component for driving the movable platform to move. The movable platform can be a movable device such as an aircraft, a vehicle, a ship, an intelligent robot, etc. The movable platform can be manned for operation or an unmanned platform device. Optionally, the movable platform includes a load for performing operations, which can be a camera, a robotic arm, a hanging system, a spraying system, etc. The embodiment of the present application does not specifically limit the type of load.

[0036] In some embodiments, the aircraft may include a rotary-wing aircraft, such as a quadrotor, a hexacopter, or an octorotor; a fixed-wing aircraft; or a combination of a rotary-wing aircraft and a fixed-wing aircraft. Optionally, the aircraft includes an unmanned aerial vehicle. The aircraft may include, but is not limited to, any of manned aircraft, logistics aircraft, aerial photography aircraft, agricultural plant protection aircraft, and industrial rescue aircraft. The above is merely illustrative, and the embodiments of this application do not specifically limit the type of aircraft.

[0037] Any description herein of an aircraft, such as an unmanned aerial vehicle, may be applicable to and used with any movable object, such as any vehicle. Furthermore, the methods, devices, and systems disclosed herein in the context of aerial motion may also be applicable to other types of motion, such as motion on land or on water, underwater motion, or motion in space.

[0038] Specifically, as shown in FIG2 , the operation planning method may include the following steps:

[0039] S202, obtaining a three-dimensional model of the operating area of ​​the movable platform;

[0040] In step S202, a three-dimensional model of the working area of ​​the movable platform can be obtained. This can be done by calling a previously generated three-dimensional model or by generating a three-dimensional model in real time based on demand. There are many ways to generate this three-dimensional model. The three-dimensional model of the working area can be obtained by taking an image of the working area and then reconstructing the working area in three dimensions based on the image, or by collecting a three-dimensional point cloud of the working area using a laser radar and then obtaining the model based on the three-dimensional point cloud. It is not difficult to understand that any method of obtaining a three-dimensional model of the working area is applicable to the embodiments of the present application, and the embodiments of the present application are not limited thereto.

[0041] S204, controlling the motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation;

[0042] In step S204, after acquiring the three-dimensional model of the work area, motion control operations on the virtual movable platform can be detected, and then the motion trajectory of the virtual movable platform in the three-dimensional model can be controlled based on the detected motion control operations. The virtual movable platform can be an identifier used to represent the movable platform. For example, the identifier can be a three-dimensional model corresponding to the movable platform, an image identifier representing the movable platform, or a simple dot representing the movable platform.

[0043] Motion control operations on the movable platform can be input by the user or automatically generated by the device. For example, taking the example of a user controlling the movement of a virtual movable platform in a three-dimensional model, as shown in Figure 3, after obtaining a three-dimensional model of the work area, the three-dimensional model can be displayed on the user interface, and the virtual movable platform can also be displayed on the interface. Then, the control method of the movable platform in the real world can be simulated to manipulate the virtual movable platform to move continuously in the three-dimensional model. After detecting the motion control operation input by the user, the device executing this method can control the virtual movable platform to perform the corresponding movement in the three-dimensional model.

[0044] Among them, the device that executes the method may include a user interaction entrance, through which the user's motion control operation is detected. Exemplarily, the device that executes the method can be connected to a remote control, and the user can input motion control operations through the joystick on the remote control to control the movement of the virtual movable platform. Alternatively, the device that executes the method can also be connected to physical control devices such as a keyboard and a mouse, and the user controls the movement of the virtual movable platform through these physical control devices. Alternatively, the device that executes the method may include a touch screen, and the user can control the movement of the virtual movable platform through virtual buttons on the touch screen. Alternatively, the device that executes the method can be connected to the user's VR glasses, and the user can control the movement of the virtual movable platform through the VR glasses. It is not difficult to understand that any method that can enable the user to interact with the device that executes the method to input motion control instructions is applicable in the embodiments of the present application, and the embodiments of the present application are not limited.

[0045] Of course, in some scenarios, motion control operations can also be automatically generated by the device. For example, the device can automatically analyze the surrounding environment or scene of the virtual movable platform and then control the movement of the virtual movable platform based on the analysis results. In other words, the device automatically simulates human control operations to achieve automatic control of the virtual movable platform.

[0046] S206 : Determine a plurality of target position points on the motion trajectory according to the detected position confirmation operation, wherein the plurality of target position points are used to generate an operation path of the movable platform in the operation area.

[0047] In step S206, while controlling the movement of the virtual movable platform in the three-dimensional model based on the detected motion control operation, a position confirmation operation can be detected in real time. After the position confirmation operation is detected, the current position of the virtual movable platform can be used as a target position point, and then, based on the determined multiple target position points, an operation path of the movable platform in the operation area is generated. The position confirmation operation can be triggered after determining that the movable platform can accurately operate on the target object when it is at the current position. The position confirmation operation can be triggered by the user. For example, the user can trigger the position confirmation operation when determining that the current position is a position suitable for operating on the target object. The user can trigger the position confirmation operation by inputting a control instruction through voice, through a control component (such as a keyboard, mouse, joystick, etc.), or by clicking a designated icon on the touch screen.

[0048] Of course, the location confirmation operation can also be automatically triggered by the device. For example, the conditions that each target location point must meet can be set in advance. When the virtual movable platform moves to a certain position, the device can determine in real time whether the position meets the preset conditions. If so, the location confirmation operation is automatically triggered.

[0049] Through this method, multiple target locations can be determined. Based on these multiple target locations, a path can then be generated for the mobile platform as it navigates the work area. This entire planning process simulates controlling the movement of the mobile platform within the work area in the real world. This allows for the control of the virtual mobile platform's trajectory within the 3D model, allowing for cost-effective verification of the appropriate target location selection. Consequently, determining the work path in this manner is safer, more efficient, and more accurate.

[0050] Furthermore, when planning the operation path, the relationship between different target position points, the passability and safety of the connection paths between different target position points can be known, such as whether the planning of the operation path is safe and reasonable, whether there are obstacles on the operation path, and whether the operation path can guarantee the operation effect, so as to achieve the effect of "what you see is what you get in operation planning", that is, the operation path formed by controlling the target position points traversed during the movement of the virtual movable platform in the three-dimensional model is the situation of the operation path when the movable platform operates in a real environment, so that determining the operation path in this way is more reasonable, safe and accurate.

[0051] Furthermore, displaying the observation image of the virtual model observed during the movement of the virtual movable platform from the first target position point to the second target position point can facilitate users to understand the accessibility and safety of the connection path between the target position points, as well as the surrounding environment information.

[0052] In some embodiments, the motion control logic for the virtual movable platform can simulate the control logic for a movable platform in the real world. Therefore, the motion control operation can include one or more of a lateral control operation, a longitudinal control operation, an altitude control operation, and a yaw control operation for the virtual movable platform. The lateral control operation, the longitudinal control operation, and the altitude control operation can respectively control the movement of the virtual movable platform in six directions: left and right, front and back, and up and down. The yaw control operation can control the yaw angle of the virtual movable platform. Through these control operations, the control of the virtual movable platform can be similar to the control of the movable platform in the real world.

[0053] In some embodiments, the motion control operation can be triggered by a control component, and the motion control amount of the motion control operation can be determined based on the detected user's manipulation speed of the control component (i.e., the change in the manipulation amount per unit time) and / or the manipulation amount. Among them, the control component can be a physical component such as a mouse, keyboard, rocker, etc., or a virtual control component on a touch screen. When controlling a virtual movable platform, the user's manipulation speed of the control component can be mapped to the movement speed of the virtual movable platform. For example, the user's stick speed is mapped to the movement speed of the virtual movable platform. In addition, the user's manipulation amount of the control component can be mapped to the movement distance or rotation angle of the virtual movable platform. For example, the rocker offset can be mapped to the movement distance of the virtual movable platform. For another example, if the control component can be a keyboard, the manipulation amount of the user's control component can be determined based on the detected pressing time of the keyboard key, and then mapped to the movement distance or rotation angle of the virtual movable platform.

[0054] In some embodiments, the motion control operation of the virtual movable platform can be triggered by a joystick, and the motion control amount of the motion control operation can be determined based on the detected offset of the joystick. The joystick can be a physical joystick or a virtual joystick.

[0055] Exemplarily, a physical joystick can be connected to a device that executes the method, and the direction of the physical joystick can be mapped to the movement direction of the virtual movable platform, and the offset of the physical joystick can be mapped to the motion control amount of the virtual movable platform.

[0056] For example, the user's keyboard or mouse control can be mapped to a change in the virtual joystick, which can then be further mapped to a movement control amount for the virtual movable platform. For example, different keys on the keyboard can be mapped to different stick directions of the virtual joystick, which can then be mapped to movement directions of the virtual movable platform.

[0057] When planning operations for a mobile platform, it is necessary not only to plan the operating positions of the mobile platform but also to plan the operating parameters of the mobile platform when operating at each operating position so that the mobile platform can operate accurately. Therefore, an operation parameter confirmation operation can also be detected. Based on the detected operation parameter confirmation operation, the operation parameters corresponding to the target location are determined and stored. The operation parameters are used to instruct the mobile platform to arrive at the target location and perform the operation.

[0058] The operation parameter confirmation operation can be triggered by the user or automatically generated by the device. The types of operation parameters vary depending on the scenarios in which the loads carried on the mobile platform are different. For example, if the load carried by the mobile platform is a shooting device and its operation task is to take pictures of the target object, then the operation parameters can be the camera orientation, camera magnification, exposure parameters, etc. If the load carried by the mobile platform is a hanging system and its operation task is to transport goods, then the operation parameters can be the weight of the loaded goods, etc. If the load carried by the mobile platform is a robotic arm and its operation task is to pick fruits or grab objects, then the operation parameters can be the grab distance, grab angle, etc. If the load carried by the mobile platform is a spraying system and its operation task is to spray medicine, fertilize or water, then the operation parameters can be the spraying amount, spraying speed, spraying time, etc. The specific types of operation parameters can be flexibly set based on the actual application scenario.

[0059] In some embodiments, the movable platform may be equipped with a camera payload, and the operation parameter may be an orientation parameter of the camera payload in space. This orientation parameter is used to instruct the camera payload to perform an operation according to the orientation parameter when the movable platform reaches the target location. For example, when the movable platform reaches the target location, the camera may adjust its orientation to the orientation indicated by the orientation parameter and then perform the photo task.

[0060] In some embodiments, the camera may be equipped with a gimbal, and the gimbal may be controlled to rotate based on the operating parameters to adjust the camera's payload to the orientation indicated by the operating parameters.

[0061] In some embodiments, the orientation of the camera payload carried on the movable platform is adjusted to determine precise operating parameters. In the process of controlling the movement of the virtual movable platform, an orientation control operation for controlling the orientation of a virtual observation device on the virtual movable platform can be detected, and then the orientation of the virtual observation device is controlled based on the detected orientation control operation. After the orientation confirmation operation is detected, the target orientation of the virtual observation device when the virtual movable platform is located at each target position point on the motion trajectory can be determined, wherein when the movable platform moves to each target position point, the camera payload carried by the movable platform can be controlled to operate according to the target orientation. The orientation control operation and the orientation confirmation operation can be triggered by the user or automatically generated by the device.

[0062] In some embodiments, the position confirmation operation and the operation parameter confirmation operation can be determined based on the same operation triggered by the user. For example, when a user-triggered confirmation operation is detected, the current position of the virtual movable platform and the current operation parameters are simultaneously recorded. In some embodiments, the position confirmation operation and the operation parameter confirmation operation can also be different operations triggered by the user. For example, the user can first trigger a confirmation operation to determine the target location point, and then trigger another confirmation operation to determine the operation parameters corresponding to the target location point. For another example, the user can first trigger a confirmation operation to determine the operation parameters corresponding to the target location point, and then automatically trigger another confirmation operation to determine the target location point corresponding to the operation parameters.

[0063] In some embodiments, after determining the target location point, the operation result corresponding to the target location point can be further determined, and the target location point and / or the operation parameters corresponding to the target location point can be adjusted based on the operation result. The operation result can be used to indicate the deviation between the completion of the operation and the ideal situation when the movable platform is located at the target location point for operation. The adjusted target location point is used to regenerate the operation path of the movable platform in the operation area, and the adjusted operation parameters are used to indicate the operation parameters when the movable platform is located at the adjusted target location point and its load is operating. By automatically adjusting the operation path and operation parameters based on the deviation between the completion of the operation and the ideal situation when the movable platform is located at the target location point for operation, the final determined operation parameters and operation path can be made more accurate.

[0064] In some embodiments, as shown in FIG3 , during the process of controlling the movement of a virtual movable platform in a three-dimensional model, the observation angle of a virtual observation device mounted on the virtual movable platform can be determined based on the position of the virtual movable platform during the movement, and the three-dimensional model can be projected onto the observation image obtained from the observation angle, and the observation image can be displayed in real time on the user interaction interface. For example, during the process of controlling the movement of the virtual movable platform in the three-dimensional model, the observation image of the virtual movable platform or the virtual load on the virtual movable platform relative to the three-dimensional model can be displayed in real time through the interaction interface, thereby making it easier for the user to understand the operating status of the movable platform when it is at the current position, such as what content is captured by the image captured by the camera mounted on the movable platform, whether the robotic arm mounted on the movable platform can grab the fruit, or whether the spraying device mounted on the movable platform can be aimed at the crops to be sprayed, etc.

[0065] In some embodiments, the payload carried by the movable platform may be a camera. During the process of controlling the movement of the virtual movable platform within a three-dimensional model, the orientation information of the virtual camera on the virtual movable platform in space may be obtained. Based on this orientation information, the virtual camera's observation image of the scene within the three-dimensional model is determined and displayed. This allows the user to easily determine whether the desired target object is included in the image captured by the movable platform when the camera takes a picture according to this orientation information.

[0066] In some embodiments, the load on the movable platform is a photographing device, and its operation task is to take pictures of the target object. The operation result can be the deviation between the image taken by the photographing device and the sample image when the movable platform is located at each target position. The sample image can be an image determined based on the user's adjustment operation on the observation image. For example, when the user finds based on the observation image that the current position of the virtual movable platform is not the best position for the movable platform to operate on the target object, the user can adjust the observation image so that the virtual movable platform can obtain a more accurate operation result when it is in a position where the adjusted observation image can be observed. Adjusting the observation image can be adjusting part or all of the pixel area of ​​the observation image, and the adjustment operation can be adjusting the imaging position and size of the content corresponding to the partial pixel area in the observation image.

[0067] For example, the payload carried by the movable platform is a camera, and its task is to photograph the target object in the working area. When controlling the movement of the virtual movable platform in the three-dimensional model, the observation image of the three-dimensional model by the virtual camera on the virtual movable platform can be displayed in real time on the interactive interface (i.e., the image that the virtual camera can capture, which is also the image captured by the camera on the movable platform in the actual working scene). That is, based on the observation image, the content of the image captured by the movable platform when the real working scene is located at that position can be determined, such as whether it can capture the entire target object, whether the target object is located in the center of the picture, etc. As shown in Figure 4, if the target object in the observation image is too small or the target object is not located in the center of the picture, the observation image can be adjusted, for example, by re-selecting the target object in the observation image, or moving the observation image to place the target object in the center of the image, or enlarging the image so that the target object occupies a larger proportion of the picture. Among them, the adjusted observation image is the sample image, and the device executing the method can readjust the target position point and / or working parameters based on the deviation between the observation image and the sample image. For example, the position of the target point can be adjusted based on the position difference of the target object in the image, or the magnification of the camera when taking pictures can be adjusted based on the proportion difference of the target object in the image, etc.

[0068] Furthermore, after detecting an adjustment to the observation image, the adjusted sample image can be displayed on the interactive interface to facilitate access to its contents. For example, the adjusted sample image can be displayed as a new observation image on the interactive interface. For example, when the user views the new observation image and determines that it meets their requirements, a position confirmation operation or an operation parameter confirmation operation can be triggered.

[0069] By adjusting the observation image, the operation path and operation parameters of the movable platform during operation can be automatically adjusted, so that a "what you see is what you get" effect can be achieved. That is, the displayed observation image is the image taken during the operation of the movable platform, making the operation planning of the movable platform more intuitive.

[0070] In some embodiments, when controlling the motion trajectory of a virtual movable platform in a three-dimensional model according to a detected motion control operation, the virtual movable platform can be controlled to leave a first position in the three-dimensional model and move to other areas in the three-dimensional model according to the detected motion control operation, and then the current position of the virtual movable platform in the three-dimensional model is determined as a second position according to the detected position confirmation operation, wherein the first position and the second position are used to generate an operating path of the movable platform in the operating area.

[0071] In some embodiments, in order to facilitate users to adjust the planned operation path, a target location point editing function can be provided. After detecting the user's editing operation, one or more of the following operations can be performed: deleting the determined target location point, adjusting the position of the determined target location point, inserting a new target location point between any two adjacent target location points, and modifying the operation parameters corresponding to the target location point. As shown in Figure 5, taking the route determination of the drone as an example, the user can click on a certain determined waypoint and then edit the waypoint, for example, edit the location of the waypoint, the operation task or the operation parameters, etc. In this way, the user can adjust the determined target location point at any time during the operation planning process, which is more convenient and quick.

[0072] In some embodiments, when the virtual movable platform moves to a certain determined target location point, the information corresponding to the target location point can be displayed on the interactive interface, such as the observation image of the three-dimensional model of the movable platform at the target location point, the operating parameters corresponding to the target location point, the distance information between the movable platform and the surrounding objects when the movable platform is located at the target location point, etc. Therefore, when looking back at the relevant information of the previously determined target location point, the virtual movable platform can be moved to the corresponding target location point. After detecting the moving operation, the device executing this method can move the virtual movable platform from the current position to any determined target location point and display the relevant information of the target location point in the interactive interface.

[0073] In some embodiments, to facilitate user understanding of the current orientation of a virtual observation device mounted on a virtual movable platform, an indicator representing the orientation of the virtual observation device may be displayed on the interactive interface. The indicator indicating the orientation of the virtual observation device may be displayed in real time while the virtual movable platform is being controlled to move within the three-dimensional model, or may be displayed when the virtual movable platform is at a target location.

[0074] In some embodiments, after determining the target location, in order to facilitate understanding of the surrounding environment information of the current location, the scenery in the three-dimensional model around the target location can be displayed in the interactive interface so that the real working environment of the movable platform can be clearly perceived.

[0075] In some embodiments, after acquiring a 3D model of the work area, the 3D model can be displayed on an interactive interface. To facilitate displaying the 3D model's information from different angles, the 3D model can be displayed from multiple perspectives on the interactive interface, and the perspective of the 3D model can be switched based on actual needs. For example, upon detecting a user input for a perspective switch, the perspective of the 3D model on the interactive interface can be switched to the perspective indicated by the perspective switch operation.

[0076] In some embodiments, any two of the multiple perspectives have different viewpoint positions and / or the directions from the viewpoint positions of any two perspectives to the virtual movable platform are different. For example, the multiple perspectives may include a first-person perspective of the movable platform, a top-down perspective of the map, a second-person perspective, and a third-person perspective.

[0077] From the first-person perspective of the movable platform, the observation image can be obtained by observing the three-dimensional model by displaying the movable platform in full screen on the interactive interface. This perspective allows users to clearly see the observation image corresponding to the target position. As shown in Figure 6(a), this is an image of the three-dimensional model displayed from the first-person perspective.

[0078] From a bird's-eye view of the map, the overall information of each target location point in the entire operation path can be displayed, making it easier for users to determine whether any target location point has been missed, as well as the horizontal information of each target location point. As shown in Figure 6(b), this is an image of the three-dimensional model displayed from a bird's-eye view of the map.

[0079] In the second-person perspective, users can easily understand the vertical environmental information of the movable platform.

[0080] When the ground is very undulating and there are many obstacles around, you can switch to the third-person perspective. In the third-person perspective, the user can clearly see the obstacle information around the movable platform.

[0081] In summary, different perspectives display different information, and users can switch to the corresponding perspective based on actual needs to obtain information of different dimensions.

[0082] In some embodiments, the position of the viewpoint of any perspective will change with the movement of the virtual movable platform, and the relative position of the viewpoint and the virtual movable platform remains fixed, thereby ensuring that the environmental information around the motion trajectory of the movable platform during movement can be accurately obtained in real time.

[0083] The related operation planning scheme that uses the first-person perspective to determine the operation parameters does not support the movable platform to operate while moving, nor does it support directly determining the operation parameters other than taking pictures at the current perspective. This application scheme allows users to freely determine the target position point directly from multiple perspectives such as the first perspective, the map bird's-eye view, the third-person following perspective following the tail of the movable platform, and the fixed-angle third-person perspective, and can quickly adjust all relevant parameters related to the target position point.

[0084] In some embodiments, when displaying a three-dimensional model of the work area and a virtual movable platform in an interactive interface, in order to present the user with the virtual movable platform's position information within the three-dimensional model from different angles, the images displayed on the interactive interface may include one or more of an image showing the three-dimensional model as viewed by the virtual movable platform or an image showing the relative positional relationship between the virtual movable platform and the three-dimensional model. The former facilitates the user's understanding of the scenery within the three-dimensional model as viewed by the virtual movable platform at its current location, while the latter facilitates the user's understanding of the surrounding environment of the virtual movable platform at its current location.

[0085] In some embodiments, in order to perceive the environmental information around the movable platform when it is at any target position during the operation planning of the movable platform, such as whether there are obstacles around, whether there are safety hazards, whether the operation path is reasonable, etc. After the target position is determined, safety prompt information can be displayed on the interactive interface. The safety prompt information can be determined based on one or more information of the relative position relationship between the target position point on the motion trajectory of the virtual movable platform and the envelope point of the three-dimensional model, and the change in the motion posture of the virtual movable platform on the motion trajectory. The envelope point can be used to represent the position of an object in the three-dimensional model, for example, it can be the position point corresponding to a tree in the three-dimensional model, or the position point corresponding to the ground, etc.

[0086] In some embodiments, the relative positional relationship between the target position point on the motion trajectory and the envelope point of the three-dimensional model can be the distance between the target position point and the obstacle and / or the height of the target position point relative to the ground. By determining the distance between the target position point and the obstacle, the relative height to the ground, etc., it can be determined whether there is a safety risk during the movement of the virtual movable platform, for example, whether the distance to the obstacle is too close, whether the height from the ground is too low, etc. If a safety risk is found, a safety prompt message can be generated and displayed on the interactive interface to alert the user to the potential risk. As shown in Figure 7, a schematic diagram of displaying safety prompt information on the interactive interface is shown.

[0087] Of course, during motion, the mobile platform must not only consider the distance from obstacles to avoid collisions, but also strive to ensure a smooth trajectory, avoiding sharp turns and the like. Therefore, the virtual mobile platform's motion posture change along the trajectory can also be determined. This motion posture change can be the virtual mobile platform's motion posture change per unit time or per unit distance. If the posture change is excessive, a safety warning message can be displayed on the interactive interface to provide a warning.

[0088] Since the three-dimensional model contains accurate environmental data, relevant obstacles can be displayed on the interactive interface in a timely and accurate manner. Optionally, the distance between the current virtual movable platform and the surrounding obstacles and the posture information of the current virtual movable platform can be displayed in the main window of the interactive interface. Optionally, the distance to obstacles around the current virtual movable platform can be quickly viewed on the interactive interface, and the obstacle status of any target location can be traced back. Optionally, for operation planning in complex environments, the relationship between relevant target locations and obstacles can also be displayed based on the advantage of the actual physical information in the virtual space where the three-dimensional model is located, so as to facilitate a global overview of the target operation point and / or the operation parameters of the target operation point after the operation planning is completed.

[0089] In some embodiments, to facilitate intuitive understanding of the current height of the virtual movable platform from the ground, the safety reminder information may include an auxiliary line indicating the height of the virtual movable platform relative to the ground at the target location, with the height value relative to the ground indicated near the auxiliary line. Alternatively, the safety reminder information may simply display an icon indicating the height of the virtual movable platform relative to the ground at the target location.

[0090] In some embodiments, in addition to displaying the safety prompt information on the interactive interface, the safety prompt information can also be broadcasted through voice prompts.

[0091] In some embodiments, the safety prompt information can be issued in real time while controlling the movement of the virtual movable platform within the 3D model, thereby facilitating the determination of the target location based on the safety prompt information. Alternatively, the prompt information can be issued after the user or device triggers a position confirmation operation to determine the target location. In this case, the target location can be adjusted based on the safety prompt information.

[0092] In some embodiments, as shown in FIG8 , after the user completes the confirmation of each target location point, all target location points of the operation path can also be displayed in the interactive interface, and then the target location points with safety risks can be marked so that the user can understand which target location points of the operation path store safety risks.

[0093] In some embodiments, the task to be performed by the movable platform is to inspect and photograph the target objects in the working area. In such a scenario of repeated inspection and photography, due to insufficient control accuracy of the position and posture of the movable platform (for example, the control accuracy of the pan-tilt head may be insufficient), it may cause a large deviation between the captured image and the expected image when reshooting at a certain target position point, especially in the scene of long-distance zoom shooting. In order to avoid the above problems, in the scenario of repeated inspection and photography, a precise reshooting method can be used to ensure accurate shooting. Precise reshooting means pre-storing sample images of the target object that the movable platform needs to shoot at each target position point, so that when the movable platform performs the shooting task at the target position point, it can adjust the position and posture of the movable platform based on whether the content of the collected image and the sample image is consistent, thereby ensuring that the captured image includes the target object.

[0094] In some embodiments, in order to obtain sample images to guide the mobile platform to capture images of target objects during repetitive inspection tasks, after completing the planning of the work path, the mobile platform can be controlled to operate within the work area according to the planned work path, and the real-life images captured by the mobile platform while operating along the work path are obtained and displayed to the user. Then, the image area selected by the user is captured from the real-life image as a sample image containing the target object. The sample image is used to instruct the mobile platform to capture real-life images containing the target object during subsequent operations.

[0095] For example, assuming that the target object to be photographed is electrical equipment, after completing the operation path planning of the movable platform and determining the operation parameters of the movable platform at each target position point, the movable platform can be controlled to operate according to the planned operation path and operation parameters, and the real-scene image is collected and then displayed to the user. The user can select the target object from the real-scene image, and then the image area selected by the user can be stored as a sample image to guide the movable platform to collect images including the target object when performing subsequent inspection tasks.

[0096] In some embodiments, considering that it is necessary to obtain a three-dimensional model of the work area when performing work planning, and the three-dimensional model is obtained by three-dimensionally reconstructing the scenery of the work area, it is a true portrayal of the work area. Therefore, it is also possible to directly obtain a sample image containing the target object based on the three-dimensional model. For example, the three-dimensional model can be observed to obtain an image containing the target object, and then the image can be directly used as a sample image. For example, in some scenarios, the three-dimensional model is obtained by three-dimensionally reconstructing the image of the work area taken. Therefore, it is also possible to select an image containing the target object with a suitable shooting angle from the image used for three-dimensional reconstruction, and use the image as a sample image. In short, by determining the sample image based on the three-dimensional model, it is possible to obtain the sample image without actually controlling the movable platform to perform a work task. This is more convenient and quicker, and greatly improves the working efficiency of the movable platform.

[0097] In some embodiments, the cloud can employ this method to achieve a closed-loop, unmanned operation planning loop, freeing up manpower. Optionally, by integrating functions such as interactive communication, precise re-shooting, cloud-based post-reconstruction, and virtual space collision monitoring within the 3D model, the safety and efficiency of mobile platform operation planning can be improved. Optionally, the mobile platform can be an unmanned aerial vehicle (UAV). Optionally, the operation planning can include route planning for the UAV.

[0098] In some embodiments, determining the sample image based on the observed image includes any of the following methods: using the observed image as the sample image; or, using the image area where the center position of the observed image is located as the sample image; or, using the image area in the observed image selected by the user as the sample image.

[0099] In some embodiments, when observing a three-dimensional model to obtain a sample image containing a target object, the user can rotate and drag the three-dimensional model to find a viewing angle from which the target object can be observed. The device that executes the method can then automatically use the image containing the target object observed by the user at that viewing angle as a sample image.

[0100] In some embodiments, an image containing the target object obtained by observing the three-dimensional model from the virtual movable platform during its motion can be used as a sample image. For example, during the motion of the virtual movable platform, the observation angle of a virtual observation device mounted on the virtual movable platform can be determined based on the position of the virtual movable platform. An observation image of the three-dimensional model projected onto the observation angle can then be obtained, and the sample image can be determined based on the observation image.

[0101] In some embodiments, considering that the determined target position point is the position point where the target object can be observed, after detecting the input position confirmation operation, when the virtual movable platform is located at the current position, the observation image obtained by observing the three-dimensional model is directly used as a sample image containing the target object.

[0102] In some embodiments, the size of the sample image can be pre-set, that is, the sample image is an image of fixed size. Therefore, after obtaining the observation image obtained by the virtual movable platform observing the three-dimensional model, the image area where the center position of the observation image is located can also be cut out based on the specified size of the sample image as a sample image.

[0103] In some embodiments, to obtain a more accurate and effective sample image, an area containing the target object can be directly selected from the observed image displayed on the interactive interface as the sample image. For example, the target object can be framed in the observed image, and the device executing this method can store the framed image area as the sample image.

[0104] For example, when the user controls the movement of the virtual movable platform in the three-dimensional model, the user can display the observation image of the virtual movable platform on the three-dimensional model in real time on the interactive interface. If the user confirms that the observation image includes the target object to be photographed and the target object is located in the middle of the image, a position confirmation operation can be triggered. At this time, the device executing the method can use the position of the virtual movable platform as the target position point and the observation image displayed on the interactive interface as the sample image, or cut out the image area in the middle position from the observation image as the sample image. Of course, if the user confirms that the target object is not in the middle of the observation image, or the target object occupies a small proportion, the target object can be framed from the observation image first. The device executing the method can then display the framed image on the interactive interface and automatically adjust the position of the target position point based on the user's frame selection operation. If the user feels that the image at this time meets the requirements, a position confirmation operation can be triggered to store the adjusted target position point and use the adjusted observation image as the sample image.

[0105] Among them, the method of determining the sample image through the real scene image and the method of obtaining the sample image directly based on the three-dimensional model each have their own advantages. In actual application, any of the above methods can be flexibly selected based on actual needs. For example, in a scene where the accuracy of the constructed three-dimensional model is low and the size of the target object to be photographed is relatively small, if the sample image is obtained directly based on the three-dimensional model, the clarity of the sample image may be low, and it is not suitable for guiding the movable platform to collect real scene images with the same content as the sample image during operation. Therefore, the sample image can be obtained by collecting the real scene image. In the scene where the accuracy of the three-dimensional model is sufficient to obtain a clear image of the target object, the sample image can be obtained directly through the three-dimensional model, thereby eliminating the need to control the actual operation of the movable platform. It is convenient and fast, and can also avoid the problem that the target object in the collected real scene image is not clear enough due to various reasons such as weather, and the resulting sample image is not clear enough.

[0106] Alternatively, the method of determining sample images from real-world images and the method of obtaining sample images directly from 3D models can be combined to achieve complementary results. For example, for portions of the 3D model that are not updated in a timely manner, real-world images can be used to determine sample images. For portions of the real-world image that are obscured by weather or external conditions, samples can be determined based on the 3D model to ensure the accuracy of the sample images.

[0107] Furthermore, in scenarios where sample images are directly derived from a 3D model, to ensure the accuracy and clarity of the sample images, the accuracy of the 3D model can be adjusted based on the size of the target object being photographed. For example, for scenes with smaller target objects, the accuracy of the reconstructed 3D model can be higher, while for scenes with larger target objects, the accuracy can be lower.

[0108] In some embodiments, the real-world images captured by the mobile platform along the planned work path can also be used to update the 3D model of the work area. For example, if the real-world image captured at a target location differs significantly from the observed image on the 3D model corresponding to that location, an update of the 3D model using the real-world image can be triggered. This update can be global or local.

[0109] In some embodiments, the target location point is displayed on an interactive interface; based on the operation result corresponding to the target location point, the display effect of the target location point on the interactive interface is adjusted. Based on the actual operation result of the movable platform, the target point can be marked to facilitate the user to know the task execution effect of the movable platform at the target point and / or the fault analysis result of the operation target of the movable platform. In another optional embodiment, the display of the target location point on the interface can be displayed in the form of a list, or it can be marked at the corresponding virtual space location point of the three-dimensional model.

[0110] In some embodiments, after taking photos during inspection using a mobile platform and capturing a real-life image of the target object, the real-life image captured for the current inspection task can be matched with the real-life image captured for the historical inspection task to determine the similarity between the two. For example, for a certain target location point on the operation path, the similarity between the image captured by the mobile platform in a certain posture during the current inspection task and the image captured by the mobile platform in the same posture during the previous inspection task can be determined. If the similarity is too low, for example, below a preset threshold, it may be that the current inspection task of the mobile platform has failed, or it may be that the target object to be inspected is blocked or has a fault. At this point, a prompt message can be sent to the user so that the user can locate the problem.

[0111] In some embodiments, considering the scenario where some minor faults occur in the target object to be inspected, such faults cannot be discovered based solely on the overall similarity of the real-scene images collected by each inspection task. For example, taking the bridge inspection as an example, if a crack appears on the bridge, the similarity matching based on the real-scene images collected by the two inspection tasks may still be high, so the user will not be prompted, resulting in the user being unable to discover such faults. In order to avoid the above problem, for the scenario where the similarity between the real-scene image collected by the current inspection task and the real-scene image collected by the historical inspection task is greater than a preset threshold, the real-scene image can be further semantically recognized, the target object can be identified from each real-scene image, and then the target object can be compared to determine whether the target objects in each real-scene image are similar. If the similarity is low, it is considered that the target object may have a fault, and a prompt message is issued. Optionally, by further combining semantic recognition, a more accurate fault analysis of the target object can be performed. Optionally, the semantic content represented by each pixel in the current real-life image can be identified. Historically collected images can be used as a data set for semantic recognition. The semantic information in the real-life image, such as trees, towers, cracks, snow, etc., can be analyzed to automatically complete fault analysis.

[0112] In some embodiments, to facilitate the user's subsequent problem location, the target location point, operation parameters, inspection task rounds, and other information corresponding to the real-life image where the problem occurred (for example, the overall similarity between the real-life image and the historical real-life image is too low, or the similarity between the target object in the real-life image and the target object in the historical real-life image is too low) can be recorded to facilitate the user's subsequent problem location. For example, the operation path can be displayed on the interactive interface, and the target location point where the problem occurred can be marked to facilitate locating the location point in the operation path where the problem occurred.

[0113] In some embodiments, the results of multiple actual operations of the mobile platform are uploaded to the cloud, and the cloud or terminal device can prompt the user of the potential risks in each operation based on the existing operation results and / or based on the user-calibrated targets, based on image matching. Optionally, the operation can include the energy industry, surveying and mapping industry, public safety and other operation fields. For example, taking the power grid inspection as an example, it is prompted that some electrical components are damaged or missing. For example, taking the oil and gas inspection plan as an example, it is prompted that the oil and gas pipeline is broken. For example, taking the AEC inspection as an example, it is prompted that the construction project or project group is illegally parked or built. In some embodiments, the three-dimensional model can be generated based on the image of the operation area collected by the mobile platform. The movable platform can be controlled to move in the operation area, and the image of the operation area can be collected, and then the operation area can be three-dimensionally reconstructed based on the image to obtain the three-dimensional model.

[0114] For example, taking the mobile platform's operating task as inspecting target objects within the operating area, the cloud can remotely send instructions to the mobile platform within the coverage area through the base station to control the mobile platform to move to the corresponding operating area and collect images of the operating area, and then transmit the collected images back to the cloud. The cloud can use the image to reconstruct a three-dimensional model of the operating area, or the cloud can also send the image to the terminal, which has modeling software and operation planning software installed in it. The modeling software completes the modeling of the operating area, and the operation planning software realizes the planning of the operation path.

[0115] In some embodiments, after multiple target locations are determined based on the movement of the virtual movable platform in the three-dimensional model, the multiple target locations can be connected to obtain a working path for the movable platform in the working area. When connecting the multiple target locations, the multiple target locations can be connected sequentially in the order in which the target locations were determined to obtain the working path.

[0116] In some embodiments, the order in which target points are confirmed may not be the optimal order for operations. Specifically, when connecting multiple target points to generate a work path, the order in which the target points are sorted may differ from the order in which they were confirmed. In this case, the target points may be sorted based on the optimal order for operations, and then connected based on the order to generate the work path.

[0117] In some embodiments, when determining the sequence of target locations, the distance or duration of movement required for the movable platform to move from the starting target location along the trajectory line to the ending target location can be used. That is, when determining the sequence of multiple target locations to obtain an operation path, the movable platform should strive to ensure that the distance or duration of movement along the operation path is as short as possible to save energy or improve operation efficiency.

[0118] In some embodiments, multiple work tasks of the virtual movable platform at the target location point can be determined based on the triggered work task confirmation operation, wherein the order in which the movable platform executes the multiple work tasks can be the same as the confirmation order of the multiple work tasks.

[0119] In some embodiments, considering that the confirmation order of multiple work tasks is not the optimal task execution order, after determining the multiple work tasks, the target order of the movable platform when executing the multiple work tasks at the target position can be determined based on the relevant information of the multiple work tasks (for example, the content of the tasks, the posture of the movable platform when executing the tasks, etc.), so that the movable platform can execute the multiple work tasks in accordance with the target order at the target position, wherein the target work order is different from the determination order of the multiple work tasks.

[0120] For example, for a specific target location, the mobile platform needs to perform multiple operations at that target location in multiple different postures. Considering that the order of these multiple tasks is not the optimal task execution order, the order of the tasks can be optimized based on the working postures of each task to obtain the optimal task execution order. For example, the overall posture change of the mobile platform when performing these multiple tasks can be minimized or the posture change can be continuously and progressively achieved, thereby improving the operating efficiency of the mobile platform.

[0121] In some embodiments, a method for planning an operation of a mobile platform is provided. The method comprises: obtaining a three-dimensional model of the mobile platform's operation area; controlling a motion trajectory of a virtual mobile platform within the three-dimensional model based on detected motion control operations; and determining multiple target locations along the motion trajectory based on detected position confirmation operations, wherein the multiple target locations are used to generate an operation path for the mobile platform within the operation area. Optionally, the three-dimensional model is generated based on images captured by controlling the mobile platform within the operation area. Optionally, the method further comprises: determining an observation angle of a virtual observation device mounted on the virtual mobile platform based on the position of the virtual mobile platform during its movement, and projecting the three-dimensional model onto the observation angle, thereby displaying an observation image obtained in real time on an interactive interface. Optionally, the interactive interface displays a display of the three-dimensional model observed by the virtual mobile platform and / or a display showing the relative positional relationship between the virtual mobile platform and the three-dimensional model. Optionally, the method further comprises determining and storing operation parameters corresponding to the target locations based on detected operation parameter confirmation operations, wherein the operation parameters are used to instruct the mobile platform to reach the target locations and perform the operation. Optionally, the movable platform is equipped with a camera payload, and the operation parameters include orientation parameters of the camera payload in space, and the orientation parameters are used to instruct the camera payload to perform operations according to the orientation parameters when the movable platform reaches the target position point. Optionally, the method further includes: obtaining a real-scene image captured when the movable platform operates along the operation path in the operation area; intercepting an image area selected by the user from the real-scene image as a sample image containing the target object, wherein the sample image is used to instruct the movable platform to capture a real-scene image containing the target object during subsequent operations. Optionally, the method further includes determining the deviation between the real-scene image corresponding to the target position point and the sample image, and adjusting the target position point and / or the operation parameters corresponding to the target position point based on the operation results.

[0122] In addition, an embodiment of the present application provides a method for planning an operation of a movable platform, the method comprising:

[0123] Obtaining a three-dimensional model of the operating area of ​​the movable platform;

[0124] Controlling the virtual observation device to move in the virtual space where the three-dimensional model is located according to the motion control operation, and displaying the observation image of the three-dimensional model by the virtual observation device in real time on the user interaction interface during the movement;

[0125] In response to a confirmation operation by the user, a target geographic position corresponding to the position of the virtual observation device in the virtual space is determined, where the target geographic position is used to instruct the movable platform to operate in the working area.

[0126] The specific details of implementing the above-mentioned operation planning method can be referred to the description in the above-mentioned embodiment and will not be repeated here.

[0127] In addition, an embodiment of the present application provides a method for planning an operation of a movable platform, the method comprising:

[0128] Acquire a work path planned in a three-dimensional model of a work area, and work parameters of path points on the work path, wherein the work path is generated based on a motion pose of a virtual movable platform in the three-dimensional model and an observation image of the three-dimensional model by the virtual movable platform;

[0129] Controlling the movable platform to move along the operation path within the operation area, and performing the operation based on the operation parameters of the path points to obtain the operation content;

[0130] Obtaining the job results obtained by analyzing the job content;

[0131] Adjusting the operation path and operation parameters of the path points on the operation path based on the operation results;

[0132] The movable platform is controlled to operate in the operating area based on the adjusted operating path and the adjusted operating parameters.

[0133] The specific details of implementing the above-mentioned operation planning method can be referred to the description in the above-mentioned embodiment and will not be repeated here.

[0134] In addition, an embodiment of the present application further provides a job planning system, including a cloud, a client, and a mobile platform;

[0135] The movable platform is used to collect images of the working area and send them to the cloud;

[0136] The cloud is used to generate a three-dimensional model of the operation area based on the image and send it to the client;

[0137] The client is used to display the three-dimensional model on an interactive interface, and control the virtual movable platform to move in the three-dimensional model according to the motion control operation input by the user, and display the observation image of the three-dimensional model by the virtual movable platform during the movement in real time on the user interactive interface; in response to the user's confirmation operation, determine the target geographic posture corresponding to the current posture of the virtual movable platform, and the target geographic posture is used to instruct the movable platform to operate in the working area.

[0138] Among them, the specific details of the above-mentioned operation planning system implementing operation planning can be referred to the description in the above-mentioned embodiment, and will not be repeated here.

[0139] It is not difficult to understand that the solutions described in the above embodiments can be combined when there is no conflict, and they are not listed one by one in the embodiments of this application.

[0140] In addition, an embodiment of the present application further provides an operation planning device for a mobile platform, as shown in FIG9 . The device includes a processor 91, a memory 92, and a computer program stored in the memory 92 and executable by the processor 91. When the processor 91 executes the computer program, the following steps can be implemented:

[0141] Obtaining a three-dimensional model of the operating area of ​​the movable platform;

[0142] controlling a motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation;

[0143] A plurality of target position points on the motion trajectory are determined according to the detected position confirmation operation, and the plurality of target position points are used to generate an operation path of the movable platform in the operation area.

[0144] In some embodiments, the motion control operation includes one or more of the following: a lateral control operation, a longitudinal control operation, an altitude control operation, and a yaw control operation of the virtual movable platform.

[0145] In some embodiments, the motion control operation is triggered by a joystick, and the motion control amount of the motion control operation is determined based on the detected offset of the joystick.

[0146] In some embodiments, the processor is further configured to: determine and store the operation parameters corresponding to the target location point based on the detected operation parameter confirmation operation, wherein the operation parameters are used to instruct the movable platform to reach the target location point to perform the operation.

[0147] In some embodiments, the movable platform is equipped with a camera payload, and the operation parameters include orientation parameters of the camera payload in space, and the orientation parameters are used to indicate that the camera payload performs the operation according to the orientation parameters when the movable platform reaches the target position point.

[0148] In some embodiments, the processor is further configured to: determine an operation result corresponding to the target location point, and adjust the target location point and / or the operation parameter corresponding to the target location point based on the operation result.

[0149] In some embodiments, the load on the movable platform includes a photographing device, and the operation result includes a deviation between an image captured by the photographing device and a sample image.

[0150] In some embodiments, the processor is also used to: determine the observation perspective of the virtual observation device carried on the virtual movable platform based on the posture of the virtual movable platform during movement, and display the observation image obtained by projecting the three-dimensional model to the observation perspective in real time on the interactive interface.

[0151] In some embodiments, the processor is configured to control a motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation, specifically to:

[0152] controlling the virtual movable platform to leave the first position in the three-dimensional model and move to another area in the three-dimensional model according to the detected motion control operation;

[0153] The determining of a plurality of target position points on the motion trajectory according to the detected position confirmation operation, wherein the plurality of target position points are used to generate an operation path of the movable platform in the operation area, comprises:

[0154] The current position of the virtual movable platform in the three-dimensional model is determined as a second position according to the detected position confirmation operation, and the first position and the second position are used to generate the working path of the movable platform in the working area.

[0155] In some embodiments, the processor is further configured to perform any of the following operations in response to the editing operation:

[0156] Deleting the determined target location point;

[0157] Adjusting the position of the determined target location point;

[0158] Insert a new target position point between any two adjacent target position points;

[0159] Modify the operation parameters corresponding to the target location point.

[0160] In some embodiments, the processor is further configured to control the virtual movable platform to move from a current position to any determined target position point in response to a movement operation.

[0161] In some embodiments, the processor is further configured to: display on an interactive interface an identifier representing the orientation of the virtual observation device carried by the virtual movable platform, and / or a scene in a three-dimensional model surrounding the target location point.

[0162] In some embodiments, the processor is further configured to: display on an interactive interface a picture of the three-dimensional model observed by the virtual movable platform, and / or a picture of the relative positional relationship between the virtual movable platform and the three-dimensional model.

[0163] In some embodiments, the processor is further configured to: display the three-dimensional model on an interactive interface, wherein the display perspectives of the three-dimensional model on the interactive interface include multiple perspectives;

[0164] The viewpoint positions of any two of the multiple viewpoints are different, and / or the directions from the viewpoints of any two viewpoints to the virtual movable platform are different.

[0165] In some embodiments, the processor is further configured to: change the position of the viewpoint of any of the perspectives following the movement of the virtual movable platform, while keeping the relative position of the viewpoint and the virtual movable platform fixed.

[0166] In some embodiments, the processor is further configured to: display safety prompt information on the interactive interface;

[0167] The security reminder information is generated based on one or more of the following information:

[0168] The relative position relationship between the position points on the motion trajectory and the envelope points of the three-dimensional model;

[0169] The change in the motion posture of the virtual movable platform on the motion trajectory.

[0170] In some embodiments, the processor is further used to: obtain observations of the three-dimensional model to obtain sample images containing the target object, wherein the sample images are used to instruct the movable platform to capture real-scene images containing the target object while operating along the operating path.

[0171] In some embodiments, when the processor is configured to obtain a sample image containing a target object by observing the three-dimensional model, the processor is specifically configured to:

[0172] determining an observation angle of a virtual observation device carried on the virtual movable platform based on the posture of the virtual movable platform during movement;

[0173] An observation image obtained by projecting the three-dimensional model onto the observation viewing angle is acquired, and the sample image is determined based on the observation image.

[0174] In some embodiments, determining the sample image based on the observed image includes any of the following methods: using the observed image as the sample image; or, using the image area where the center position of the observed image is located as the sample image; or, using the image area in the observed image selected by the user as the sample image.

[0175] In some embodiments, the processor is further configured to: obtain a real-scene image captured when the movable platform operates along the operation path in the operation area;

[0176] An image area selected by a user is captured from the real scene image as a sample image containing the target object, wherein the sample image is used to instruct the movable platform to capture a real scene image containing the target object during a subsequent operation.

[0177] In some embodiments, the three-dimensional model is generated based on controlling a movable platform to capture images within the working area.

[0178] In some embodiments, the plurality of target location points are used to generate an operation path of the movable platform in the operation area, including:

[0179] Determine a target position point sequence for connecting a plurality of the target position points to generate an operation path for the operation area, wherein the target position point sequence is different from a sequence of confirming the target position points.

[0180] In some embodiments, the processor is further configured to: determine a plurality of operating tasks for the virtual movable platform at the target location based on an operating task confirmation operation;

[0181] The movable platform is instructed to perform the plurality of operation tasks at the target location point according to a target operation sequence, where the target operation sequence is different from a determined sequence of the plurality of operation tasks.

[0182] In some embodiments, the processor is further configured to: display the target location point on an interactive interface;

[0183] Based on the operation result corresponding to the target location point, the display effect of the target location point on the interactive interface is adjusted.

[0184] In addition, an embodiment of the present application further provides an operation planning device for a mobile platform, the device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, the following steps can be implemented:

[0185] Obtaining a three-dimensional model of the operating area of ​​the movable platform;

[0186] Controlling the virtual observation device to move in the virtual space where the three-dimensional model is located according to the motion control operation, and displaying the observation image of the three-dimensional model by the virtual observation device in real time on the user interaction interface during the movement;

[0187] In response to a confirmation operation by the user, a target geographic position corresponding to the position of the virtual observation device in the virtual space is determined, where the target geographic position is used to instruct the movable platform to operate in the working area.

[0188] In addition, an embodiment of the present application further provides an operation planning device for a mobile platform, the device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, the following steps can be implemented:

[0189] Acquire a work path planned in a three-dimensional model of a work area, and work parameters of path points on the work path, wherein the work path is generated based on a motion pose of a virtual movable platform in the three-dimensional model and an observation image of the three-dimensional model by the virtual movable platform;

[0190] Controlling the movable platform to move along the operation path within the operation area, and performing the operation based on the operation parameters of the path points to obtain the operation content;

[0191] Obtaining the job results obtained by analyzing the job content;

[0192] Adjusting the operation path and operation parameters of the path points on the operation path based on the operation results;

[0193] The movable platform is controlled to operate in the operating area based on the adjusted operating path and the adjusted operating parameters.

[0194] Accordingly, an embodiment of the present application further provides a computer storage medium, in which a program is stored. When the program is executed by a processor, the method in any of the above embodiments is implemented.

[0195] The embodiments of the present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0196] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0197] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0198] The above is a detailed introduction to the methods and devices provided in the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for planning an operation of a movable platform, characterized in that: The method comprises: Obtaining a three-dimensional model of the operating area of ​​the movable platform; controlling the motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation; A plurality of target position points on the motion trajectory are determined according to the detected position confirmation operation, and the plurality of target position points are used to generate an operation path of the movable platform in the operation area.

2. The method according to claim 1, characterized in that The motion control operation includes one or more of the following: a lateral control operation, a longitudinal control operation, an altitude control operation, and a yaw control operation of the virtual movable platform.

3. The method according to claim 1 or 2, characterized in that: The motion control operation is triggered by a joystick, and a motion control amount of the motion control operation is determined based on a detected offset of the joystick.

4. The method according to claim 1, characterized in that: The method further comprises: The operation parameters corresponding to the target location point are determined and stored according to the detected operation parameter confirmation operation, and the operation parameters are used to instruct the movable platform to arrive at the target location point to perform the operation.

5. The method according to claim 4, characterized in that The movable platform is equipped with a camera payload, and the operation parameters include orientation parameters of the camera payload in space, and the orientation parameters are used to indicate that the camera payload performs the operation according to the orientation parameters when the movable platform reaches the target position point.

6. The method according to claim 4, characterized in that The method further comprises: An operation result corresponding to the target location point is determined, and the target location point and / or the operation parameter corresponding to the target location point is adjusted based on the operation result.

7. The method according to claim 6, characterized in that The load on the movable platform includes a photographing device, and the operation result includes a deviation between an image captured by the photographing device and a sample image.

8. The method according to claim 1, characterized in that The method further comprises: Based on the posture of the virtual movable platform during movement, the observation angle of the virtual observation device carried on the virtual movable platform is determined, and the observation image obtained by projecting the three-dimensional model to the observation angle is displayed in real time on the interactive interface.

9. The method according to claim 1, characterized in that: The controlling the motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation comprises: According to the detected motion control operation, controlling the virtual movable platform to leave the first position in the three-dimensional model and move to another area in the three-dimensional model; The determining of a plurality of target position points on the motion trajectory according to the detected position confirmation operation, wherein the plurality of target position points are used to generate an operation path of the movable platform in the operation area, comprises: According to the detected position confirmation operation, the current position of the virtual movable platform in the three-dimensional model is determined as the second position, and the first position and the second position are used to generate the movable platform in the operation The working path of the area.

10. The method according to claim 1, characterized in that In response to an edit operation, do any of the following: Deleting the determined target location point; Adjusting the position of the determined target location point; Insert a new target position point between any two adjacent target position points; Modify the operation parameters corresponding to the target location point.

11. The method according to claim 1, characterized in that: In response to the moving operation, the virtual movable platform is controlled to move from a current position to any determined target position point.

12. The method according to claim 1, characterized in that The method further comprises: An identification indicating the orientation of the virtual observation device carried by the virtual movable platform and / or a scene in the three-dimensional model around the target location point are displayed on the interactive interface.

13. The method according to claim 1, characterized in that The method further comprises: A picture showing the three-dimensional model observed by the virtual movable platform and / or a picture showing the relative position relationship between the virtual movable platform and the three-dimensional model is displayed on the interactive interface.

14. The method according to claim 1, characterized in that The method further comprises: Displaying the three-dimensional model on an interactive interface, wherein the display perspectives of the three-dimensional model on the interactive interface include multiple perspectives; Wherein, viewpoint positions of any two of the multiple viewpoints are different, and / or directions from the viewpoints of any two viewpoints to the virtual movable platform are different.

15. The method according to claim 14, characterized in that The method further comprises: The position of the viewpoint of any of the viewing angles changes following the movement of the virtual movable platform, and the relative position of the viewpoint and the virtual movable platform remains fixed.

16. The method according to claim 1, characterized in that The method further comprises: Display safety tips on the interactive interface; The security prompt information is generated based on one or more of the following information: The relative position relationship between the position points on the motion trajectory and the envelope points of the three-dimensional model; The change amount of the motion posture of the virtual movable platform on the motion trajectory.

17. The method according to claim 1, characterized in that The method further comprises: A sample image containing the target object is obtained by observing the three-dimensional model, wherein the sample image is used to instruct the movable platform to capture a real scene image containing the target object during operation along the operation path.

18. The method according to claim 17, characterized in that Acquiring observations of the three-dimensional model to obtain a sample image containing a target object, including: Determining the observation angle of a virtual observation device carried on the virtual movable platform based on the position and posture of the virtual movable platform during the movement; An observation image obtained by projecting the three-dimensional model to the observation viewing angle is acquired, and the sample image is determined based on the observation image.

19. The method according to claim 18, characterized in that The determining of the sample image based on the observed image includes any of the following methods: Use the observed image as a sample image; or, The image area where the center of the observed image is located is used as a sample image; or, The image area in the observed image selected by the user is used as a sample image.

20. The method according to claim 1, characterized in that The method further comprises: Acquire a real scene image collected when the movable platform operates along the operating path in the operating area; An image area selected by a user is captured from the real scene image as a sample image containing the target object, wherein the sample image is used to instruct the movable platform to capture a real scene image containing the target object during a subsequent operation.

21. The method according to claim 1, characterized in that The three-dimensional model is generated based on controlling the movable platform to collect images within the working area.

22. The method according to claim 1, characterized in that The plurality of target position points are used to generate an operation path of the movable platform in the operation area, including: Determine a sequence of target position points for connecting a plurality of the target position points to generate an operation path for the operation area, wherein the sequence of the target position points is different from a sequence of confirming the target position points.

23. The method according to claim 1, characterized in that The method further comprises: Determining a plurality of operation tasks for the virtual movable platform at the target location based on the operation task confirmation operation; The movable platform is instructed to perform the plurality of operation tasks at the target location point according to a target operation sequence, wherein the target operation sequence is different from a determined sequence of the plurality of operation tasks.

24. The method according to claim 1, characterized in that The method further comprises: Displaying the target location point on the interactive interface; Based on the operation result corresponding to the target location point, the display effect of the target location point on the interactive interface is adjusted.

25. An operation planning device for a movable platform, characterized in that: The device includes a processor, a memory, and a computer program stored in the memory and executable by the processor. When the processor executes the computer program, the following steps can be implemented: Obtaining a three-dimensional model of the operating area of ​​the movable platform; controlling the motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation; A plurality of target position points on the motion trajectory are determined according to the detected position confirmation operation, and the plurality of target position points are used to generate an operation path of the movable platform in the operation area.

26. The device according to claim 25, characterized in that The motion control operation includes one or more of the following: a lateral control operation, a longitudinal control operation, an altitude control operation, and a yaw control operation of the virtual movable platform.

27. The device according to claim 25 or 26, characterized in that The motion control operation is triggered by a joystick, and a motion control amount of the motion control operation is determined based on a detected offset of the joystick.

28. The device according to claim 25, characterized in that The processor is further configured to: The operation parameters corresponding to the target location point are determined and stored according to the detected operation parameter confirmation operation, and the operation parameters are used to instruct the movable platform to arrive at the target location point to perform the operation.

29. The device according to claim 28, characterized in that The movable platform is equipped with a camera payload, and the operation parameters include orientation parameters of the camera payload in space, and the orientation parameters are used to indicate that the camera payload performs the operation according to the orientation parameters when the movable platform reaches the target position point.

30. The device according to claim 28, characterized in that The processor is further configured to: An operation result corresponding to the target location point is determined, and the target location point and / or the operation parameter corresponding to the target location point is adjusted based on the operation result.

31. The device according to claim 30, characterized in that The load on the movable platform includes a photographing device, and the operation result includes a deviation between an image captured by the photographing device and a sample image.

32. The device according to claim 25, characterized in that The processor is further configured to: Based on the posture of the virtual movable platform during movement, the observation angle of the virtual observation device carried on the virtual movable platform is determined, and the observation image obtained by projecting the three-dimensional model to the observation angle is displayed in real time on the interactive interface.

33. The device according to claim 25, characterized in that The processor is used to control the motion trajectory of the virtual movable platform in the three-dimensional model according to the detected motion control operation, specifically to: According to the detected motion control operation, controlling the virtual movable platform to leave the first position in the three-dimensional model and move to another area in the three-dimensional model; The determining of a plurality of target position points on the motion trajectory according to the detected position confirmation operation, wherein the plurality of target position points are used to generate an operation path of the movable platform in the operation area, comprises: The current position of the virtual movable platform in the three-dimensional model is determined as a second position according to the detected position confirmation operation, and the first position and the second position are used to generate the working path of the movable platform in the working area.

34. The device according to claim 25, characterized in that The processor is further configured to, in response to the editing operation, perform any of the following operations: Deleting the determined target location point; Adjusting the position of the determined target location point; Insert a new target position point between any two adjacent target position points; Modify the operation parameters corresponding to the target location point.

35. The device according to claim 25, characterized in that The processor is further configured to control the virtual movable platform to move from a current position to any determined target position point in response to a moving operation.

36. The device according to claim 25, characterized in that The processor is further configured to: An identification indicating the orientation of the virtual observation device carried by the virtual movable platform and / or a scene in the three-dimensional model around the target location point are displayed on the interactive interface.

37. The device according to claim 25, characterized in that The processor is further configured to: A picture showing the three-dimensional model observed by the virtual movable platform and / or a picture showing the relative position relationship between the virtual movable platform and the three-dimensional model is displayed on the interactive interface.

38. The device according to claim 25, characterized in that The processor is further configured to: Displaying the three-dimensional model on an interactive interface, wherein the display perspectives of the three-dimensional model on the interactive interface include multiple perspectives; Wherein, viewpoint positions of any two of the multiple viewpoints are different, and / or directions from the viewpoints of any two viewpoints to the virtual movable platform are different.

39. The device according to claim 38, characterized in that The processor is further configured to: The position of the viewpoint of any of the viewing angles changes following the movement of the virtual movable platform, and the relative position of the viewpoint and the virtual movable platform remains fixed.

40. The device according to claim 25, characterized in that The processor is further configured to: Display safety tips on the interactive interface; The security prompt information is generated based on one or more of the following information: The relative position relationship between the position points on the motion trajectory and the envelope points of the three-dimensional model; The change amount of the motion posture of the virtual movable platform on the motion trajectory.

41. The device according to claim 25, characterized in that The processor is further configured to: A sample image containing the target object is obtained by observing the three-dimensional model, wherein the sample image is used to instruct the movable platform to capture a real scene image containing the target object during operation along the operation path.

42. The device according to claim 41, characterized in that When the processor is used to obtain a sample image containing a target object by observing the three-dimensional model, it is specifically used to: Determining the observation angle of a virtual observation device carried on the virtual movable platform based on the position and posture of the virtual movable platform during the movement; An observation image obtained by projecting the three-dimensional model to the observation viewing angle is acquired, and the sample image is determined based on the observation image.

43. The device according to claim 42, characterized in that The determining of the sample image based on the observed image includes any of the following methods: Use the observed image as a sample image; or, The image area where the center of the observed image is located is used as a sample image; or, The image area in the observed image selected by the user is used as a sample image.

44. The device according to claim 25, characterized in that The processor is further configured to: Acquire a real-scene image collected when the movable platform operates along the operating path in the operating area; An image area selected by a user is captured from the real scene image as a sample image containing the target object, wherein the sample image is used to instruct the movable platform to capture a real scene image containing the target object during a subsequent operation.

45. The device according to claim 25, characterized in that The three-dimensional model is generated based on controlling the movable platform to collect images within the working area.

46. ​​The device according to claim 25, characterized in that The plurality of target position points are used to generate an operation path of the movable platform in the operation area, including: Determine a sequence of target position points for connecting a plurality of the target position points to generate an operation path for the operation area, wherein the sequence of the target position points is different from a sequence of confirming the target position points.

47. The device according to claim 25, characterized in that The processor is further configured to: Determining a plurality of operation tasks for the virtual movable platform at the target location based on the operation task confirmation operation; The movable platform is instructed to perform the plurality of operation tasks at the target location point according to a target operation sequence, wherein the target operation sequence is different from a determined sequence of the plurality of operation tasks.

48. The device according to claim 25, characterized in that The processor is further configured to: Displaying the target location point on the interactive interface; Based on the operation result corresponding to the target location point, the display effect of the target location point on the interactive interface is adjusted.

49. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 24 is implemented.

Images