Control methods, devices and control systems for mobile platforms

Abstract

A mobile platform control method and apparatus. The method includes: acquiring a first target object selection operation and a second target object selection operation input by a user on an interactive interface; determining a first target orientation and a second target orientation of the mobile platform when performing operations on the first target object and the second target object, respectively, based on the orientation of the three-dimensional model displayed on the interactive interface when acquiring the first target object selection operation and the second target object selection operation; determining a first target position and a second target position of the mobile platform when performing operations on the first target object and the second target object, respectively, based on the first target object selection operation and the second target object selection operation input by the user; and displaying a line connecting the first target position and the second target position on the interactive interface, wherein the line forms the operation path of the mobile platform.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and more specifically, to a control method, device and control system for a mobile platform. Background Technology

[0002] Many fields require precise control of mobile platforms such as drones, unmanned vehicles, and intelligent robots to perform specific tasks. For example, in scenarios where drones are used for inspections, it is typically necessary to control the drone to photograph objects such as power insulators and dams from specific locations and angles. Similarly, in scenarios where robotic arms are used for fruit harvesting, it is necessary to control the robotic arm to move to specific positions and angles to complete the harvest. Therefore, it is necessary to provide a solution that allows users to perform precise control of mobile platforms. Summary of the Invention

[0003] In view of this, this application provides a control method, apparatus, mobile platform, and control system for a mobile platform.

[0004] According to a first aspect of this application, a control method for a mobile platform is provided, the method comprising:

[0005] The system obtains a first target object selection operation and a second target object selection operation input by the user in the interactive interface, wherein the interactive interface displays a three-dimensional model of the work area, and the first target object selection operation and the second target object selection operation are used to determine the position of the first target object and the position of the second target object in the work area, respectively.

[0006] Based on the orientation of the 3D model displayed on the interactive interface when the first target object selection operation and the second target object selection are obtained, the first target orientation and the second target orientation of the mobile platform when the mobile platform performs operations on the first target object and the second target object are determined respectively.

[0007] The first target position and the second target position when the mobile platform performs operations on the first target object and the second target object are determined according to the first target object selection operation and the second target object selection operation input by the user, respectively.

[0008] The interactive interface displays a line connecting the first target location and the second target location, wherein the line forms the working path of the movable platform.

[0009] According to a second aspect of this application, a control device for a mobile platform is provided, the device comprising a processor, a memory, and a computer program executable by the processor stored in the memory, wherein the processor executes the computer program to perform the following steps:

[0010] The system obtains a first target object selection operation and a second target object selection operation input by the user in the interactive interface, wherein the interactive interface displays a three-dimensional model of the work area, and the first target object selection operation and the second target object selection operation are used to determine the position of the first target object and the position of the second target object in the work area, respectively.

[0011] Based on the orientation of the 3D model displayed on the interactive interface when the first target object selection operation and the second target object selection are obtained, the first target orientation and the second target orientation of the mobile platform when the mobile platform performs operations on the first target object and the second target object are determined respectively.

[0012] The first target position and the second target position when the mobile platform performs operations on the first target object and the second target object are determined according to the first target object selection operation and the second target object selection operation input by the user, respectively.

[0013] The interactive interface displays a line connecting the first target location and the second target location, wherein the line forms the working path of the movable platform.

[0014] By applying the solution provided in this application, users can adjust the 3D model of the work area displayed on the interactive interface to a suitable orientation for the target object in the work area, thereby determining the target orientation when performing work on the target object. By browsing and adjusting the 3D model on the interactive interface, users can determine the appropriate orientation when performing work on the target object more intuitively, without having to repeatedly adjust the orientation by constantly inputting angles, which is more convenient and faster. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a control method for a mobile platform according to an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of images obtained by a virtual camera from different orientations when capturing images of a three-dimensional object, according to an embodiment of this application.

[0018] Figure 3 This is a schematic diagram illustrating the determination of target orientation according to one embodiment of this application.

[0019] Figure 4 This is a schematic diagram illustrating the determination of a target location according to an embodiment of this application.

[0020] Figure 5 This is a schematic diagram illustrating the determination of the location of a target object and the target area corresponding to the target object according to an embodiment of this application.

[0021] Figure 6 This is a schematic diagram of the adjustment operation according to one embodiment of this application.

[0022] Figure 7 This is an example diagram of the adjustment path for the first adjustment operation in one embodiment of this application.

[0023] Figure 8 This is a schematic diagram of the adjustment path for the second adjustment operation in one embodiment of this application.

[0024] Figure 9 This is a schematic diagram of the interactive interface of one embodiment of this application.

[0025] Figure 10 This is a schematic diagram illustrating an application scenario of one embodiment of this application.

[0026] Figure 11 This is a schematic diagram of a mobile platform control device according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Mobile platforms such as drones, unmanned vehicles, and intelligent robots are widely used in many fields. For example, drones are used for inspections, robotic arms for fruit harvesting, and drones and unmanned vehicles for spraying pesticides and watering. When performing these tasks, precise control of the mobile platform is usually required. For instance, controlling the platform to move to a specific position and angle before operating on the target object is crucial for achieving optimal results. Currently, controlling mobile platforms involves either manual operation—for example, manually adjusting the platform's position and orientation based on user observations or photos returned by the platform—but this method requires manual intervention for each operation, making it labor-intensive. Some technologies allow users to determine and store the position and orientation of a mobile platform during operation through an interactive interface displaying a 3D model of the work area. The platform is then controlled according to this stored position and orientation. This method involves the user clicking near the work object in the 3D model to determine the position. For example, clicking a ground point and dragging it a certain distance vertically. The user then manually inputs the platform's orientation angle and continuously adjusts it based on the preview window in the interactive interface to determine the final orientation. However, this method cannot accurately control the working distance of the mobile platform and requires repeated adjustments to the input angle, making it particularly cumbersome.

[0029] Based on this, this application provides a control method for a mobile platform. Users can move and rotate the 3D model on an interactive interface displaying the work area. The orientation of the 3D model can be adjusted according to a suitable viewing angle for observing the target object in the work area. The position of the target object is determined based on the user's target object selection operation input on the interactive interface. The target orientation for working on the target object is determined based on the orientation of the 3D model when the user inputs the target object selection operation. The target position of the mobile platform for working on the target object is determined based on the target object's position, target orientation, and work distance. Users can intuitively determine the suitable orientation for the mobile platform's work using the 3D model on the interactive interface, eliminating the need for multiple angle inputs for adjustment, which is convenient and quick. Furthermore, by adjusting the work position with the target object as the center, the work distance of the mobile platform can be accurately controlled.

[0030] Specifically, the method is as follows: Figure 1 As shown, it includes the following steps:

[0031] S102. Obtain the target object selection operation input by the user in the interactive interface, wherein the interactive interface displays a three-dimensional model of the work area, and the target object selection operation is used to determine the position of the target object in the work area.

[0032] S104. Determine the target orientation when performing operations on the target object based on the orientation of the three-dimensional model displayed on the interactive interface when the target object selection operation is performed.

[0033] S106. Determine the target position of the mobile platform when it performs operations on the target object based on the position of the target object, the target orientation, and the working distance of the mobile platform, so that the mobile platform moves to the target position and performs operations on the target object according to the target orientation.

[0034] The mobile platform control method of this application can be executed by the mobile platform itself. For example, in some embodiments, the mobile platform can provide a human-machine interface (HMI) where the user can operate to determine the target position and orientation of the mobile platform when performing operations on the target object, and then control the mobile platform to perform the operations based on the determined target position and orientation. In some embodiments, the mobile platform control method of this application can also be executed by a control terminal, which can be a laptop, remote control, mobile phone, tablet, or other terminal device. The control terminal can provide an HMI to acquire user interaction operations and control the mobile platform based on the user's interaction operations.

[0035] The mobile platform of this application is a mobile platform used for performing operations on a target object. The mobile platform includes a power component for driving the mobile platform to move. The mobile platform can be a drone, unmanned vehicle, intelligent robot or other equipment.

[0036] The target object of this application refers to the object on which the mobile platform will perform operations. This target object can be power equipment, dams, bridges, etc., that need to be inspected, crops that need to be sprayed with pesticides or irrigated, or fruits that need to be harvested. There can be one or more target objects, and the work area refers to the area that includes these one or more target objects. The 3D model of the work area can be a 3D model obtained through photogrammetric 3D reconstruction, a 3D model obtained through LiDAR scanning, or a CAD model from the design process.

[0037] The device executing this control method can load a 3D model of the work area into a 3D engine and display the 3D model to the user through a human-computer interaction interface. When determining the target orientation and position of the target object for the mobile platform's operation, the user can move and rotate the 3D model in the interface to adjust its orientation to a suitable position for observing the target object, such as ensuring the target object is not obstructed by other objects and can be clearly seen. After adjusting the 3D model's orientation in the interface, the user can input a target object selection operation. The device executing this control method can acquire the user's target object selection operation, determine the target object's position based on this operation, and then determine the target orientation for operation based on the orientation of the 3D model displayed in the interface at the time of acquisition. The target orientation is consistent with the orientation of the 3D model observed by the user from the interface; therefore, the user's perspective of observing the 3D model from the interface is the perspective from which the mobile platform observes the target object during operation. Once the target orientation is determined, the target position of the mobile platform when performing operations on the target object can be determined based on the location of the target object, the target orientation, and the working distance of the mobile platform. This allows the mobile platform to be controlled to move to the target position and perform operations on the target object according to the target orientation.

[0038] It should be noted that the target position and target orientation in this application can be either the position and orientation of the movable platform when performing operations on the target object, or the position and orientation of the parts on the movable platform that perform operations on the target object. For example, assuming a scenario where a drone is used to inspect electronic equipment, since the positional difference between the drone's center and the camera's center is small, the drone's center can be directly adjusted to the target position. Of course, for more precise control of the drone, the target position can also be compensated based on the positional relationship between the camera's center and the drone's center to obtain a compensated position. The drone can then be moved to the compensated position so that the drone's camera is positioned at the target position for operation. The target orientation can be the orientation of the movable platform, or it can be the orientation of the parts performing operations on the movable platform, such as the orientation of the camera on the drone or the orientation of the robotic arm on an unmanned vehicle.

[0039] Using the method provided in this application, users can adjust the orientation of the 3D model based on the visual effect of the target object displayed on the interactive interface, thereby determining a more suitable orientation for observing the target object and determining the target orientation when performing operations on the target object. In this way, users can intuitively determine the appropriate orientation when performing operations on the target object without having to repeatedly adjust the orientation by constantly inputting angles, which is more convenient and faster. At the same time, the target position when performing operations on the target object can be determined by the determined position of the target object, the target orientation, and the working distance, and the working distance can be precisely controlled according to user needs.

[0040] User input for selecting a target object can be any operation used to determine the location of a target object based on a 3D model. For example, it could be clicking on a target object within the 3D model displayed on the interactive interface, or using a bounding box to select a target object within the 3D model. Based on the user's selection, the target object's location can be determined. For instance, a user can click on the center area of ​​a target object on the interactive interface, and that clicked point becomes the target object's location. If the user uses a bounding box to select the target object, the center of the selection box becomes the target object's location. Alternatively, a user can click on multiple points on the target object to define its location. For example, if the target object is a person, the user can click on the head, body, and feet, then connect these three points with a line, and the center of this line becomes the target object's location. Since the 3D model carries geographic location information, the 3D coordinates corresponding to the target object's location can be determined based on the user's interactive actions within the 3D model.

[0041] Typically, a 3D engine corresponds to a virtual camera. The view of the work area presented in the interactive interface is equivalent to the virtual camera taking pictures of the work area from different perspectives. For example, as the user moves and rotates the 3D model in the interactive interface, the position and orientation of the virtual camera also change continuously. Figure 2 As shown, assuming the 3D model is a car model, when the user adjusts the orientation of the 3D model, i.e., the image presents the car at different angles (the car image within the rectangle in the figure), it can be assumed that the image is captured by a virtual camera (the black dot in the figure) at different orientations. Specifically, when the user manipulates the 3D model through the interactive interface and adjusts its displayed posture, the position and orientation of the corresponding virtual camera can be determined by the 3D engine. In some embodiments, the orientation of the 3D model displayed on the interactive interface corresponds one-to-one with the orientation of the virtual camera. Therefore, the orientation of the virtual camera corresponding to the orientation of the 3D model currently displayed on the interactive interface can be used as the target orientation when performing operations on the target object.

[0042] In some embodiments, such as Figure 3 As shown, the target orientation can be a line connecting the position of the target object and the position of the virtual camera, pointing towards the position of the target object. The target orientation can be determined based on the position of the target object and the position of the virtual camera. For example, when the user stops moving or rotating the 3D model in the interactive interface, the orientation of the 3D model displayed in the interactive interface is determined at this moment. The 3D engine can automatically determine the current position of the virtual camera (which can be the 3D coordinates corresponding to the center of the virtual camera), and then determine the position of the target object (which can be the 3D coordinates corresponding to the center of the target object) based on the target object selection operation entered by the user in the interactive interface. Then, a line is obtained by connecting the center of the virtual camera and the center of the target object. When performing operations on the target object, the target orientation of the movable platform can be pointed towards the target object along this line.

[0043] In some embodiments, such as Figure 4 As shown, the target position when the mobile platform performs operations on the target object can be located on the line connecting the target object's position and the virtual camera's position, and the distance from the target object is the operation distance. After determining the position of the virtual camera (which can be the 3D coordinates corresponding to the center of the virtual camera) and the position of the target object (which can be the 3D coordinates corresponding to the center of the target object), a line is drawn connecting the center of the virtual camera and the center of the target object. Then, using the target object as a reference, the platform moves along this line for the operation distance to reach the target position. In this way, the distance at which the mobile platform performs operations on the target object can be determined to be the user's expected operation distance.

[0044] After the user determines the orientation of the 3D model displayed on the interactive interface by translating and rotating the model, and inputs a target object selection operation, the device executing this control method can determine the target position and orientation of the movable platform when working on the target object based on the target object selection operation and the orientation of the 3D model displayed on the interactive interface. Of course, the user can further fine-tune the target position and orientation to achieve a better result. In some embodiments, the device executing this control method can acquire the adjustment operation input through the interactive interface, then determine the adjusted target position based on the target object's position and the user's input adjustment operation, and adjust the target orientation based on the adjusted target position, wherein the adjusted target orientation points from the adjusted target position to the target object's position.

[0045] The working distance of the mobile platform can be adjusted according to the user's actual needs. It can be input by the user or automatically determined by the device executing the control method. For example, in the scenario of spraying pesticides on crops, in order to ensure the effect of pesticide spraying, there are requirements for the distance between the nozzle and the crops. In this scenario, the user can input the working distance through the interactive interface. In the scenario of collecting images of target objects, the working distance can be determined according to the user's spatial resolution of the collected images. The spatial resolution refers to the physical size corresponding to the pixels in the image. For example, if the user needs to observe the details of a local area of ​​the target object, the mobile platform should be as close to the target object as possible, and the working distance can be set to be smaller. If the user needs to observe the whole picture of the target object, the working distance should ensure that the mobile platform can capture the entire target object. In some embodiments, the working distance of the mobile platform can be determined according to the size of the object being worked on, or according to the spatial resolution, or by combining the size of the object being worked on and the spatial resolution. The spatial resolution can be input by the user through the interactive interface or preset. For example, the user can input the spatial resolution according to the requirements of the spatial resolution of the collected target object images, and the device executing the control method can automatically calculate the working distance according to the following formula (1):

[0046] d= gsd*w / 35*f formula (1)

[0047] Where: d is the working distance, w is the pixel width of the image sensor, f is the 35mm equivalent focal length of the lens, and gsd is the spatial resolution, representing the actual distance of the object in three-dimensional space corresponding to the size of the pixel on the image.

[0048] Of course, the working distance can also be determined based on the size of the target object. For example, in some scenarios, it is necessary to capture the entire target object in one image. Therefore, the working distance can be determined based on the size of the target object. The working distance can be determined according to formula (2):

[0049] d = L / 35*f Formula (2)

[0050] Where L is the size of the target object and f is the 35mm equivalent focal length of the lens.

[0051] Of course, in some embodiments, the working distance can also be determined by combining spatial resolution and the size of the target object. For example, the user can input the range of spatial resolution, and then the working distance can be determined by combining the user-input spatial resolution range and the size of the target object so that the shooting can be carried out at the working distance, which can meet the user's input spatial resolution requirements and capture the complete target object.

[0052] In some embodiments, after determining the working distance based on the spatial resolution input by the user, if the complete target object cannot be captured at that working distance, the orientation of the camera on the movable platform can be adjusted to capture multiple images, and then these multiple images can be combined to obtain an image that includes the complete target object.

[0053] In some embodiments, if the camera device on the mobile platform is zoomable, the focal length can also be adjusted according to the desired working distance to meet the user's requirements for the working distance.

[0054] In some embodiments, when determining the size of a target object, a target region corresponding to the target object can be determined based on a user-input target object selection operation. The size of this target region is then used as the size of the target object. The target region is typically a three-dimensional spatial region including the target object. For example, ... Figure 5 As shown, when the target object selection operation involves the user clicking a point at the starting end and another point at the ending end of the target object 51 in the interactive interface, the 3D spatial coordinates of these two points can be determined based on the 3D model. The line connecting these two points is then established, and the center of this line is taken as the center of the target object 51. A spherical region 52 is then defined with the line as its diameter. This spherical region 52 is taken as the region corresponding to the target object, and its size is the size of the target object. Of course, the shape of the target region is not limited to a sphere; it can be a 3D spatial region of various shapes, such as a cuboid region or other shapes. When determining the target location, a location that can capture the entire spherical region can be selected as the target location.

[0055] like Figure 6 As shown, since the location of the target object is already determined, in order to ensure that the mobile platform operates on the target object at the expected working distance and to guarantee the working effect, the target position can be adjusted with the target object as the center, while keeping the working distance of the mobile platform unchanged. The angle between the line connecting the target object's location and the horizontal plane can be defined as the pitch angle, and the angle between the projection of the line connecting the target object's location and the north direction on the horizontal plane can be defined as the yaw angle. When adjusting the target position, the distance between the target object's location and the target location can be kept constant, and the pitch angle or yaw angle can be adjusted to fine-tune the target position. Therefore, in some embodiments, the target position adjustment operation includes a first adjustment operation, which can be the operation of adjusting the aforementioned pitch angle, such as... Figure 7 As shown, the first adjustment operation enables the target position to be adjusted on the first target circle, wherein the center of the first target circle is located at the position of the target object, the plane in which the first target circle is located is perpendicular to the horizontal plane, and the radius of the first target circle is the working distance of the movable platform.

[0056] In some embodiments, the adjustment operation of the target position includes a second adjustment operation, which may be an operation to adjust the yaw angle, such as... Figure 8 As shown, the second adjustment operation enables the target position to be adjusted on the second target circle, wherein the center of the second target circle is located at the projection position of the target object, which is obtained by projecting the position of the target object onto a horizontal plane passing through the target position, and the radius of the second target circle is the distance from the projection position to the target position.

[0057] In some embodiments, to allow users to intuitively see the adjustment path of the target position, a first target circle and / or a second target circle can be displayed on the interactive interface so that users can determine the adjustment path of the target position based on the displayed first and second target circles.

[0058] In some embodiments, to allow users to more intuitively understand whether the adjusted target position and orientation are suitable and optimal, the interactive interface may also include a preview window. During the adjustment of the target position and orientation, the preview window can display a preview of the mobile platform operating at the adjusted target position and orientation. For example, when using a drone to capture images of a target object, the preview window can display a preview of the images captured by the drone at the adjusted target position and orientation, allowing users to determine whether further adjustments to the drone's position and orientation are needed, and to decide on an adjustment strategy. Similarly, in scenarios involving a robotic arm on a mobile platform harvesting fruit, the preview window can display a dynamic diagram of the mobile platform at the adjusted target position and the robotic arm adjusted to the target orientation for fruit harvesting. This allows users to clearly see whether the mobile platform can successfully harvest fruit at the current target position and orientation, and then determine how to adjust the target position and orientation accordingly.

[0059] Since the adjustment of the target position and orientation is centered on the target object, in some embodiments, the position of the target object in the preview window can be fixed when adjusting the target position and orientation. For example, the target object can be kept in the center of the screen, and its position remains unchanged during the adjustment process so that the user can better observe the adjustment effect.

[0060] In some embodiments, an adjustment control can be provided in the interactive interface, and the adjustment operation to adjust the target position can be triggered through the adjustment control. For example... Figure 9 As shown, the interactive interface can display adjustment button 91, which users can click to adjust the target position and target orientation.

[0061] In some embodiments, to ensure the safe operation of the mobile platform, after determining the target location when the mobile platform is working on the target object, the distance between the target location and the nearest obstacle can be determined. If this distance is less than a preset safe distance, the target location and / or obstacle is marked so that the user can adjust the target location based on the marking. For example, the target location and obstacle can be highlighted in red on the interactive interface, or the target location and obstacle can be selected by box, so that the user can adjust the target location based on the marking.

[0062] In some embodiments, there can be multiple target objects. Assuming there are a first target object and a second target object, after determining the target location for performing work on the first target object based on the user's input of a first target object selection operation on the interactive interface (hereinafter referred to as the target location corresponding to the first target object selection operation), the user's input of a second target object selection operation can be obtained. Based on the user's input of the second target object selection operation, the target location for performing work on the second target object (hereinafter referred to as the target location corresponding to the second target object selection operation) can be determined. Then, the connection between the target location corresponding to the first target object selection operation and the target location corresponding to the second target object selection operation is displayed on the interactive interface. That is, each time the user inputs a target object selection operation on the interactive interface, the device executing this control method can automatically determine the target location corresponding to that target object selection operation and connect it with the previously determined target location to form the work path of the movable platform, which is then displayed on the interactive interface. Optionally, the aforementioned first target object selection operation and second target object selection operation can also correspond to the same target object; this embodiment of the application does not limit this.

[0063] Since obstacles may exist during the process of moving the mobile platform from one target location to another, in order to ensure the safety of the mobile platform operation, in some embodiments, after determining the line connecting the target location corresponding to the first target selection operation and the target location corresponding to the second target selection operation, the distance between the nearest obstacle to the line and the line can be determined. When the distance is less than a preset safety distance, the line and / or obstacle can be marked so that the user can adjust the target location corresponding to the first target selection operation and / or the target location corresponding to the second target selection operation.

[0064] Since there may be a need to repeatedly perform operations on target objects within the work area. For example, in a scenario where pesticides are sprayed on crops, multiple sprayings may be required. Therefore, after the user determines the target location and orientation for each target object through an interactive interface displaying a 3D model and generates a work path for the mobile platform, the user can store the work path. For instance, the user can store target parameters used to determine the target location and orientation so that when performing operations on target objects in the work area again, these target parameters can be directly called to determine the target location and orientation for the next operation.

[0065] In some embodiments, target parameters may include one or more of the following: target object position, target orientation, target location, working distance, target object size, and virtual camera position. For example, in some embodiments, the target location and target orientation can be stored, and during operation, the movable platform can be directly controlled to move to the target location and perform operations on the target object according to the target orientation. Alternatively, the target object position, target orientation, and working distance can also be stored, and when using the movable platform to operate on the target object, the target location can be directly determined based on the target object's position, target orientation, and working distance.

[0066] Because the location of the target object can be stored, and the target position and orientation can be determined based on the target object's location, after determining the target position and orientation for working on the target object in the work area based on the hardware parameters of the mobile platform, if another mobile platform is used to work on the same target object in the work area, the user does not need to re-set the work path through the interactive interface. In some embodiments, the aforementioned mobile platform is a first mobile platform. When a second mobile platform is used to work on the target object, the stored target parameters can be obtained, and then the position and orientation of the second mobile platform when working on the target object can be determined based on the stored target parameters and the hardware parameters of the second mobile platform. The hardware parameters of the second mobile platform can be the same as or different from those of the first mobile platform. For example, when the user switches to the second mobile platform to work on the target object, the focal length and sensor pixel width of the second mobile platform may be different from those of the first mobile platform. Therefore, the work distance can be re-determined based on the focal length and sensor pixel width of the second mobile platform. Since the location of the target object is known, and the target position and orientation are determined with the target object as the center, the position and orientation of the second mobile platform during operation can be re-determined based on the re-determined working distance and the location of the target object, so as to update the working path.

[0067] To further explain the mobile platform control method of this application, the following explanation is based on a specific embodiment.

[0068] Drones are typically used to inspect power equipment, bridges, dams, and other similar structures. They capture images of these targets and analyze them to determine if any faults have occurred. For example... Figure 10 The diagram illustrates an application scenario according to an embodiment of this application. A user can use software applications installed on the control terminal 102 to set the target orientation and position of the drone 101 when acquiring images of a target object in the work area. Based on the target orientation and position, a control command is generated and sent to the drone 101. Optionally, the control command can be sent to the drone 101 via a communication link between the control terminal 102 and the drone 101. Alternatively, the control command can be sent to a mobile platform via a communication link between the control terminal 102 and the remote controller 103, and between the remote controller 103 and the drone 101. Optionally, the control terminal 102 can be a laptop computer with 3D reconstruction software installed.

[0069] Users can open the software application on the control terminal 102. The 3D engine in the software application can load a 3D model of the work area. The 3D model can be obtained through photogrammetric 3D reconstruction or by scanning the work area with LiDAR. Users can input the spatial resolution through the interactive interface to determine the working distance of the UAV 101 based on the spatial resolution.

[0070] The 3D model of the work area can be displayed to the user through an interactive interface. Users can move and rotate the 3D model to a suitable orientation to ensure an unobstructed view of the target object, facilitating observation. A virtual camera exists within the 3D engine; the image displayed on the interface represents the view of the work area captured by this virtual camera. The orientation of the 3D model corresponds to the orientation of the virtual camera. When the user stops moving or rotating the 3D model, the 3D engine determines its current orientation and position. Users can click on the start and end positions of the target object within the 3D model. A line is generated connecting these positions, and the center of this line is determined as the target object's location. Once the target object's position is determined, a line is generated connecting it to the virtual camera's position. When the drone operates on the target object, the camera's orientation follows this line, pointing towards the target. During the operation, the drone's position is located on this line at a distance equal to the operation distance from the target object.

[0071] Furthermore, the angle between the line connecting the target object's location and the drone's operational location and the horizontal plane can be defined as the pitch angle, and the angle between the projection of this line onto the horizontal plane and the north direction can be defined as the yaw angle. After determining the drone's orientation and location during operation, the pitch and yaw angles can be fine-tuned to determine a better position and orientation for shooting. The interactive interface can include a preview window. When the user adjusts the pitch and yaw angles, a preview of the image captured by the drone at its current position and orientation can be displayed in the preview window, allowing the user to determine the adjustment strategy based on the preview. Once the user's click operation on the interactive interface determines the operational location for one target object, if the user clicks on another target object, the operational location for that other target object is determined. The two locations are then connected by a line, and this connection is displayed on the interactive interface. To ensure the drone's safety during operation, the nearest obstacle to the drone's operational location can be identified. It is determined whether the distance between the obstacle and the drone during operation is less than a preset safe distance. If it is less, the obstacle or its location is marked on the interactive interface so that the user can adjust the location accordingly. Simultaneously, it can identify the obstacle closest to the line connecting the drone's location to different target objects, and determine if the distance between the obstacle and the connecting line is less than a preset safe distance. If it is, the obstacle and the connecting line are marked for user adjustment. After determining the drone's position and orientation for each target object within the work area, the position, pitch angle, yaw angle, and working distance of the target object can be stored, and the drone's operation can be controlled based on these stored parameters. Of course, if the drone operating on the target object within the work area is changed—for example, if the drone's focal length, sensor pixel width, or other hardware parameters change—the working distance can be re-determined based on the drone's hardware parameters and the expected resolution. Then, the drone's position and orientation during operation can be re-determined based on the target object's position, pitch angle, yaw angle, and working distance. Since the position and orientation during operation are determined with the target object as the center, the user only needs to set the work path once in the interactive interface. Even if the hardware parameters of the drone change, the position during operation can be re-determined based on the stored target object's position and the drone's orientation, without requiring the user to reset it.

[0072] Using the methods described above, users can easily determine the drone's position and orientation during operation. Furthermore, any fine-tuning of the position is centered on the target object, ensuring it remains in the center of the frame. Since adjustments only change the orientation and not the operating distance, the operational effect is guaranteed. Because the target object's position is determined, the position can be easily adjusted based on the optimal operating distance for different drones to adapt to different aircraft and achieve the same shooting results, eliminating the need for users to redetermine the operating path for each drone.

[0073] In addition, this application also provides a control device for a mobile platform, such as Figure 11 As shown, the device includes a processor 111, a memory 112, and a computer program executable by the processor 111 stored in the memory 112. When the processor 111 executes the computer program, it performs the following steps:

[0074] The system obtains the target object selection operation input by the user in the interactive interface, which displays a three-dimensional model of the work area. The target object selection operation is used to determine the position of the target object in the work area.

[0075] Based on the orientation of the 3D model displayed on the interactive interface when the target object is selected, the target orientation for performing the operation on the target object is determined.

[0076] The target position for the mobile platform to perform operations on the target object is determined based on the position of the target object, the target orientation, and the working distance of the mobile platform, so that the mobile platform can move to the target position and perform operations on the target object according to the target orientation.

[0077] In some embodiments, the orientation of the 3D model displayed on the interactive interface corresponds to the orientation of the virtual camera.

[0078] In some embodiments, the target is oriented along a line connecting the position of the target object and the position of the virtual camera and points towards the position of the target object.

[0079] In some embodiments, the target location is located on the line connecting the location of the target object and the location of the virtual camera, and the distance from the target object is the working distance.

[0080] In some embodiments, the processor is further configured to:

[0081] Obtain the adjustment operation input by the user through the interactive interface;

[0082] Based on the position of the target object and the adjustment operation, the adjusted target position is determined;

[0083] The target orientation is adjusted according to the adjusted target position so that the target orientation points from the adjusted target position to the position of the target object.

[0084] In some embodiments, the adjustment operation includes a first adjustment operation that enables the target position to be adjusted on a first target circle, wherein the center of the first target circle is located at the position of the target object, the plane containing the first target circle is perpendicular to the horizontal plane, and the radius of the first target circle is the working distance of the movable platform.

[0085] In some embodiments, the adjustment operation includes a second adjustment operation that enables the target position to be adjusted on a second target circle, wherein the center of the second target circle is located at the projection position of the target object, the projection position being obtained by projecting the position of the target object onto a horizontal plane passing through the target position, and the radius of the second target circle is the distance from the projection position to the target position.

[0086] In some embodiments, the processor is further configured to:

[0087] The first target circle and / or the second target circle are displayed in the interactive interface.

[0088] In some embodiments, the interactive interface includes a preview window, and after adjusting the target position and the target orientation based on the position of the target object and the adjustment operation, it further includes:

[0089] The preview window displays a preview of the movable platform being positioned at the adjusted target location and operating according to the adjusted target orientation.

[0090] In some embodiments, when adjusting the target position and the target orientation, the position of the target object in the preview window is fixed.

[0091] In some embodiments, the interactive interface includes adjustment controls, and the adjustment operation is triggered based on the adjustment controls.

[0092] In some embodiments, the processor is further configured to: identify the target location and / or the obstacle when it is determined that the distance between the target location and the nearest obstacle is less than a preset safe operating distance.

[0093] In some embodiments, the processor is further configured to: the target object selection operation is a first target object selection operation, and the method further includes:

[0094] Get the user's selection of a second target object from the user interface;

[0095] The interactive interface displays the line connecting the target position corresponding to the first target object selection operation and the target position corresponding to the second target object selection operation.

[0096] In some embodiments, the processor is further configured to: identify the line and / or the obstacle when it is determined that the distance between the line connecting the target position corresponding to the first target selection operation and the target position corresponding to the second target selection operation and the nearest obstacle to the line is less than a preset safe operating distance.

[0097] In some embodiments, the processor is further configured to:

[0098] The working distance is determined based on spatial resolution and / or the size of the target object;

[0099] The spatial resolution refers to the physical size corresponding to a pixel, and the spatial resolution is the spatial resolution input by the user in the interactive interface or preset.

[0100] In some embodiments, the processor is further configured to:

[0101] Based on the target object selection operation, the target area corresponding to the target object is determined;

[0102] The size of the target region is used as the size of the target object.

[0103] In some embodiments, the processor is further configured to: store target parameters for determining the target position and the target orientation.

[0104] In some embodiments, the target parameters include one or more of the following: the position of the target object, the target orientation, the working distance, the size of the target object, and the position of the virtual camera.

[0105] In some embodiments, the movable platform is a first movable platform, and the processor is further configured to:

[0106] The stored target parameters are obtained, and the position and orientation of the second mobile platform during operation are determined based on the hardware parameters of the second mobile platform and the target parameters.

[0107] The control device of the mobile platform can be equivalent to or a part of the aforementioned control terminal. Specific details regarding the determination of the target position and orientation when the mobile platform operates on the target object can be found in the descriptions of the various embodiments of the above method, and will not be repeated here.

[0108] In addition, this application also provides a mobile platform, which is used to receive control commands, move to a target position based on the control commands, and perform operations on target objects in the work area according to the target orientation;

[0109] The control command is determined based on the following method: obtaining the target object selection operation input by the user in the interactive interface, the interactive interface displaying a three-dimensional model of the work area, and the target object selection operation being used to determine the position of the target object in the work area;

[0110] Based on the orientation of the 3D model displayed on the interactive interface when the target object is selected, the target orientation for performing the operation on the target object is determined.

[0111] The target position of the mobile platform when performing operations on the target object is determined based on the position of the target object, the target orientation, and the working distance of the mobile platform; the control command is generated based on the target orientation and the target position.

[0112] The specific implementation details of generating the control commands can be found in the descriptions of the various embodiments of the above method, and will not be repeated here.

[0113] Furthermore, this application also provides a control system, which includes a mobile platform and a control terminal. The control terminal is used to acquire a target object selection operation input by a user on an interactive interface. The interactive interface displays a three-dimensional model of the work area, and the target object selection operation is used to determine the position of the target object in the work area.

[0114] Based on the orientation of the 3D model displayed on the interactive interface when the target object is selected, the target orientation for performing the operation on the target object is determined.

[0115] The target position when the mobile platform performs operations on the target object is determined based on the position of the target object, the target orientation, and the working distance of the mobile platform;

[0116] The control command is generated based on the target orientation and the target position, and then sent to the mobile platform.

[0117] The mobile platform is used to move to the target location based on the control command and perform operations on the target object according to the target orientation.

[0118] The specific implementation details of the control terminal controlling the mobile platform to perform operations can be found in the descriptions of the various embodiments of the above method, and will not be repeated here.

[0119] Accordingly, embodiments of this specification also provide a computer storage medium storing a program that, when executed by a processor, implements the control method of the portable platform in any of the above embodiments.

[0120] The embodiments of this specification may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0121] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0122] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0123] The methods and apparatus provided in the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A control method for a mobile platform, characterized in that, The method includes: The system obtains a first target object selection operation and a second target object selection operation input by the user in the interactive interface, wherein the interactive interface displays a three-dimensional model of the work area, and the first target object selection operation and the second target object selection operation are used to determine the position of the first target object and the position of the second target object in the work area, respectively. The first target position of the mobile platform when performing operations on the first target object is determined based on the first target object selection operation, and the first target orientation of the mobile platform when performing operations on the first target object is determined based on the orientation of the three-dimensional model displayed on the interactive interface when the first target object selection operation is performed. The second target position of the mobile platform when performing operations on the second target object is determined based on the second target object selection operation, and the second target orientation of the mobile platform when performing operations on the second target object is determined based on the orientation of the three-dimensional model displayed on the interactive interface when the second target object selection operation is performed. The interactive interface displays a line connecting the first target location and the second target location, wherein the line forms the working path of the mobile platform.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the adjustment operation input by the user through the interactive interface; Based on the position of the first target object and the adjustment operation, the adjusted first target position is determined when the mobile platform performs operations on the first target object; The first target orientation is adjusted according to the adjusted first target position, so that the first target orientation points from the adjusted first target position to the position of the first target object.

3. The method according to claim 2, characterized in that, The adjustment operation includes a first adjustment operation, which enables the first target position to be adjusted on a first target circle, wherein the center of the first target circle is located at the position of the first target object, the plane in which the first target circle is located is perpendicular to the horizontal plane, and the radius of the first target circle is a first working distance, wherein the first working distance is the distance between the first target position and the position of the first target object.

4. The method according to claim 2, characterized in that, The adjustment operation includes a second adjustment operation, which enables the first target position to be adjusted on a second target circle, wherein the center of the second target circle is located at the projection position of the first target object, the projection position is obtained by projecting the position of the first target object onto a horizontal plane passing through the first target position, and the radius of the second target circle is the distance from the projection position to the first target position.

5. The method according to any one of claims 2-4, characterized in that, The interactive interface includes a preview window, and after adjusting the position and orientation of the first target based on the position of the first target object and the adjustment operation, it further includes: The preview window displays a preview of the movable platform being positioned at the adjusted first target location and operating according to the adjusted first target orientation.

6. The method according to claim 5, characterized in that, When adjusting the position and orientation of the first target, the position of the first target object in the preview window remains fixed.

7. The method according to claim 1, characterized in that, Also includes: When it is determined that the distance between the first target location and the nearest obstacle to the first target location is less than a preset safe operating distance, the first target location and / or the obstacle are marked.

8. The method according to claim 1, characterized in that, in, After the connection forms the operation path of the mobile platform, the operation path is stored; The step of determining the first target position when the mobile platform performs operations on the first target object based on the first target object selection operation includes: The first target position when the mobile platform performs operations on the first target object is determined based on the position of the first target object, the orientation of the first target, and the first working distance of the mobile platform, wherein the first working distance is the distance between the first target position and the position of the first target object; The step of determining the second target position when the mobile platform performs operations on the second target object based on the second target object selection operation includes: The second target position when the mobile platform performs operations on the second target object is determined based on the position of the second target object, the orientation of the second target, and the second working distance of the mobile platform, wherein the second working distance is the distance between the second target position and the position of the second target object.

9. The method according to claim 8, characterized in that, The first working distance and the second working distance are input by the user.

10. The method according to claim 1, characterized in that, The first target orientation or the second target orientation is the orientation of the movable platform, or the orientation of the camera device on the movable platform, or the orientation of the robotic arm on the movable platform.

11. A control device for a mobile platform, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executed by the processor. When the processor executes the computer program, it performs the following steps: The system acquires a first target object selection operation and a second target object selection operation input by the user on an interactive interface, wherein the interactive interface displays a 3D model of the work area, and the first target object selection operation and the second target object selection operation are used to determine the positions of the first target object and the second target object in the work area, respectively; based on the first target object selection operation, the system determines the first target position of the mobile platform when performing work on the first target object, and based on the orientation of the 3D model displayed on the interactive interface when the first target object selection operation is performed, the system determines the first target orientation of the mobile platform when performing work on the first target object; The second target position of the mobile platform when performing operations on the second target object is determined based on the second target object selection operation, and the second target orientation of the mobile platform when performing operations on the second target object is determined based on the orientation of the three-dimensional model displayed on the interactive interface when the second target object selection operation is performed. The interactive interface displays a line connecting the first target location and the second target location, wherein the line forms the working path of the mobile platform.

Images