Interaction method for unmanned aerial vehicle and mobile platform, computer device, storage medium, program product, and mobile platform

Abstract

The present application discloses an interaction method for an unmanned aerial vehicle and a mobile platform. The method comprises: obtaining first location information acquired by an unmanned aerial vehicle; obtaining second location information acquired by a mobile platform; and determining a relative location view of the unmanned aerial vehicle and the mobile platform at least on the basis of the first location information and the second location information. In the present application, by means of a configured display interface, a user can view a relative location view in a mobile platform, so that the user can observe, in the mobile platform, the relative location relationship between an unmanned aerial vehicle and the mobile platform.

Description

Interaction method between unmanned aerial vehicle and mobile platform, computer device, storage medium, program product and mobile platform Technical Field

[0001] The present application relates to the technical field of drones and mobile platforms, and in particular to an interaction method, computer equipment, storage medium, program product, and mobile platform for drones and mobile platforms. Background Art

[0002] In current vehicle-mounted drone applications, most drone storage, takeoff, and landing platforms are mounted on the vehicle's roof. This structural design prevents users from fully viewing the drone's flight posture through the vehicle's roof skylight, creating a blind spot and preventing the vehicle's computer from viewing the drone's landing process.

[0003] Summary of the Invention

[0004] The embodiments of the present application provide a method for interaction between a drone and a mobile platform, a computer device, a storage medium, a program product, and a mobile platform, which are used to solve at least one of the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present application provides an interaction method for a drone and a mobile platform, including: obtaining first position information collected by the drone; obtaining second position information collected by the mobile platform; and determining a relative position view of the drone and the mobile platform based on at least the first position information and the second position information.

[0006] In some embodiments, the relative position view includes a drone rendered image, a mobile platform rendered image, and relative position information of the drone and the mobile platform.

[0007] In some embodiments, the interaction method for a drone and a mobile platform further includes: obtaining first environmental information collected by the drone; obtaining second environmental information collected by the mobile platform; determining a relative position view of the drone and the mobile platform based on at least the first position information and the second position information, including: determining a relative position view between the drone, the mobile platform and environmental objects based on the first position information, the second position information, the first environmental information and the second environmental information.

[0008] In some embodiments, the relative position view also includes a rendering of environmental objects and relative position information between the drone and mobile platform and the environmental objects.

[0009] In some embodiments, the relative position view between the drone, the mobile platform and the environmental object is determined based on the first position information, the second position information, the first environmental information and the second environmental information, including: performing data fusion based on the first position information, the second position information, the first environmental information and the second environmental information to obtain the relative position view between the drone, the mobile platform and the environmental object.

[0010] In some embodiments, data fusion is performed based on the first position information, the second position information, the first environment information, and the second environment information to obtain a relative position view between the drone, the mobile platform, and the environmental objects, including:

[0011] determining at least one environmental object located around the UAV and the mobile platform and corresponding third position information based on the first environmental information and the second environmental information;

[0012] Performing data fusion on the first environment information and the second environment information to generate a drone rendering, a mobile platform rendering, and an environment object rendering;

[0013] A relative position view between the drone, the mobile platform and the environmental objects is generated based on the first position information, the second position information and the third position information as well as the drone rendering, the mobile platform rendering and the environmental object rendering.

[0014] In some embodiments, the first environmental information includes first perception information of at least one environmental object, and the second environmental information includes second perception information of at least one environmental object; the first perception information and the second perception information include perception information corresponding to different parts of the same environmental object; performing data fusion on the first environmental information and the second environmental information to generate an environmental object rendering includes: generating an environmental object rendering based on the perception information corresponding to different parts of the same environmental object in the first perception information and the second perception information.

[0015] In some embodiments, the method for interacting with a drone and a mobile platform further includes:

[0016] Acquire first environmental information collected by a drone, where the first environmental information is collected by a sensor on the drone side; acquire second environmental information collected by a mobile platform, where the second environmental information is collected by a sensor on the mobile platform side; perform feature extraction and fusion processing on the first environmental information and the second environmental information; and perform beyond-visual-range environmental perception based on the fused data.

[0017] In some embodiments, feature extraction and fusion processing are performed on the first environmental information and the second environmental information, including: feature extraction of the first environmental information and the second environmental information; determining the coordinate transformation relationship between the drone-side sensor and the mobile platform-side sensor; projecting the extracted features to the same feature space according to the coordinate transformation relationship; and fusing the features in the same feature space.

[0018] In some embodiments, beyond-visual-range environmental perception is performed based on the fused data, including: performing target detection on the fused features using a target detector; and / or generating a beyond-visual-range view based on the beyond-visual-range environmental perception results, the beyond-visual-range view including at least one of the following: vehicle, vehicle number, vehicle driving data, lane line, and traffic light.

[0019] In some embodiments, the method for interaction between a drone and a mobile platform further includes: acquiring a video stream captured by the drone; and presenting the video stream and a relative position view to a user.

[0020] In some embodiments, when an environmental object is within a preset range of the drone's moving path, the method further includes at least one of the following: generating an early warning message to alert the user; planning a new moving path; prompting the user to move the mobile platform to an open area; or prompting the user to control the drone.

[0021] In some embodiments, the method for interaction between a drone and a mobile platform further includes: automatically controlling the mobile platform to an open area in response to a user control operation of the mobile platform on the interaction interface.

[0022] In some embodiments, the method for interaction between a drone and a mobile platform further includes: controlling the flight of the drone in response to a user control operation of the user on the relative position view.

[0023] In some embodiments, the user control operation includes at least one of the following: an object selection operation, an object movement operation, and a path planning operation.

[0024] In some embodiments, the method for interacting with a drone and a mobile platform further includes: sending the mobile platform's destination information to the drone; receiving image information captured by the drone at the destination; and generating a planned path for the mobile platform to travel to the destination based on the image information. In a second aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement any of the steps of the method for interacting with a drone and a mobile platform described above.

[0025] In a third aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed by a processor, the steps of any one of the above-mentioned methods for interaction between a drone and a mobile platform in the present application are implemented.

[0026] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, it implements the steps of any of the above-mentioned methods for interaction between a drone and a mobile platform in the present application.

[0027] In this application, the user can view the relative position view through the configured display interface on the mobile platform, so that the user can observe the relative position relationship between the drone and the mobile platform on the mobile platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] FIG1 is a system diagram illustrating a method for interacting between a drone and a mobile platform according to the present application;

[0030] FIG2 is a schematic block diagram of an embodiment of a system for a method for interacting between a drone and a mobile platform in the present application;

[0031] FIG3 is a principle block diagram of an embodiment of a system for a method for interaction between a drone and a mobile platform in the present application;

[0032] FIG4 is a flow chart of an embodiment of a method for interaction between a drone and a mobile platform in the present application;

[0033] FIG5 is a relative position diagram displayed on a display interface provided by an embodiment of the present application;

[0034] FIG6 is a flow chart of another embodiment of the method for interaction between a drone and a mobile platform in the present application;

[0035] FIG7 is a flow chart of another embodiment of the method for interaction between a drone and a mobile platform in the present application;

[0036] FIG8 is a flow chart of another embodiment of the method for interaction between a drone and a mobile platform in the present application;

[0037] FIG9 is a schematic diagram of beyond visual range provided by an embodiment of the present application;

[0038] FIG10 is a flow chart of another embodiment of the method for interaction between a drone and a mobile platform in the present application;

[0039] FIG11 is a flow chart of another embodiment of the method for interaction between a drone and a mobile platform in the present application;

[0040] FIG12 is a schematic structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0042] It should also be noted that, in this document, the terms "include" and "comprising" include not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "include..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.

[0043] In the method for interaction between a drone and a mobile platform of the present application, the mobile platform includes but is not limited to any mobile platform that can carry a drone, such as a vehicle, a biped robot, a quadruped robot, a tracked robot and an aircraft. As shown in Figure 1, a system for implementing the method for interaction between a drone and a mobile platform of the present application is shown. In the system, the mobile platform is exemplified as a vehicle 100, and the system includes a vehicle 100 and a drone 200. The vehicle 100 can be a passenger car, a pickup truck, a truck, etc., and the present application does not limit the specific type of vehicle. The vehicle 100 and the drone 200 are connected in communication, and the user can send control instructions to the drone 200 through the vehicle 100. The drone 100 can perform flight missions (for example, shooting missions, pathfinding missions, etc.) in response to the control instructions, and send mission data obtained from performing the flight mission (for example, captured image data, sensor perception data, etc.) to the vehicle 100.

[0044] Figure 2 shows a block diagram of a system for a method of interacting with a drone and a mobile platform according to the present application. In this embodiment, drone 200 includes a flight controller, a positioning module, and a payload module. The positioning module may include at least one visual sensor, an inertial measurement module, a visual inertial navigation module, a GNSS module, and a multi-sensor fusion positioning module. It should be understood that the positioning module shown in the figure is for illustration only and may also include a lidar, millimeter-wave radar, or ultrasonic sensor. The payload module may include gimbal control, at least one main camera mounted on the gimbal, gimbal pose estimation, and a wireless data transmission module (on the air side). The flight controller may include a perception module and a decision-making, planning, and control module. The perception module is used to construct a local map based on data collected by the positioning module. For example, the local map can be obtained by processing images collected by the visual sensor (e.g., grayscale images) in combination with motion data collected by the inertial measurement module to obtain a depth map and semantic recognition results, and then obtaining a local map based on the combination of the depth map and semantic recognition results. The decision-making, planning, and control module is used to make mission decisions, plan the flight trajectory of drone 200, and control the flight of drone 200. It should be noted that the above content is only one implementation of the drone 200 that can be used to execute the interaction method for the drone and the mobile platform. Other forms can also be used, and this application does not limit this.

[0045] UAV 200 can receive control commands from vehicle 100 via a wireless data transmission module (including a wireless data transmission-air end and a wireless data transmission-on-vehicle end). Based on the control commands, it then makes mission decisions to generate specific commands for different flight missions or specific commands for acquisition tasks fixed to vehicle 100. It then controls the gimbal and / or UAV 200 based on the characteristic commands. For example, for gimbal control, the flight controller can control the gimbal's motion based on specific commands and the gimbal's pose estimation data. For example, for UAV 200 flight control, the flight controller can plan a flight trajectory based on specific commands and a local map generated by UAV 200. During the execution of the UAV 200 flight mission, the flight controller can perform flight control based on the planned flight trajectory and the UAV's pose determined based on data collected by various sensors in the sensing system. For example, the UAV's pose can be obtained through multi-sensor fusion positioning using images collected by the visual sensor, motion data collected by the inertial measurement module, and positioning data collected by the GNSS positioning module. The UAV 200 can also transmit the mission data collected during the flight mission to vehicle 100 via the data transmission module. Mission data includes, but is not limited to, drone pose data, gimbal pose data, image data collected by the main camera, and environmental perception data collected by various sensors. It is understood that the mission data fed back to vehicle 100 is merely illustrative, and different types of flight missions correspond to different mission data. For example, in a traffic information collection mission, the mission data sent back to vehicle 100 may include, in addition to drone pose data, gimbal pose data, and image data collected by the main camera, traffic environment information collected by other sensors, local maps obtained by the perception module, and so on. For example, in an aerial photography mission, the mission data sent back to vehicle 100 may include image data collected by the main camera.

[0046] As shown in Figure 3, it is a principle block diagram of an embodiment of the system for the interaction method between a drone and a mobile platform in this application. In this embodiment, a vehicle 100 and a drone 200 are included. Among them, the vehicle 100 includes a vehicle-side sensor, a drone data analysis module, an air perception module, a vehicle-side perception module, an air-ground fusion module, a planning and control module and an HMI module (human-machine interface). It should be noted that the above content only illustrates one implementation method of the drone 200 that can be used to execute the interaction method between a drone and a mobile platform. Other forms can also be used, and this application does not limit this. The following is an example of a traffic information collection task:

[0047] The drone data analysis module includes a wireless data transmission-vehicle-side module, which is used to receive mission data sent back by the drone 200, which may include image and video data of the drone 200, drone posture and other data; then the drone data analysis module parses the mission data of the drone 200, converts it into a data format suitable for algorithm processing, and outputs it to the aerial perception module for processing.

[0048] Vehicle-side sensors refer to the sensor hardware installed on the vehicle, which may include binocular cameras, monocular cameras, IMU, GPS, lidar and other sensors.

[0049] The aerial perception module is used to achieve beyond-visual-range traffic scene perception based on mission data transmitted by drone 200. It may include the following perception functions: parking space detection, road sign detection, drivable area detection, traffic flow analysis, and traffic scene recognition. Different perception functions are applied to different scenarios. For example, in parking scenarios, functions such as parking space detection and road sign detection are activated; in congestion scenarios, functions such as traffic flow analysis and traffic scene recognition are activated.

[0050] The vehicle-side perception module is used to realize ground perception functions based on the data collected by the vehicle-side sensors, which may include dynamic object detection, drivable area detection, lane line detection, road sign detection or parking position estimation.

[0051] The air-ground fusion module is used to fuse the output results of the aerial perception module and the vehicle-side perception module. Through technologies such as relative pose estimation, coordinate system alignment, and static map fusion, the vehicle-side perception range is expanded to hundreds of meters or even thousands of meters.

[0052] The planning and control module is used to plan the vehicle's driving trajectory based on the air-ground fusion perception results and user intentions, thereby achieving vehicle control.

[0053] The HMI module is used to realize beyond-visual-range multi-scenario autonomous driving visualization and human-computer interaction based on the results of air-ground fusion perception, which may include real-time image transmission display of drones (for example, flight images and map displays) and flight control. Among them, the drone can capture flight images in real time during flight and transmit them back to the vehicle for display in real time. In addition, map information can also be displayed in real time on the vehicle to realize navigation functions for the vehicle. Furthermore, the HMI can realize flight control of the drone in response to the user's control operation (for example, the user clicks, drags, slides, etc. on the vehicle's screen to generate control instructions and send them to the drone, thereby realizing flight control of the drone).

[0054] The HMI module also includes a cloud-based ADC (Airborne Digital Computer) and / or a gateway. The cloud-based ADC processes and forwards air-ground fusion perception results, air-based perception results, and / or vehicle-side perception results to the HMI module's app (for example, via a 4G or 5G link). The cloud-based ADC can also forward air-ground fusion perception results, air-based perception results, and / or vehicle-side perception results to the HMI module's app via a gateway.

[0055] As shown in FIG4 , an embodiment of the present application provides a method for interacting with a drone and a mobile platform, comprising: S10, obtaining first location information collected by the drone. Exemplarily, the first location information of the drone is collected using a location sensor configured on the drone. The location sensor can be a GPS, GNSS, Beidou navigation-related location sensor, or other location sensor capable of achieving positioning, which is not limited in this application. The first location information includes at least one of the drone's coordinate information and the drone's altitude information.

[0056] S20. Acquire second location information collected by the mobile platform. Exemplarily, the second location information of the mobile platform is collected using a location sensor configured on the mobile platform. The location sensor may be a GPS, GNSS, or Beidou navigation-related location sensor, or other location sensor capable of achieving positioning, which is not limited in this application. The second location information includes at least one of the following: coordinate information of the mobile platform, altitude information of the landing plane of the drone on the mobile platform, and the like.

[0057] S30. Determine a relative position view between the drone and the mobile platform based on at least the first position information and the second position information. Exemplarily, the relative position view between the drone and the mobile platform is determined based on at least one of the drone's coordinate information and its altitude information, and at least one of the mobile platform's coordinate information and the altitude information of the drone's landing plane on the mobile platform. Exemplarily, the drone's coordinate information and the mobile platform's coordinate information are three-dimensional coordinate information. For example, the relative position relationship between the drone and the mobile platform is calculated based on the drone's coordinate information and the mobile platform's coordinate information, and this is calibrated in the relative position view.

[0058] The mobile platform in the above embodiment may be a vehicle, and the user may view the relative position view through a display interface configured on the vehicle, so that the user can observe the relative position relationship between the drone and the vehicle in the vehicle.

[0059] In some embodiments, the relative position view includes a drone rendered image, a mobile platform rendered image, and relative position information of the drone and the mobile platform.

[0060] Figure 5 shows an example of a relative position view displayed on a display interface in this application. The figure shows a rendering of both the drone and the vehicle, and the height of the drone from the top of the vehicle is 5 meters. This embodiment presents the relative positions of the drone and the mobile platform in the form of a rendering, allowing users to more intuitively observe the relative positions of the drone and the vehicle.

[0061] In some embodiments, the relative position view includes a rendering of the mobile platform and relative position information between the drone and the mobile platform. The relative position information between the drone and the mobile platform includes an indicator showing the relative position of the drone and the mobile platform and / or angle information and / or distance information.

[0062] For example, when the drone is far away from the vehicle, the vehicle's display screen may only display a rendering of the vehicle, and then mark the position of the drone relative to the vehicle on the display screen (for example, using an arrow combined with angle and / or distance on the display screen to display the relative position of the drone and the vehicle).

[0063] FIG6 is a flow chart of another embodiment of the method for interacting with a drone and a mobile platform according to the present application. In this embodiment, the method for interacting with a drone and a mobile platform further includes:

[0064] S40: Acquire first environmental information collected by the drone.

[0065] Exemplarily, the first environmental information is collected by an environmental perception sensor provided on the drone. Environmental perception sensors include, but are not limited to, a visual perception module, a camera sensor, etc. The corresponding first environmental information is collected by at least one of the visual perception module, the camera sensor, etc.

[0066] The visual perception module includes, but is not limited to, cameras and radar. For example, multiple cameras are positioned around the drone and / or on its top and / or bottom to perceive the surrounding environment. For example, the camera-based perception data is converted into point cloud data for analysis and processing by the processor to model the surrounding environment (e.g., modeling trees or obstacles in the environment).

[0067] The camera sensor, for example, can be a drone's camera, primarily used to capture images or videos of the drone in flight. Furthermore, the data collected by the camera sensor can be combined with environmental data sensed by the visual sensor module to more realistically present the surrounding environment as perceived by the visual sensor.

[0068] S50: Acquire second environment information collected by the mobile platform.

[0069] Exemplarily, the second environmental information is collected by an environmental perception sensor provided on a mobile platform (e.g., a vehicle). The environmental perception sensors provided on a mobile platform (e.g., a vehicle) include, but are not limited to, visual cameras, laser radars, millimeter-wave radars, and ultrasonic radars. Accordingly, the second environmental information is collected by at least one of the visual cameras, laser radars, millimeter-wave radars, and ultrasonic radars. Among them, the visual cameras include, but are not limited to, 8M forward-looking inertial navigation binoculars, 3M surround-looking fisheyes, 3M / 8M rear-view monoculars, 3M side-view monoculars, etc.; the laser radars include, but are not limited to, forward laser radars and rear-view laser radars; the millimeter-wave radars include, but are not limited to, forward millimeter-wave radars and angular millimeter-wave radars.

[0070] In some embodiments, determining a relative position view of the drone and the mobile platform based on at least the first position information and the second position information includes:

[0071] A relative position view between the drone, the mobile platform, and the environmental object is determined based on the first position information, the second position information, the first environmental information, and the second environmental information. For example, the environmental object and its position information are determined based on the first environmental information and the second environmental information, and the relative position view between the drone, the mobile platform, and the environmental object can be further determined by combining the first position information of the drone and the second position information of the mobile platform.

[0072] Exemplarily, the position information of the environmental object, the first position information of the drone and the second position information of the mobile platform include three-dimensional coordinate information. Based on the three-dimensional coordinate information of the above three, the relative position relationship between the three can be calculated, and then calibration can be performed in the view to obtain a relative position view between the drone, mobile platform and environmental object.

[0073] In this embodiment, not only is the relative position between the drone and the mobile platform calibrated and displayed in the relative position view, but the relative positions of environmental objects to the drone and the mobile platform are also calibrated and displayed. Thus, through this embodiment of the application, the user can simultaneously grasp the relative position relationship between the drone and the mobile platform and environmental objects in the surrounding environment (for example, trees, signal towers, street lights, high-voltage line towers, etc.).

[0074] In some embodiments, as shown in Figure 5, the relative position view also includes renderings of environmental objects (such as the trees on either side of the drone in Figure 5) and information about the relative positions of the drone and mobile platform relative to the environmental objects. In this embodiment, by presenting the relative positions of the environmental objects, the drone, and the mobile platform in the form of renderings, the user can more intuitively observe the relative positions of the drone, the environmental objects, and the vehicle.

[0075] In some embodiments, the relative position view includes renderings of environmental objects and drones. When the distance between the drone and an environmental object exceeds a danger threshold, the relative position view also includes a danger warning message to alert the user to take evasive action. For example, the danger warning message may include at least a portion of the rendering of the drone and environmental object in proximity displayed in a warning color (red, as shown in FIG5 ).

[0076] As shown in Figure 5, when the environment object is a tree, the branches and leaves closest to the drone are highlighted in red, and the position of the drone near the tree is also highlighted in red to better remind the user of the collision risk. The user can choose whether to manually control the drone as needed to reduce the risk of collision and explosion.

[0077] In some embodiments, the relative position view between the drone, the mobile platform and the environmental object is determined based on the first position information, the second position information, the first environmental information and the second environmental information, including: performing data fusion based on the first position information, the second position information, the first environmental information and the second environmental information to obtain the relative position view between the drone, the mobile platform and the environmental object.

[0078] FIG7 is a flow chart of another embodiment of the method for interacting with a drone and a mobile platform according to the present application. In this embodiment, data fusion is performed based on the first position information, the second position information, the first environment information, and the second environment information to obtain a relative position view between the drone, the mobile platform, and the environmental objects, including:

[0079] S31. Determine at least one environmental object located around the UAV and the mobile platform and corresponding third position information based on the first environmental information and the second environmental information.

[0080] S32: Perform data fusion on the first environment information and the second environment information to generate a drone rendering, a mobile platform rendering, and an environment object rendering.

[0081] For example, the first environmental information is collected by the sensors of the drone, and the second environmental information is collected by the sensors of the mobile platform. The drone sensors and the mobile platform sensors are synchronized in time and space (i.e., sensor coordinates are aligned and time is synchronized). Then, a processing method such as a neural network is used to extract features from the first and second environmental information. The extracted features are projected into the same feature space using the aligned drone-vehicle sensor external parameters. The projected features are then fused using a processing method such as a neural network to obtain fused features. Finally, various detectors (e.g., target detectors) are used to detect the fused features and generate renderings of each target (including drone renderings, mobile platform renderings, and environmental object renderings).

[0082] S33: Generate a relative position view between the drone, the mobile platform, and the environmental objects based on the first position information, the second position information, the third position information, the drone rendering, the mobile platform rendering, and the environmental object rendering.

[0083] In this embodiment, environmental objects are determined by fusing the first environmental information collected by the drone and the second environmental information collected by the vehicle, and a drone rendering, a mobile platform rendering, and an environmental object rendering are generated, thereby improving the accuracy and completeness of the determined environmental objects.

[0084] In this embodiment, data fusion is performed based on the first position information, the second position information, the first environment information, and the second environment information to obtain a relative position view between the drone, the mobile platform, and the environmental objects, including:

[0085] determining at least one environmental object located around the UAV and the mobile platform and corresponding third position information based on the first environmental information and the second environmental information;

[0086] The first position information, the second position information, and the third position information are combined to perform data fusion on the first environment information and the second environment information to generate a relative position view between the UAV, the mobile platform, and the environmental objects.

[0087] For example, the first environmental information is collected by the sensors of the drone, and the second environmental information is collected by the sensors of the mobile platform. The drone sensors and the mobile platform sensors are synchronized in time and space (i.e., sensor coordinates are aligned and time is synchronized). A neural network or other processing method is then used to extract features from the first and second environmental information. The extracted features are projected into the same feature space using the aligned drone-vehicle sensor external parameters. The projected features are then fused using a neural network or other processing method to obtain fused features. Finally, the fused features are rendered in 3D using the first, second, and third position information to obtain a relative position view of the drone, mobile platform, and environmental objects.

[0088] In some embodiments, the first environmental information includes first perception information of at least one environmental object, and the second environmental information includes second perception information of at least one environmental object. In some embodiments, the first perception information and the second perception information include perception information corresponding to different parts of the same environmental object. Performing data fusion on the first environmental information and the second environmental information to generate an environmental object rendering includes: generating an environmental object rendering based on the perception information corresponding to different parts of the same environmental object in the first perception information and the second perception information. Exemplarily, the same environmental object is a tree, the first perception information includes perception information of the upper half of the tree, the second perception information includes perception information of the lower half of the tree, and further performing fusion processing (such as splicing) based on the perception information of the upper half of the tree and the perception information of the lower half of the tree to generate a rendering of the tree.

[0089] In some embodiments, the first perception information includes first-angle perception information corresponding to the first environmental object and the second environmental object, and the second perception information includes second-angle perception information corresponding to the first environmental object and the second environmental object; performing data fusion on the first environmental information and the second environmental information to generate an environmental object rendering includes: generating a rendering of the first environmental object and the second environmental object based on the first-angle perception information and the second-angle perception information.

[0090] In this embodiment, environmental perception is performed on the first object and the second object from different perspectives at the same time to obtain environmental perception information from two different perspectives, and fusion processing (for example, splicing) is performed based on the environmental perception information from two different perspectives to generate renderings of the first environmental object and the second environmental object.

[0091] The above embodiments solve the problem that the sensing range of drone sensors and vehicle sensors is limited and they cannot fully perceive environmental objects when used alone.

[0092] FIG8 is a flow chart of another embodiment of the method for interacting with a drone and a mobile platform according to the present application. In this embodiment, the method for interacting with a drone and a mobile platform further includes:

[0093] S71. Obtain first environmental information collected by the drone. The first environmental information is collected by a sensor on the drone. Exemplarily, the first environmental information is collected by an environmental perception sensor provided on the drone. Environmental perception sensors include, but are not limited to, visual perception modules, camera sensors, etc. The corresponding first environmental information is collected by at least one of the visual perception module, camera sensor, etc.

[0094] The visual perception module can be, for example, a camera. For example, multiple cameras are positioned around the drone and / or on its top and / or bottom, and these cameras perceive environmental data surrounding the drone. For example, the camera-based perception data is converted into point cloud data for analysis and processing by the processor to model the surrounding environment (e.g., modeling trees or obstacles in the environment).

[0095] The camera sensor, for example, can be a drone's camera, primarily used to capture images or videos of the drone in flight. Furthermore, the data collected by the camera sensor can be combined with environmental data sensed by the visual sensor module to provide a more realistic representation of the surrounding environment as perceived by the visual sensor.

[0096] S72. Acquire the second environmental information collected by the mobile platform, and the second environmental information is collected by the mobile platform-side sensor. Exemplarily, the second environmental information is collected by the environmental perception sensor provided on the mobile platform. The environmental perception sensors provided on the mobile platform (for example, a vehicle) include but are not limited to: visual cameras, laser radars, millimeter-wave radars, and ultrasonic radars. Accordingly, the second environmental information is collected by at least one of the visual cameras, laser radars, millimeter-wave radars, and ultrasonic radars. Among them, the visual cameras include but are not limited to 8M forward-looking inertial navigation binoculars, 3M surround-looking fisheyes, 3M / 8M rear-view monoculars, 3M side-view monoculars, etc.; the laser radars include but are not limited to forward laser radars and rear-view laser radars; the millimeter-wave radars include but are not limited to forward millimeter-wave radars and angular millimeter-wave radars.

[0097] S73. Extract and fuse features from the first and second environmental information. For example, extract features from the first and second environmental information; determine a coordinate transformation relationship between the drone-side sensor and the mobile platform-side sensor; project the extracted features into a common feature space based on the coordinate transformation relationship; and fuse the features in the common feature space.

[0098] S74. Perform beyond-visual-range environmental perception based on the fused data. For example, it is necessary to synchronize the drone-side sensor and the vehicle-side sensor in time and space, that is, align the sensor coordinate systems and synchronize the time. Then, use neural networks and other processing methods to extract features from the drone sensor data and the vehicle-side sensor data. Using the aligned drone-vehicle sensor external parameters, the extracted features are projected into the same feature space to obtain the beyond-visual-range feature representation range. The projected features are further fused using a neural network to obtain the fused features. Finally, various detectors (such as lane detectors and target detectors) are used to perform beyond-visual-range environmental perception based on the fused features.

[0099] For example, the drone-vehicle sensor extrinsic parameter refers to the conversion relationship between the two sensor coordinate systems. Specifically, the conversion relationship between the drone sensor coordinate system and the vehicle sensor coordinate system needs to be calculated.

[0100] For example, the beyond-visual-range feature representation range means that the vehicle-side camera can observe the surrounding 100m, and the drone can observe the area of ​​100m-500m because it flies far away. Then, after fusing the image seen by the drone camera and the image seen by the vehicle-side camera, the beyond-visual-range range of 0-500m can be observed.

[0101] For example, the neural network structure may employ a convolutional neural network (CNN), a transformer, or the like, which is not limited in this application. In this embodiment, feature extraction and fusion are performed simultaneously based on the first environmental information collected by the drone and the second environmental information collected by the vehicle. This enables environmental detection and observation within the short range that the vehicle can perceive, while also enabling beyond-visual-range environmental perception beyond the vehicle's line of sight through the first environmental information collected by the drone. This allows users to observe environmental conditions beyond the line of sight even while in the vehicle.

[0102] In some embodiments, performing beyond-visual-range environmental perception based on the fused data includes performing object detection on the fused features using an object detector. Exemplary object detectors include, but are not limited to, lane detectors, lane detectors, road sign detectors, vehicle detectors, ramp detectors, intersection detectors, and traffic light detectors.

[0103] In some embodiments, the method for interaction between a drone and a mobile platform further includes: generating a beyond-visual-range view based on the beyond-visual-range environmental perception result, wherein the beyond-visual-range view includes at least one of the following: vehicle, vehicle number, vehicle driving data, lane line, and traffic light. As shown in FIG9 , this is a beyond-visual-range schematic diagram provided in an embodiment of the present application. In this beyond-visual-range schematic diagram, the drone performs a pathfinding function, captures the traffic conditions on the road through the camera carried by the drone, and marks each vehicle and the speed, number, etc. of each vehicle in the view. Users can clearly understand the traffic conditions on the road through this beyond-visual-range schematic diagram.

[0104] FIG10 is a flow chart of another embodiment of the method for interacting with a drone and a mobile platform according to the present application. In this embodiment, the method for interacting with a drone and a mobile platform further includes:

[0105] S81. Obtain a video stream captured by a drone. For example, the video stream can be captured by a camera on the drone. S82. Present the video stream and relative position view to a user.

[0106] For example, the video stream and the relative position view between the drone and the vehicle are displayed to the user at the same time. On the one hand, the user can observe the actual picture taken by the drone, and on the other hand, the user can also grasp the relative position relationship between the drone and the vehicle in real time through the relative position view, that is, the user can clearly know where the drone is flying relative to the vehicle.

[0107] In some embodiments, the video stream and the relative position view can be displayed in different parts of the same display interface, or can be displayed on two different display interfaces. It can also be implemented as the relative position view being embedded in the video stream or the video stream being embedded in the relative position view. This application does not limit the specific presentation method of the video stream and the relative position view.

[0108] In some embodiments, when an environmental object is within a preset range of the drone's moving path, at least one of the following is also included: generating an early warning message to alert the user; planning a new moving path; prompting the user to move the mobile platform to an open area; prompting the user to control the drone. Among them, the moving path includes but is not limited to the drone's return path, the path planned when the drone takes off, and the path planned in real time during the drone's flight. For example, a combination scenario of a drone and a mobile platform (for example, a vehicle) is: a vehicle-mounted drone. In this scenario, the vehicle is usually provided with a position for the drone to take off and land, and the drone's one-button takeoff and one-button return function can be realized on the vehicle side. In the one-button return function, the drone's return path is usually planned based on the position of the vehicle and the position of the drone, and the drone is controlled to automatically return to the warehouse according to the path.

[0109] During the development of this application, the inventors discovered that the one-touch takeoff and one-touch return method is more suitable for relatively open environments. However, if there are obstacles such as trees near the vehicle, these obstacles may affect the drone's automatic return. Therefore, the inventors proposed the above-mentioned embodiment of this application. After determining the return path, the inventors further determine whether there are environmental objects (e.g., obstacles) within a preset range of the return path. If so, an alert message is generated to alert the user; a new movement path is planned; the user is prompted to move the mobile platform to an open area; and the user is prompted to control the drone. This ensures the safe return of the drone, preventing collisions with environmental obstacles during the return process. Exemplarily, the alert message includes but is not limited to a warning tone, graphic information (e.g., the tree on the left side of the drone is displayed red in Figure 5, and the left side of the drone is also displayed red), and voice prompts. Planning a new return path can, for example, involve planning a new return path that bypasses an environmental object if an environmental object is determined to be within the preset range of the return path. In addition to replanning the path, the user can also be prompted to move the vehicle to an open area or manually control the drone to return.

[0110] In some embodiments, the method for interacting with a drone and a mobile platform further includes automatically controlling the mobile platform to an open area in response to a user control operation on the mobile platform on an interactive interface. The user control operation includes at least one of the following: an object selection operation, an object movement operation, and a path planning operation. For example, the user operation on the mobile platform (e.g., a vehicle) on the interactive interface can be a drag operation. After selecting the vehicle, the user drags the vehicle to an open area displayed on the interactive interface. In response to this operation, the vehicle automatically drives to a corresponding location in the real scene. Alternatively, the user can directly click to select the vehicle and then click again on the open area displayed on the interactive interface. In response to this operation, the vehicle automatically drives to a corresponding location in the real scene. The user can also directly draw a vehicle trajectory on the interactive interface to instruct the vehicle to automatically drive to a specified location according to the trajectory, where the starting point of the trajectory is the vehicle's current location and the end point of the trajectory is the specified location the vehicle will be heading to. For example, the user can simply draw the trajectory directly on the vehicle's display screen with their finger. In response to the user's operation, the vehicle's autonomous driving system generates an actual trajectory mapped to the real environment and controls the vehicle based on the actual trajectory.

[0111] In some embodiments, the method for interacting with a drone and a mobile platform further includes controlling the flight of the drone in response to a user control operation on the relative position view. The user control operation includes at least one of the following: an object selection operation, an object movement operation, and a path planning operation.

[0112] For example, a user can draw a new return route on the relative position view displayed on the interactive interface that circumvents obstacles. After the user completes the operation, a new return path is generated based on the user's operation, and the drone is controlled to complete the return along the new return path. In addition, the user can also use other operations on the interactive interface to control the drone to achieve safe return. This application does not limit the specific operation method of the user.

[0113] FIG11 is a flow chart of another embodiment of the method for interacting with a drone and a mobile platform according to the present application. In this embodiment, the method for interacting with a drone and a mobile platform further includes:

[0114] S91. Send the destination information of the mobile platform to the UAV.

[0115] For example, the destination is a parking lot, and the mobile platform is a vehicle. The user sends the location information of the destination to the drone through the vehicle's onboard terminal. For example, the user opens the destination search function on the onboard terminal and shares the selected destination with the drone through an operation.

[0116] S92. Receive image information captured by the drone at the destination.

[0117] Exemplarily, after receiving destination information, the drone plans a flight path to the destination and flies to the destination according to the flight path. After arriving at the destination, the drone captures imagery of the destination. For example, in a parking lot, the drone captures images of available parking spaces and transmits them to the vehicle computer. Exemplarily, the imagery includes location information of vacant parking spaces. For example, after capturing images of the parking lot, the drone analyzes the images to identify vacant spaces and uses the drone's positioning capabilities to determine the locations of these vacant spaces.

[0118] S93. Generate a planned path for the mobile platform to travel to the destination based on the image information. For example, after receiving the image information of the destination captured by the drone, the vehicle-mounted terminal analyzes the image information and plans the vehicle's driving path after arriving at the destination. For example, if the destination is a parking lot, the vehicle-mounted terminal determines the available parking spaces in the parking lot based on the image information and plans a reasonable driving path (for example, selecting the parking space closest to the parking lot entrance and planning the driving path; or selecting the parking space closest to the parking lot exit and planning the driving path; or selecting the parking space closest to the parking lot elevator and planning the driving path, etc.).

[0119] For example, when the image information received by the vehicle computer from the drone includes the location information of an empty parking space, the vehicle computer directly generates a planned path to the empty parking space based on the location information of the empty parking space.

[0120] Through the method of this embodiment, the drone can be used to realize the pathfinding function, especially in the parking lot scene, to determine the available parking spaces and plan the driving path, which greatly saves the user's time in blindly searching for parking spaces in the parking lot.

[0121] In some embodiments, the interaction method for a drone and a mobile platform of the present application further includes: a. The mobile platform generates a moving path for the drone in response to a path planning operation.

[0122] For example, a user plans a drone's path through a mobile platform. The mobile platform automatically generates a drone's path based on the user's path planning operations and controls the drone to follow the path. For example, a user plans a drone's path through touch, voice, or gesture control on a vehicle's head unit display. Another example is a user setting a drone's mode (different modes may correspond to different preset paths) to control the drone to follow the mobile platform and shoot (e.g., surround shots).

[0123] b. The mobile platform sets the shooting parameters of the UAV in response to the parameter setting operation.

[0124] For example, the user sets the shooting parameters of the drone during flight (including but not limited to shooting angle, shooting height, shooting focal length and shooting mode, etc.) through the mobile platform. The mobile platform can obtain the image data taken by the drone in real time.

[0125] c. The mobile platform automatically generates a video clip based on the image data captured by the drone in response to a video generation operation. For example, a user controls the mobile platform through an operating interface on the mobile platform to automatically generate a video clip based on the received image data. For example, a user clicks a button on the vehicle's display screen that allows them to generate a video clip. In response to the user's click, the vehicle's control system automatically generates a video clip based on the currently received image data.

[0126] In some embodiments, the interaction method for a drone and a mobile platform of the present application may further include: d. The mobile platform embeds the clipped video in a relative position view for playback.

[0127] For example, after the clip video is generated, the mobile platform embeds the clip video in the relative position view for playback, so that the user can view the wonderful scenes in the image data collected by the drone during the flight while observing the drone and the mobile platform.

[0128] This application realizes the customization of the drone's movement path and shooting parameters during movement (such as shooting angle, altitude, focal length, mode, etc.) on the mobile platform side (for example, the vehicle side), and the real-time acquisition of image data. Finally, the one-click production function on the vehicle side can be used to automatically edit the captured video to form a driving blockbuster, and it can also be embedded in the 3D rendering view for display.

[0129] In some embodiments, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the steps of any of the above-mentioned methods for interaction between a drone and a mobile platform in the present application. In some embodiments, an embodiment of the present application provides a computer-readable storage medium, on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed by the processor, the steps of any of the above-mentioned methods for interaction between a drone and a mobile platform in the present application are implemented. In some embodiments, an embodiment of the present application provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by the processor, the steps of any of the above-mentioned methods for interaction between a drone and a mobile platform in the present application are implemented.

[0130] The computer device, computer-readable storage medium, and computer program product of the embodiments of the present application described above can be used to implement the method for interacting with a drone and a mobile platform of the embodiments of the present application, and accordingly achieve the technical effects achieved by the method for interacting with a drone and a mobile platform of the embodiments of the present application, which will not be described in detail here. In the embodiments of the present application, the relevant functional modules can be implemented by a hardware processor.

[0131] FIG12 is a schematic diagram of the hardware structure of a computer device for executing a method for interacting with a drone and a mobile platform according to another embodiment of the present application. As shown in FIG12 , the device includes:

[0132] One or more processors 1210 and memory 1220, with one processor 1210 used as an example in FIG12 . The device for executing the method for interacting with a drone and a mobile platform may also include an input device 1230 and an output device 1240. The processor 1210, memory 1220, input device 1230, and output device 1240 may be connected via a bus or other means, with FIG12 using a bus connection as an example. Memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the method for interacting with a drone and a mobile platform in the embodiments of the present application. By running the non-volatile software programs, instructions, and modules stored in memory 1220, the processor 1210 executes various functional applications and data processing of the server, thereby implementing the method for interacting with a drone and a mobile platform in the aforementioned method embodiment. Memory 1220 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the device for interacting with a drone and a mobile platform. Furthermore, memory 1220 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, memory 1220 may optionally include memory located remotely from processor 1210. Such remote memory may be connected to the device for interacting with a drone and a mobile platform via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. Input device 1230 may receive input digital or character information and generate signals related to user settings and function control of the device for interacting with a drone and a mobile platform. Output device 1240 may include a display device, such as a display screen. One or more modules stored in memory 1220, when executed by one or more processors 1210, perform the method for interacting with a drone and a mobile platform described in any of the above-described method embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for interaction between a drone and a mobile platform, comprising: Obtain the first location information collected by the drone; Acquiring second location information collected by the mobile platform; A relative position view of the UAV and the mobile platform is determined based on at least the first position information and the second position information.

2. The method according to claim 1, characterized in that The relative position view includes a drone rendering image, a mobile platform rendering image, and relative position information of the drone and the mobile platform.

3. The method according to claim 2, characterized in that Also includes: Obtain the first environmental information collected by the drone; Acquire the second environment information collected by the mobile platform; Determining a relative position view of the UAV and the mobile platform based on at least the first position information and the second position information includes: A relative position view between the drone, the mobile platform, and environmental objects is determined based on the first position information, the second position information, the first environmental information, and the second environmental information.

4. The method according to claim 3, characterized in that The relative position view also includes a rendering of an environmental object and relative position information between the UAV, the mobile platform, and the environmental object.

5. The method according to any one of claims 1 to 4, characterized in that Determining a relative position view among the UAV, the mobile platform, and an environmental object according to the first position information, the second position information, the first environmental information, and the second environmental information includes: Data fusion is performed based on the first position information, the second position information, the first environmental information, and the second environmental information to obtain a relative position view between the UAV, the mobile platform, and environmental objects.

6. The method according to claim 5, characterized in that Performing data fusion based on the first position information, the second position information, the first environment information, and the second environment information to obtain a relative position view between the UAV, the mobile platform, and environmental objects includes: Determine at least one environmental object located around the UAV and the mobile platform and corresponding third position information based on the first environmental information and the second environmental information; Performing data fusion on the first environmental information and the second environmental information to generate a drone rendering, a mobile platform rendering, and an environmental object rendering; A relative position view between the drone, the mobile platform and the environmental objects is generated according to the first position information, the second position information and the third position information and the drone rendering, the mobile platform rendering and the environmental object rendering.

7. The method according to claim 6, characterized in that The first environmental information includes first perception information of at least one environmental object, and the second environmental information includes second perception information of at least one environmental object; the first perception information and the second perception information include perception information corresponding to different parts of the same environmental object; Performing data fusion on the first environmental information and the second environmental information to generate an environmental object rendering includes: generating an environmental object rendering according to perception information corresponding to different parts of the same environmental object in the first perception information and the second perception information.

8. The method according to any one of claims 1 to 4, characterized in that Also includes: Acquire first environmental information collected by the drone, where the first environmental information is collected by a sensor on the drone end; Acquire second environmental information collected by the mobile platform, where the second environmental information is collected by a sensor on the mobile platform; performing feature extraction and fusion processing on the first environmental information and the second environmental information; Perform beyond-visual-range environment perception based on fused data.

9. The method according to claim 8, characterized in that Performing feature extraction and fusion processing on the first environmental information and the second environmental information, including: performing feature extraction on the first environmental information and the second environmental information; Determining a coordinate transformation relationship between the drone-side sensor and the mobile platform-side sensor; Projecting the extracted features into the same feature space according to the coordinate transformation relationship; The features in the same feature space are fused.

10. The method according to claim 8, characterized in that Performing beyond-visual-range environmental perception based on the fused data includes: performing target detection on the fused features using a target detector; and / or generating a beyond-visual-range view based on the beyond-visual-range environmental perception results, wherein the beyond-visual-range view includes at least one of the following: a vehicle, a vehicle number, vehicle driving data, a lane line, and a traffic light.

11. The method according to any one of claims 1 to 4, characterized in that Also includes: The mobile platform generates a movement path of the UAV in response to a path planning operation; The mobile platform sets shooting parameters for the drone in response to the parameter setting operation; The mobile platform automatically generates a clipped video based on the image data captured by the drone in response to a video generation operation.

12. The method according to any one of claims 1 to 4, characterized in that Also includes: Get the video stream collected by the drone; The video stream and the relative position view are presented to a user.

13. The method according to claim 3 or 4, characterized in that When the environmental object is within a preset range of the moving path of the drone, the method further includes at least one of the following: generating an early warning message to alert the user; planning a new moving path; prompting the user to move the mobile platform to an open area; and prompting the user to control the drone.

14. The method according to claim 13, characterized in that Also includes: In response to a user control operation on the mobile platform by a user on an interactive interface, the mobile platform is automatically controlled to an open area.

15. The method according to any one of claims 1 to 4, characterized in that Also includes: controlling the flight of the drone in response to a user control operation of the user on the relative position view; The user control operation includes at least one of the following: an object selection operation, an object movement operation, and a path planning operation.

16. The method according to claim 1, wherein Also includes: Send the destination information of the mobile platform to the drone; Receive image information taken by the drone at the destination; A planned path for the mobile platform to travel to the destination is generated based on the image information.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 16.

18. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

19. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

20. A mobile platform, characterized in that: The computer device according to claim 17 is installed.

Images