Control system and control method of unmanned aerial vehicle, host control system and vehicle

Abstract

The invention provides a control system and a control method of an unmanned aerial vehicle, a host control system and a vehicle. The control system of the unmanned aerial vehicle comprises a communication system, a host control system, at least one holographic image presentation device and at least one dual-channel image acquisition device, the dual-channel image acquisition device is arranged on the unmanned aerial vehicle; the dual-channel image acquisition device acquires first image data and / or second image data and sends the first image data and / or the second image data to the host control system; and the host control system performs three-dimensional scene reconstruction according to the first image data and controls the holographic image presentation device to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction. The two-channel image acquisition device sends the image data of different image modes to the host control system at the same time so as to meet the watching requirement of a user and the definition requirement of an algorithm at the same time, and the intelligent level of a vehicle and the user experience feeling are improved.

Description

Technical Field

[0001] The present invention belongs to the field of drone control technology, and specifically relates to a drone control system, a control method, a host control system and a vehicle, and more specifically to a drone control system and method, electronic equipment, computer storage media, a host control system and a vehicle. Background Art

[0002] With the development of drone technology, drone applications are becoming increasingly widespread, and the demand for a superior drone control experience is also increasing. In particular, with the development of smart cars, the integration of drones and smart cars has become increasingly popular. Currently, there are applications that equip passenger cars with drones, that is, installing vehicle-mounted drones on passenger cars. However, current drones generally cannot simultaneously meet user viewing needs and algorithm-based application requirements. Therefore, the deep integration of drones with automotive smart cockpits and intelligent driving is urgently needed. Summary of the Invention

[0003] The present invention was developed in response to the aforementioned issues. It provides a drone control system and method, electronic equipment, computer storage media, host control system, and vehicle. By employing a dual-channel image acquisition device to capture and provide accurate depth information and display it within the vehicle cabin, the present invention improves the safety and driving experience of passengers in passenger vehicles equipped with drones.

[0004] According to one aspect of the present invention, a control system for an unmanned aerial vehicle is provided, comprising:

[0005] Communication system, host control system, at least one holographic image presentation device, at least one dual-channel image acquisition device;

[0006] The dual-channel image acquisition device is arranged on the UAV; the dual-channel image acquisition device is used to acquire the first image data and / or the second image data, and send the first image data and / or the second image data to the host control system through the communication system;

[0007] The host control system is used to reconstruct a three-dimensional scene based on the first image data; and to control the holographic image presentation device to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction.

[0008] Exemplarily, the host control system is further configured to send a dual-channel acquisition instruction to the drone, so that the drone obtains the first image data and / or the second image data according to the dual-channel acquisition instruction.

[0009] Exemplarily, the first image data is an infrared image, and the second image data is a color image and / or an infrared image.

[0010] Exemplarily, the dual-channel image acquisition device is also used to send first image data to the host control system in a first image output mode, and to send automatically switched color images and / or infrared images to the host control system in a second image output mode, wherein the color images and / or infrared images are automatically switched according to a preset light intensity threshold, and to send color images to the host control system in a third image output mode.

[0011] Exemplarily, the host control system is further configured to control the dual-channel image acquisition device to be in the first image output mode and / or the second image output mode and / or the third image output mode.

[0012] Exemplarily, the control system further includes at least one holographic projection device, which is used to present a holographic image of the drone operating system through holographic projection.

[0013] Exemplarily, the control system further includes a camera module, which is used to obtain user operation data of the holographic image of the drone operating system and send the operation data to the host control system;

[0014] The host control system is further configured to generate corresponding control instructions based on the operation data, and send the control instructions to the drone so that the drone executes the control instructions.

[0015] Exemplarily, the control system also includes a voice system, which is used to obtain voice information provided by the user and send the voice information to the host control system. The host control system generates drone control instructions based on the voice information to control the operation of the drone.

[0016] According to another aspect of the present invention, a method for operating a drone is provided, comprising:

[0017] The host control system sends a dual-channel acquisition instruction to the drone, so that the unmanned dual-channel image acquisition device acquires the first image data and / or the second image data;

[0018] The host control system receives the first image data and / or the second image data sent by the dual-channel image acquisition device;

[0019] The host control system reconstructs a three-dimensional scene based on the first image data, and controls the holographic image presentation device to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction.

[0020] Exemplarily, the first image data is an infrared image, and the second image is a color image and / or an infrared image.

[0021] Exemplarily, the dual-channel acquisition instruction sent by the host control system controls the dual-channel image acquisition device to send first image data to the host control system in the first image output mode, and to send automatically switched color images and / or infrared images to the host control system in the second image output mode, wherein the color images and / or infrared images are automatically switched according to a preset light intensity threshold, and the color image is sent to the host control system in the third image output mode.

[0022] Exemplarily, the control method further includes: the host control system controls the dual-channel image acquisition device to be in the first image output mode and / or the second image output mode and / or the third image output mode.

[0023] Exemplarily, the control method further includes:

[0024] The host control system controls the holographic projection device to present a holographic image of the drone operating system through holographic projection;

[0025] The host control system sends an image acquisition instruction to the camera module to control the camera module to acquire image data of the user's operation on the holographic image of the drone operating system;

[0026] The host control system obtains the user's operation on the holographic image of the drone operating system based on the image data, and sends a drone control instruction to the drone based on the operation, so that the drone executes the drone control instruction.

[0027] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the drone control method as described above.

[0028] According to yet another aspect of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program for executing the above-mentioned method for controlling a drone.

[0029] According to another aspect of the present invention, a host control system is provided, comprising a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the drone control method as described above.

[0030] According to another aspect of the present invention, there is provided a vehicle comprising:

[0031] The drone and the drone's control system or host control system or electronic equipment installed on the vehicle;

[0032] The vehicle is provided with a drone platform for parking the drone.

[0033] According to the control system of the drone of an embodiment of the present invention, a dual-channel image acquisition device is installed on the drone, and the host control system controls the drone body and the control of the dual-channel image acquisition device. The first image data and the second image data are output simultaneously through the dual-channel image acquisition device, that is, the image data is output simultaneously through the two channels of the dual-channel image acquisition device, which can meet the user's viewing needs while also meeting the application needs of the algorithm end, thereby improving the vehicle's intelligence level and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0035] Figure 1 A schematic block diagram of a first UAV control system according to an embodiment of the present invention is shown;

[0036] Figure 2 A schematic structural diagram showing a dual-channel telephoto binocular camera in a drone control system according to one embodiment of the present invention;

[0037] Figure 3 A schematic block diagram of a second UAV control system according to an embodiment of the present invention is shown;

[0038] Figure 4 A schematic flow chart illustrating a method for controlling a drone according to an embodiment of the present invention is shown;

[0039] Figure 5 A schematic block diagram of a vehicle according to one embodiment of the present invention is shown;

[0040] Figure 6 A schematic diagram illustrating a location for mounting a drone in a vehicle according to one embodiment of the present invention;

[0041] Figure 7a A schematic diagram illustrating the field of view of an image captured by a drone after the drone leaves a vehicle and reaches a first altitude according to one embodiment of the present invention;

[0042] Figure 7bA schematic diagram illustrating the field of view of an image captured by a drone after the drone leaves the vehicle and reaches a second altitude according to one embodiment of the present invention;

[0043] Figure 8a A schematic diagram illustrating the arrangement of a host control system, a camera module, a holographic image presentation device, and a holographic projection device in a vehicle cabin according to one embodiment of the present invention is shown;

[0044] Figure 8b A schematic diagram illustrating the interior layout of a vehicle cabin including a holographic image of a drone operating system according to one embodiment of the present invention is shown;

[0045] Figure 9 A schematic block diagram of the control execution of a vehicle-mounted drone according to one embodiment of the present invention is shown;

[0046] Figure 10 A schematic diagram of an application scenario of a vehicle-mounted drone control system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application more apparent, example embodiments according to the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0048] With the development of drone technology, drone applications are becoming increasingly widespread, and the demand for a superior control experience is also increasing. In particular, with the development of smart cars, the integration of drones and smart cars has become increasingly popular. Currently, there are applications where drones are installed on passenger cars. However, current drones often cannot simultaneously meet both the viewing needs of users and the application requirements of algorithms. Current drones can only output images in a single image mode at a time. For example, color images can satisfy the user's viewing experience, but when these images are used for algorithmic applications, the algorithm application becomes more difficult, such as increasing the complexity of the algorithm. Furthermore, when drones output images that are more suitable for algorithmic applications, such as non-color images, these images can easily produce a poor visual experience when displayed, resulting in a poor user viewing experience. Therefore, when drones can only output images in a single image mode at a time, they cannot balance the requirements of image algorithms with the sensory needs of the human eye, and thus the user experience.

[0049] In addition, current drones lack deep information fusion with intelligent driving and intelligent cockpits. For example, the image and video information collected by drones cannot reflect the depth information of objects outside the vehicle, and cannot display depth images, resulting in incomplete displayed information. The vehicle perception system also has system blind spots, affecting the safety of drivers and passengers. At the same time, the current remote control device for vehicle-mounted drones is usually a physical remote control device independent of the drone. Users can control the drone in the cabin or outside the vehicle. However, if the drone is controlled in the cabin, the remote control device needs to be configured in the limited space of the cabin, which reduces the comfort of the vehicle drivers and passengers. If the drone is controlled outside the vehicle, in encountering bad weather, such as high temperature, cold or rainy and snowy days, the user's experience of controlling the drone outside the vehicle is also greatly reduced.

[0050] Furthermore, in open areas without traffic signs, the intelligent driving perception system's sensors collect limited image information and road sign information, limiting the decision-making capabilities of the regulatory control system. Current solutions for providing distance information typically fuse data from cameras and sensors, such as using lidar and millimeter-wave radar to generate 3D images. However, this approach results in large blind spots at close range, high hardware costs, and significant implementation challenges.

[0051] Based on the above situation, it is very necessary to propose a solution to achieve deep integration of drones and smart cars to bring a better product experience. Therefore, this application provides a drone control system and method, electronic equipment, computer storage medium, host control system and vehicle.

[0052] Below, refer to Figures 1 to 3 as well as Figures 6 to 10 The control system of the UAV according to the embodiment of the present invention is described.

[0053] like Figure 1 As shown, the control system 100 of the UAV includes:

[0054] Communication system 110, host control system 120, at least one dual-channel image acquisition device 130, at least one holographic image presentation device 140;

[0055] The dual-channel image acquisition device 130 is disposed on the drone 10; the dual-channel image acquisition device 130 is used to acquire first image data and / or second image data, and transmit the first image data and / or second image data to the host control system 120 via the communication system 110;

[0056] The host control system 120 is used to reconstruct a three-dimensional scene based on the first image data; and control the holographic image presentation device 140 to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction.

[0057] The dual-channel image acquisition device 130 includes two data transmission channels, one data channel for acquiring first image data, and the other data channel for acquiring second image data. After acquiring the first image data and / or the second image data, the dual-channel image acquisition device 130 transmits the first image data and / or the second image data to the host control system 120 via the communication system 110. When the data acquired by the dual-channel image acquisition device 130 is the first image data and the second image data, the communication system 110 can simultaneously transmit the first image data and the second image data to the host control system 120. After receiving the first image data, the host control system 120 reconstructs the three-dimensional scene based on the first image data to obtain a clear three-dimensional image. The clear three-dimensional image can be provided to the algorithm end to meet the algorithm end's image requirements. After receiving the second image data, the host control system 120 holographically displays the second image data through the holographic image presentation device 140, meeting the user's visual requirements for the image, thereby improving the user experience.

[0058] A communication system 110 is provided on the drone, the host control system 120, and the holographic image presentation device 140 to complete the interactive transmission of data. The communication system 110 includes a positioning device and a wireless communication device. The positioning device includes but is not limited to positioning navigation satellites, and the wireless communication device includes but is not limited to 4G / 5G networks and mobile hotspots (Wireless Fidelity, wifi), and Bluetooth. Specifically, a positioning device is provided on the drone, and a wireless communication device is also provided on the drone. At the same time, the dual-channel image acquisition device 130 also uses the wireless communication device on the drone for data transmission; a wireless communication device is provided on the holographic image presentation device 140, and a wireless communication device is provided on the host control system 110. Among them, the host control system 110 includes but is not limited to a central control multimedia host controller, a global positioning system (GPS), and an inertial measurement unit (IMU).

[0059] Exemplarily, the host control system is further configured to send a dual-channel acquisition instruction to the drone, causing the drone to obtain the first image data and / or the second image data in accordance with the dual-channel acquisition instruction. The drone flies in accordance with the flight instruction, and upon receiving the dual-channel acquisition instruction, the dual-channel image acquisition device on the drone acquires the first image data and / or the second image data in accordance with the dual-channel acquisition instruction.

[0060] For example, the dual-channel image acquisition device uses a dual-channel telephoto binocular camera, such as Figure 2As shown, the telephoto binocular camera includes two image acquisition units, each of which is dual-channel. Therefore, each image acquisition unit is connected to two video interfaces, namely a first video interface 1310 and a second video interface 1320. The first image data collected in the first image acquisition mode is output through the first video interface 1310, and the second image data collected in the second image acquisition mode is output through the first video interface 1320. Since the binocular camera is composed of two image acquisition units with overlapping fields of view to form the hardware part, it performs image acquisition and calculates the coordinates of any point in space using the parallax principle through steps such as intrinsic parameter calibration, image correction, and stereo matching of the image acquisition units to obtain depth information of objects in the scene. Therefore, when a dual-channel telephoto binocular camera performs image acquisition, the image acquisition modes of the two image acquisition units should be consistent. That is, the first image acquisition mode of the first image acquisition unit is the same as the first image acquisition mode of the second image acquisition unit, and the second image acquisition mode of the first image acquisition unit is the same as the second image acquisition mode of the second image acquisition unit. When the first image data and the second image data are transmitted to the host control system at the same time, this can be achieved through the method defined by the Camera Serial Interface (CSI) protocol of the hardware interface or by extending the virtual channel of the host control system.

[0061] Exemplarily, the first image data is an infrared image, and the second image data is a color image and / or an infrared image. The infrared (IR) image is used to reconstruct the 3D scene, and the reconstructed 3D scene is then used in the algorithm. The second image data is a color (RED, GREEN, BLUE, RGB) image or an infrared image (IR image). An RGB image or IR image is obtained based on actual conditions, allowing users to obtain images in true color to meet their viewing needs.

[0062] Exemplarily, the dual-channel image acquisition device is also used to send first image data to the host control system in a first image output mode, and to send automatically switched RGB images and / or IR images to the host control system in a second image output mode, wherein the RGB images and / or IR images are automatically switched according to a preset light intensity threshold, and to send RGB images to the host control system in a third image output mode.

[0063] The dual-channel image acquisition device can have three output modes, including: fixed IR mode, fixed RGB mode and RGB / IR mode. In RGB / IR mode, it can automatically switch between RGB mode and IR mode. In fixed IR mode, the output image mode is IR image, and in fixed RGB mode, the output image is RGB image. In RGB / IR mode, the output image will be RGB image or IR image according to the light intensity. The dual-channel image acquisition device includes an RGB / IR photosensitive chip. Taking a dual-channel telephoto binocular camera as an example, Figure 2 As shown, both image acquisition units of the telephoto binocular camera contain an RGB / IR photosensitive chip. Each RGB / IR chip contains two data channels. The image mode captured by each data channel can be a fixed RGB mode, a fixed IR mode, or an RGB / IR mode. In the RGB mode, the image is an RGB image, which can be considered a color image. In the IR mode, the image is an IR image. In the RGB / IR mode, the image mode can be automatically switched according to the light intensity captured by the RGB / IR photosensitive chip.

[0064] Exemplarily, the host control system is further configured to control the dual-channel image acquisition device to be in a first image output mode and / or a second image output mode and / or a third image output mode, and to control the selection of the image mode of the dual-channel image acquisition device in the first image acquisition mode and the second image acquisition mode. The host control system selects and determines the image output mode of the dual-channel image acquisition device, that is, controls the dual-channel image acquisition device to be in the first image output mode, the second image output mode, or the third image output mode, outputs the first image data in the first image output mode, and outputs the second image data in the second image output mode. The host control system controls which image output mode the dual-channel image acquisition device is in, including but not limited to sending a separate instruction to control the image output mode of the dual-channel image acquisition device or integrating the operation of controlling the output image mode into the dual-channel acquisition instruction.

[0065] In another embodiment, by setting up at least one holographic projection device in the control system, the drone operating system is holographically projected, thereby converting the current need to control the drone through an entity into controlling the drone without an entity.

[0066] For example, Figure 3 、 Figure 8bAs shown, the control system also includes at least one holographic projection device 150, which is used to present a holographic image 1510 of the drone operating system through holographic projection. The drone operating system is displayed through the holographic projection device 150. At this time, operating the buttons and joysticks in the holographic image 1510 of the drone operating system is equivalent to operating the buttons and joysticks in the physical drone operating system. The holographic projection device 150 forms a holographic image 1510 of the drone operating system in the air. The holographic image 1510 of the drone operating system in the air can be moved and scaled, making it easier for users to use. The holographic image 1510 of the drone operating system can also be transmitted to the holographic image presentation device via the communication system 110 for display.

[0067] For example, the control system further includes a camera module, which is used to obtain the user's operation data on the holographic image of the drone operating system and send the operation data to the host control system; the host control system is also used to generate corresponding control instructions according to the operation data, and send the control instructions to the drone 10 so that the drone executes the control instructions. Figure 3 、 Figure 8a 、 Figure 8b As shown, the camera module can be a Time of Flight (TOF) camera module 160, which transmits the user's operation data of the holographic image 1510 of the drone operating system to the host control system 120 through the TOF camera module 160. The host control system 120 analyzes these operation data, generates control instructions for the drone 10 through analysis, and then sends the generated control instructions to the drone 10. The drone 10 performs corresponding flight according to the received control instructions. The camera module can also be a common camera used for gesture recognition (such as an RGB camera, a depth camera, etc.), as well as other emerging cameras.

[0068] Exemplarily, the control system also includes a voice system, which is used to obtain voice information provided by the user and send the voice information to the host control system. The host control system generates drone control instructions based on the voice information to control the operation of the drone.

[0069] The user's voice information is obtained through the voice system and transmitted to the host control system through the communication system. The host control system recognizes the voice information and generates drone control instructions and sends them to the drone. The drone performs the corresponding operations and feeds back information such as the drone's position, speed, and image data to the host control system.

[0070] like Figure 4 According to another aspect of the present invention, a method for operating a drone is provided, comprising:

[0071] S210: The host control system sends a dual-channel acquisition instruction to the drone, so that the dual-channel image acquisition device of the drone acquires the first image data and / or the second image data;

[0072] When the UAV is performing its mission, it collects images through the dual-channel image acquisition device on the UAV. Specifically, Figure 1 and Figure 10 As shown, the host control system 120 sends a travel operation instruction to the drone 10 and the drone 10 flies along the specified route. The host control system 120 sends a dual-channel acquisition instruction to the drone 10. After the drone 10 receives the dual-channel acquisition instruction, the dual-channel image acquisition device 130 installed on the drone 10 acquires images according to the dual-channel acquisition instruction. The dual-channel image acquisition device 130 acquires the first image data and / or the second image data according to the instructions of the host control system 120.

[0073] S220: The host control system receives the first image data and / or the second image data sent by the dual-channel image acquisition device;

[0074] After obtaining the first image data and / or the second image data, the dual-channel image acquisition device transmits the first image data and / or the second image data to the host control system. In embodiments of the present invention, since the dual-channel image acquisition device has two channels, the first image data and the second image data can be simultaneously transmitted to the host control system. After receiving the first and second image data, the host control system performs corresponding processing for subsequent use. In this way, the user can obtain the relevant first and second image data simultaneously, and perform different functions at the same time.

[0075] S230: The host control system reconstructs a three-dimensional scene according to the first image data, and controls the holographic image presentation device to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction.

[0076] After the host control system receives the first image data and the second image data, it reconstructs the three-dimensional scene for the first image data. In one embodiment of the present invention, the dual-channel image acquisition device adopts a dual-channel telephoto binocular camera. The first image data includes 2D image data and 3D image data, wherein the image data collected in the non-overlapping area of the field of view of the telephoto binocular camera is 2D image data, and the image data collected in the overlapping area of the field of view of the telephoto binocular camera is 3D image data. The 2D image data is converted into 3D image data through the binocular ranging principle and combined with the fusion algorithm, the stitching algorithm and the smooth transition algorithm. The 3D data is then further processed by the AI algorithm in the host control system to construct a clear 3D reconstructed image. The 3D reconstructed image can be displayed by a holographic image presentation device, such as a holographic display device, and can also be used for other applications, such as when a drone is installed. In the field of smart cars, 3D reconstructed images can be used for intelligent driving algorithm applications, such as lane keeping, road condition marking, parking functions, etc. Among them, the binocular ranging principle is as follows: binocular vision is constructed by two cameras with overlapping fields of view to form the hardware part for image acquisition, and through the camera's internal parameter calibration, image correction, stereo matching and other steps, the coordinates of any point in the space are calculated using the parallax principle to obtain the depth information of objects in the scene; the host control system receives the second image data and processes it, and then transmits the data to the holographic image presentation device through the communication system for display. For example, the color image is sent to the host control system in the form of the second image data, and the host control system sends the color image to the holographic image presentation device. The true color of the image taken by the user with the drone can be displayed through the holographic image presentation device, thereby meeting the user's viewing needs.

[0077] When image data of different image modes is needed, the first image data and the second image data of different image modes are collected by a dual-channel image acquisition device, and the first image data and the second image data are sent to the host control system at the same time. The host control system processes the first image data and the second image data differently, so as to meet the application requirements of the image algorithm and the viewing requirements of the user at the same time, thereby improving the user experience of the drone.

[0078] Exemplarily, the first image data is an IR image, and the second image is an RGB image and / or an IR image. The IR image is used for 3D scene reconstruction, and the reconstructed 3D image data is used for algorithm applications. The second image acquisition mode is set to RGB / IR mode. In this mode, the dual-channel image acquisition device automatically captures RGB or IR images based on light intensity. Image data in RGB / IR mode can be used for outdoor photography, avoiding the discomfort and color distortion caused by pure IR image display, thereby providing a better user experience.

[0079] Exemplarily, the dual-channel acquisition instruction sent by the host control system controls the dual-channel image acquisition device to send first image data to the host control system in a first image output mode, to send automatically switched RGB images and / or IR images to the host control system in a second image output mode, wherein the RGB images and / or IR images are automatically switched according to a preset light intensity threshold, and to send an RGB image to the host control system in a third image output mode. Specifically, the first image output mode is a fixed IR mode, the second image output mode is an RGB / IR mode, and the third image output mode is a fixed RGB mode. In the RGB / IR mode, the output image can automatically switch to collect RGB image data or IR image data according to the light intensity. When the light is strong, RGB image data is collected, and when the light is weak, IR image data is collected. The light intensity threshold can be set in the host control system.

[0080] Exemplarily, the control method further includes: the host control system controlling the dual-channel image acquisition device to be in a first image output mode and / or a second image output mode and / or a third image output mode. The host control system may control the image output mode selection of the dual-channel image acquisition device by separately sending an image mode selection instruction, or by setting the image mode selection function in the dual-channel acquisition instruction. If the image mode selection function is set in the dual-channel acquisition instruction, the host control system sends the dual-channel acquisition instruction to the drone, and the dual-channel image acquisition device determines the first image acquisition mode and the second image acquisition mode upon receiving the instruction.

[0081] Exemplarily, the control method further includes:

[0082] The host control system controls the holographic projection device to present a holographic image of the drone operating system through holographic projection;

[0083] like Figure 8a and Figure 8b As shown, a holographic image 1510 of the drone operating system is obtained through the holographic projection device 150. The holographic image 1510 of the drone operating system includes all functions of the physical operating system of the drone 10, such as buttons and joysticks. The user controls the flight of the drone 10 by operating the holographic image 1510 of the drone operating system. The holographic image 1510 of the drone operating system can be displayed in the air, and the user can move and scale the holographic image 1510 of the drone operating system according to actual needs to facilitate user operation.

[0084] The host control system sends an image acquisition instruction to the camera module to control the camera module to capture image data of the user's operation on the holographic image of the drone operating system. The camera module can be a common camera used for gesture recognition, such as an RGB camera, a depth camera, an infrared camera, a stereo vision camera, an ultrasonic camera, and a multispectral camera. The camera module can also be other emerging cameras. The camera module of the present invention is described using a TOF camera module in a depth camera as an example.

[0085] Specifically, if Figure 3 、 Figure 8a and Figure 8b As shown, the TOF camera module 160 is a camera module that is used to collect image data, including user operation data on the holographic image 1510 of the drone operating system. The host control system 120 sends an image acquisition instruction to the TOF camera module 160, and the TOF camera module 160 collects image data, including user operation data on the holographic image 1510 of the drone operating system, according to the image acquisition instruction. In a specific embodiment, for example, in a smart car equipped with a drone, the image data collected by the TOF camera module 160 may also include gesture operations of the driver and passengers.

[0086] The host control system obtains the user's operation on the holographic image of the drone operating system based on the image data, and sends a drone control instruction to the drone based on the operation, so that the drone executes the drone control instruction.

[0087] Specifically, if Figure 3 、 Figure 8a and Figure 8b As shown, the TOF camera module 160 is a camera module. After the TOF camera module 160 collects the corresponding TOF image data according to the image acquisition instruction, the TOF image data is sent to the host control system 120 through the communication system 110. After receiving the TOF image data, the host control system 120 analyzes the TOF image data and generates a drone control instruction according to the analysis result. The host control system 120 sends the drone control instruction to the drone through the communication system 110. After receiving the drone control instruction, the drone executes the relevant operations in the drone control instruction.

[0088] A holographic image 1510 of the drone operating system is generated by the holographic projection device 150. The holographic image 1510 of the drone operating system is a non-physical drone operating system. The TOF camera module 160 collects the user's operation data on the holographic image 1510 of the drone operating system, and sends the user's operation data on the holographic image of the drone operating system to the host control system 120. The host control system 120 generates and sends drone control instructions. The drone 10 completes the corresponding operation according to the drone control instructions, thereby realizing non-physical operation of the drone operating system. This saves space resources and can operate the drone 10 at a suitable location according to the user's actual needs. If necessary, the user can also set up multiple holographic projection devices 150 to form holographic images 1510 of the drone operating system at different locations, so that different users can operate the drone 10 at different locations. In this scenario, multiple users cooperate to operate the drone 10, improving the playability of drone control.

[0089] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the drone control method as described above.

[0090] According to another aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for executing the above-described method for controlling a drone. According to another aspect of the present invention, a host control system is provided, comprising a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing a device equipped with the processor to execute the method for controlling a drone.

[0091] According to another aspect of the present invention, there is provided a vehicle comprising:

[0092] A drone mounted on the vehicle and a control system for the drone, a host control system, or an electronic device as described above;

[0093] The vehicle is provided with a drone platform for parking the drone 10 .

[0094] like Figure 5 As shown, the vehicle 300 is equipped with a drone 10 and a drone control system 100.

[0095] In one embodiment, Figure 6As shown, a drone platform is provided on the roof of the vehicle 300, a groove is provided in the drone platform for parking the drone 10, and an openable drone cabin cover is provided above the groove of the drone platform. When the drone 10 is not in use, it is parked in the groove of the drone platform, and the drone cabin cover is closed to prevent the drone 10 from being affected by rain, snow and other environments. When the drone 10 is in use, the drone cabin cover is opened, and the drone 10 receives the instruction of the host control system 120 to take off from the drone platform, and collects the first image data and the second image data through the dual-channel image acquisition device 130 on the drone 10. In a specific embodiment, as Figure 7a As shown in FIG. 1 , when the UAV 10 flies 3 meters above the roof, the visible area of the dual-channel image acquisition device 130 includes most of the vehicle body. When the UAV 10 flies 10 meters above the roof, as shown in FIG. Figure 7b As shown, the visible area of the dual-channel image acquisition device 130 includes the entire vehicle body and a large area around the vehicle 300. At this time, the image data collected by the dual-channel image acquisition device 130 is transmitted to the host control system 120 in the vehicle cabin. The host control system 120 performs corresponding processing and then holographically displays it in the holographic image presentation device 140, so that the user can obtain a clear image of the vehicle 300 and its surroundings. The user can adjust the flight altitude of the drone 10 according to actual needs to obtain an image of the vehicle 300 and its surroundings that meets the needs, thereby obtaining a wider and more comprehensive image, greatly reducing the blind spots in the close-range area of the vehicle 300, improving the driving experience of the driver and passengers, and also improving the driving safety of the driver and passengers.

[0096] In one embodiment, the camera module is a TOF camera module 160, such as Figure 8a As shown, a TOF camera module 160, a holographic projection device 150, and a holographic image presentation device 140 (such as a holographic display device) are set in the cabin of the vehicle 300. The TOF camera module 160 is used to obtain TOF image data in the cabin of the vehicle 300, including the operation data of the holographic image 1510 of the driver and passenger on the drone operating system. Preferably, as Figure 8a 、 Figure 8b As shown, the TOF camera module 160 is arranged above the rearview mirror in the vehicle cabin so that the TOF camera module 160 can collect TOF image data in the vehicle cabin in a wide range as possible, that is, the field of view 1610 of the TOF camera module 160 is as large as possible. Figure 8bAs shown, the holographic projection device is set between the main driver and the front passenger seat and near the back seat, so that the holographic image 1510 of the drone operating system is presented in the back seat. The passengers in the back seat can move and scale the holographic image 1510 of the drone operating system according to their usage habits to facilitate their use. The holographic image presentation device 140 is set in the middle position of the center console of the vehicle 300 to facilitate the driver's observation and use during driving. To meet the needs of other passengers, the holographic projection device 150 and the holographic image presentation device 140 can also be installed in the front passenger seat and the back seat. This can meet the viewing needs of more people in the car for image data and improve the playability of the drone 10 in the vehicle.

[0097] Exemplarily, the vehicle further includes a voice system disposed in the vehicle cabin, the voice system being used to acquire voice information and send the voice information to the host control system, and the host control system generating drone control instructions based on the voice information to control the operation of the drone.

[0098] like Figure 9 、 Figure 10As shown, in a specific embodiment, the camera module is a TOF camera module 160, and the holographic projection device presents a holographic image 1510 of a drone operating system in the cabin of the vehicle 300. The driver and passengers can operate the holographic image 1510 of the drone operating system. The TOF camera module 160 collects the gestures of the driver and passengers in the vehicle 300 or the operation data of the driver and passengers on the drone operating system to form TOF image data, and sends the TOF image data to the host control system 120. The host control system 120 locates the gestures of the driver and passengers in the vehicle 300 through algorithm recognition, and also locates the holographic image 1510 of the drone operating system. Then, the algorithm recognizes and locates the driver and passengers. The host control system 120 uses an algorithm to identify the location of the gesture within the drone operating system's holographic image 1510, specifically the specific part of the drone operating system's holographic image 1510 the driver is operating, such as a button or joystick. The host control system 120 then generates drone control instructions, including dual-channel acquisition instructions, through an algorithmic location recognition algorithm. The drone control instructions and dual-channel acquisition instructions are then transmitted to the drone 10 via the communication system 110. The drone 10 executes the corresponding instructions and provides feedback to the host control system 120, including information such as the drone's position, speed, and image data. The host control system 120 receives the relevant information, performs information fusion, and transmits the processed data to the vehicle 300 for display or other applications in intelligent driving. Alternatively, the driver's voice information can be captured through the in-cabin voice system and transmitted to the host control system 120 via the communication system. The host control system 120 recognizes the voice information and generates drone control instructions, which are then transmitted to the drone 10. The drone 10 executes the corresponding instructions and provides feedback to the host control system 120, including information such as its position, speed, and image data. The communication system of the vehicle 300 includes but is not limited to the in-cabin controller area network bus (CAN) or the variable-speed controller area network bus (CAN with Flexible Data rate, CAN FD), 4G / 5G network, Wi-Fi, Bluetooth, and the positioning system of the vehicle 300. The communication system in the vehicle 300 is used for the opening and closing operations of the drone control system, and the network outside the vehicle 300 is used for remote control commands, remote transmission of image data, and positioning of the drone 10.The flight control operation of the drone 10 is performed through the holographic image 1510 of the voice system or the drone operating system to control the position of the drone 10 relative to the vehicle 300. The positioning devices and communication systems of the drone 10 and the vehicle 300 are combined to obtain the relationship between the coordinates of the drone 10 and the vehicle 300. Through trigonometric function calculations, data with smaller blind spots around the vehicle 300 can be obtained. The reconstructed three-dimensional scene can be used for intelligent driving functions such as Adas function, low-speed congested road condition detection and warning. In the actual application of the drone 10, the TOF image data collected by the dual-channel image acquisition device 130 and the TOF camera module 160 are updated in real time, and the host control system 120 also updates the relevant data it processes in real time, thereby providing the latest data for the algorithm application to obtain the latest intelligent driving data and present real-time images on the holographic image presentation device.

[0099] In an embodiment of the present invention, a dual-channel image acquisition device is provided in a drone, enabling the drone to simultaneously send image data in different image modes to a host control system. The host control system performs different processing based on the image data in different image modes to obtain image data that meets both user viewing requirements and algorithm requirements, thereby improving the user experience. Holographic projection equipment, a camera module, and a holographic image presentation device are simultaneously used to enable users to operate the drone operating system without physical objects and obtain clear holographic image displays. Vehicle and drone positioning and communication signal transmission are achieved through satellite systems and wireless communication devices between vehicles and drones, enabling real-time interaction between vehicle signals (such as vehicle speed and steering) and drone signals (such as speed and binocular camera video signals). Three-dimensional scene reconstruction and display on the holographic image presentation device reduce blind spots around the vehicle, improving driver and passenger safety. The system can also be used in areas such as intelligent driving collision warning. Depth information of the vehicle obtained through the camera module and the dual-channel image acquisition device enables deep integration of the drone and vehicle, enhancing the driving and entertainment experience of smart cockpits and intelligent driving.

[0100] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim listing several on-board systems, several of these on-board systems may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0101] The above is merely a description of specific embodiments of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A control system for an unmanned aerial vehicle, characterized in that: The control system includes: a communication system, a host control system, at least one holographic image presentation device, and at least one dual-channel image acquisition device; The dual-channel image acquisition device is arranged on the UAV; the dual-channel image acquisition device is used to acquire the first image data and / or the second image data, and send the first image data and / or the second image data to the host control system through the communication system; The host control system is used to reconstruct a three-dimensional scene based on the first image data; and is used to control the holographic image presentation device to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction.

2. The control system according to claim 1, wherein: The host control system is further configured to send a dual-channel acquisition instruction to the drone, so that the drone obtains the first image data and / or the second image data according to the dual-channel acquisition instruction.

3. The control system according to claim 1, wherein: The first image data is an infrared image, and the second image data is a color image and / or an infrared image.

4. The control system according to claim 3, wherein: The dual-channel image acquisition device is also used to send the first image data to the host control system in a first image output mode, and to send the automatically switched color image and / or the infrared image to the host control system in a second image output mode, wherein the color image and / or the infrared image are automatically switched according to a preset light intensity threshold, and to send the color image to the host control system in a third image output mode.

5. The control system according to claim 4, wherein: The host control system is further configured to control the dual-channel image acquisition device to be in the first image output mode and / or the second image output mode and / or the third image output mode.

6. The control system according to claim 1, wherein: The control system also includes at least one holographic projection device, which is used to present a holographic image of the drone operating system through holographic projection.

7. The control system according to claim 6, wherein: The control system further includes a camera module, which is used to obtain user operation data of the holographic image of the drone operating system and send the operation data to the host control system; The host control system is further configured to generate corresponding drone control instructions based on the operation data, and send the drone control instructions to the drone so that the drone executes the drone control instructions.

8. The control system according to claim 1, wherein: The control system also includes a voice system, which is used to obtain voice information provided by the user and send the voice information to the host control system. The host control system generates drone control instructions based on the voice information to control the operation of the drone.

9. A method for controlling a drone, characterized in that: The control method includes: The host control system sends a dual-channel acquisition instruction to the drone, so that the dual-channel image acquisition device of the drone acquires the first image data and / or the second image data; The host control system receives the first image data and / or the second image data sent by the dual-channel image acquisition device; The host control system reconstructs a three-dimensional scene based on the first image data, and controls the holographic image presentation device to holographically display the second image data and / or the image data after the three-dimensional scene reconstruction.

10. The control method according to claim 9, characterized in that: The first image data is an infrared image, and the second image is a color image and / or an infrared image.

11. The control method according to claim 10, wherein: The dual-channel acquisition instruction sent by the host control system controls the dual-channel image acquisition device to send the first image data to the host control system in the first image output mode, and to send the automatically switched color image and / or the infrared image to the host control system in the second image output mode, wherein the color image and / or the infrared image are automatically switched according to a preset light intensity threshold, and the color image is sent to the host control system in the third image output mode.

12. The control method according to claim 11, characterized in that: The control method further includes: the host control system controls the dual-channel image acquisition device to be in the first image output mode and / or the second image output mode and / or the third image output mode.

13. The control method according to claim 9, wherein: The control method further includes: The host control system controls the holographic projection device to present a holographic image of the drone operating system through holographic projection; The host control system sends an image acquisition instruction to the camera module to control the camera module to acquire image data of the user's operation on the holographic image of the drone operating system; The host control system obtains the user's operation on the holographic image of the drone operating system based on the image data, and sends a drone control instruction to the drone based on the operation, so that the drone executes the drone control instruction.

14. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the method for controlling a drone as described in any one of claims 9 to 13.

15. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program is used to execute the method for controlling a drone according to any one of claims 9 to 13.

16. A host control system, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the device equipped with the processor executes the method for controlling a drone as described in any one of claims 9 to 13.

17. A vehicle, characterized in that: The vehicle comprises: A drone provided on the vehicle and a control system for the drone according to any one of claims 1 to 8, or an electronic device according to claim 14, or a host control system according to claim 16; The vehicle is provided with a drone platform for parking the drone.

Images