Vehicle-mounted unmanned aerial vehicle control method and system, electronic equipment and storage medium

Abstract

The invention provides a vehicle-mounted unmanned aerial vehicle control method and system, electronic equipment and a storage medium, the method is applied to the vehicle-mounted unmanned aerial vehicle control system comprising vehicle-mounted equipment and VR equipment, and the method comprises the steps that the vehicle-mounted equipment determines a control mode of the vehicle-mounted unmanned aerial vehicle in response to acquired indication information of exploration or search and rescue of the vehicle-mounted unmanned aerial vehicle; in response to the determined attitude voice control mode, acquiring an attitude image and a voice instruction of a user by the VR equipment, and controlling the vehicle-mounted unmanned aerial vehicle to move by the vehicle-mounted equipment; in response to the determined automatic cruise mode, acquiring a cruise space range and a cruise height by the vehicle-mounted unmanned aerial vehicle equipment, and controlling the vehicle-mounted unmanned aerial vehicle to move based on the head action information of the user collected by the VR equipment; and in the moving process of the vehicle-mounted unmanned aerial vehicle, the vehicle-mounted equipment or the VR equipment displays the image acquired by the vehicle-mounted unmanned aerial vehicle. By means of the method, the control accuracy of the vehicle-mounted unmanned aerial vehicle is improved, and then the use experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle-mounted drones, and in particular to a vehicle-mounted drone control method, system, electronic device and storage medium. Background Art

[0002] Adding a vehicle-mounted drone to the current cockpit system can extend the cockpit's field of view to high altitudes. The drone plus the camera can feedback ground information from high altitudes, which can not only remind the driver of road conditions, but also allow users to experience the scenery from high altitudes. In addition, the vehicle-mounted drone can be used for exploration and search and rescue work in dangerous areas, which can not only bring a rich exploration experience, but also confirm the specific circumstances of various natural disasters and assist in search and rescue work.

[0003] At present, the conventional control method of vehicle-mounted drones generally requires users to control the take-off of the drone through the vehicle's large screen and control the drone through a handle. The images transmitted back by the drone are displayed on the vehicle's large screen. If the handle is not operated proficiently, there is a possibility of misoperation, which increases the risk of the vehicle-mounted drone crashing. In addition, the images transmitted by the vehicle-mounted drone are generally displayed on the vehicle's central control screen, and the user needs to look at the central control screen to operate the vehicle-mounted drone. If the viewing angle is incorrect, the user's control experience is affected. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, system, electronic device and storage medium for controlling a vehicle-mounted drone. According to the instruction information issued by the user on the need to use the vehicle-mounted drone for exploration or search and rescue, the control mode of the vehicle-mounted drone is set by the vehicle-mounted device. In the posture voice control mode, the VR device collects the user's posture image and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move. In the automatic cruise mode, based on the user's head movement information collected by the VR device, the vehicle-mounted drone is controlled to move at the cruising altitude within the cruising space. The vehicle-mounted device or VR device displays the image collected by the vehicle-mounted drone. Through the posture voice control of the VR device, the accuracy of the vehicle-mounted drone control is improved, so as to accurately identify the user's intention, thereby improving the user's experience of using the vehicle-mounted drone.

[0005] The embodiment of the present application provides a vehicle-mounted drone control method, the method is applied to a vehicle-mounted drone control system, the vehicle-mounted drone control system includes a vehicle machine device and a VR device, the method includes:

[0006] The vehicle-mounted device determines, in response to obtaining instruction information from a user indicating that the vehicle-mounted drone needs to be used for exploration or search and rescue, a control mode of the vehicle-mounted drone based on the instruction information; wherein the control mode includes a posture voice control mode, an automatic cruise mode, and a handle control mode;

[0007] In response to determining that the control mode is the posture voice control mode, the VR device collects the posture image of the user and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move based on the posture image and the voice command;

[0008] In response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains the cruise space range and the cruise altitude set by the user, and controls the vehicle-mounted drone to move within the cruise space range at the cruise altitude based on the user's head movement information collected by the VR device;

[0009] During the movement of the vehicle-mounted drone, the vehicle-mounted device or the VR device displays the images collected by the vehicle-mounted drone.

[0010] Furthermore, the vehicle-mounted drone control system further includes a remote control handle device, and the method further includes:

[0011] In response to determining that the control mode is the handle control mode, the remote control handle device obtains a control instruction issued by the user using the remote control handle device, and controls the vehicle-mounted drone to move based on the control instruction.

[0012] Further, in response to determining that the control mode is the posture voice control mode, the VR device collects the posture image of the user and the voice command issued by the user, and the vehicle-mounted drone is controlled to move based on the posture image and the voice command information by the vehicle-mounted device, including:

[0013] In response to determining that the control mode is the posture voice control mode, a posture monitoring module disposed in the VR device collects a posture image of the user, and a voice collection module disposed in the VR device collects a voice command issued by the user, so as to send the posture image and the voice command to the vehicle machine device;

[0014] The vehicle-mounted device recognizes head movement information or hand movement information in the received posture image, and recognizes movement instruction information in the received voice command;

[0015] Determine first control command information for controlling the vehicle-mounted drone based on the head motion information or the hand motion information and a motion combination corresponding to the motion instruction information;

[0016] Based on the first control command information, the vehicle-mounted drone is controlled to move.

[0017] Further, in response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains the cruise space range and the cruise altitude set by the user, and based on the user's head movement information collected by the VR device, controls the vehicle-mounted drone to move within the cruise space range at the cruise altitude, including:

[0018] In response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains a cruise space range and a cruise altitude set by a user, and determines a cruise path of the vehicle-mounted UAV by using a preset simulated annealing method based on the cruise space range and the cruise altitude;

[0019] Controlling the vehicle-mounted UAV to perform automatic cruising according to the cruising path;

[0020] In response to the head movement information of the user collected by the VR device during the automatic cruising, based on the head movement information, determining the second control command information for controlling the lens orientation of the vehicle-mounted drone;

[0021] Based on the second control command information, the lens of the vehicle-mounted drone is controlled to change its orientation during the automatic cruising process.

[0022] The present application also provides a vehicle-mounted drone control system, the system comprising:

[0023] A vehicle-mounted device, for, in response to obtaining indication information from a user indicating the need to use a vehicle-mounted drone for exploration or search and rescue, determining a control mode of the vehicle-mounted drone based on the indication information; wherein the control modes include a posture voice control mode, an automatic cruise mode, and a handle control mode; controlling the vehicle-mounted drone to move based on a posture image and a voice command; in response to determining that the control mode is the automatic cruise mode, obtaining a cruise space range and a cruise altitude set by the user, and controlling the vehicle-mounted drone to move within the cruise space range at the cruise altitude based on the user's head movement information collected by a VR device; and displaying images collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone;

[0024] The VR device is used for collecting the user's posture image and the voice command issued by the user in response to determining that the control mode is the posture voice control mode; collecting the user's head movement information in response to determining that the control mode is the automatic cruise mode; and displaying the image collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

[0025] Furthermore, the system also includes:

[0026] The remote control handle device is used to obtain a control instruction issued by a user using the remote control handle device in response to determining that the control mode is the handle control mode, and control the vehicle-mounted drone to move based on the control instruction.

[0027] Furthermore, the vehicle equipment includes:

[0028] An interactive setting module, for, in response to obtaining instruction information from a user indicating the need to use a vehicle-mounted drone for exploration or search and rescue, determining a control mode of the vehicle-mounted drone based on the instruction information; wherein the control mode includes a gesture voice control mode, an automatic cruise mode, and a handle control mode; in response to determining that the control mode is the automatic cruise mode, obtaining a cruise space range and a cruise altitude set by the user;

[0029] The drone control module is used to control the vehicle-mounted drone to move based on the posture image and voice command; based on the user's head movement information collected by the VR device, control the vehicle-mounted drone to move at the cruising altitude within the cruising space;

[0030] The first display module is used to display images collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

[0031] Furthermore, the VR device includes:

[0032] a posture monitoring module, for collecting a posture image of a user in response to determining that the control mode is a posture voice control mode; and collecting head movement information of a user in response to determining that the control mode is an automatic cruise mode;

[0033] a voice collection module, for collecting a voice command issued by a user in response to determining that the control mode is a gesture voice control mode;

[0034] The second display module is used to display the images collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

[0035] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned vehicle-mounted drone control method are performed.

[0036] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned vehicle-mounted drone control method are executed.

[0037] The vehicle-mounted drone control method, system, electronic device and storage medium provided in the embodiment of the present application are applied to a vehicle-mounted drone control system, wherein the vehicle-mounted drone control system includes a vehicle-mounted drone device and a VR device. The method includes: in response to obtaining indication information issued by a user that a vehicle-mounted drone needs to be used for exploration or search and rescue, the vehicle-mounted drone control mode is determined by the vehicle-mounted drone device based on the indication information; wherein the control mode includes a posture voice control mode, an automatic cruise mode and a handle control mode; in response to determining that the control mode is the posture voice control mode, the VR device collects a posture image of the user and a voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted drone device to move based on the posture image and the voice command; in response to determining that the control mode is the automatic cruise mode, the vehicle-mounted drone device obtains a cruising space range and a cruising altitude set by the user, and controls the vehicle-mounted drone to move within the cruising space range at the cruising altitude based on the head movement information of the user collected by the VR device; during the movement of the vehicle-mounted drone, the vehicle-mounted drone or the VR device displays the image collected by the vehicle-mounted drone.

[0038] Compared with the method in the prior art that requires the user to control the takeoff of the drone through the large screen of the vehicle computer and control the drone through the handle, and the image sent back by the drone is displayed on the large screen of the vehicle computer, the control mode of the vehicle-mounted drone is set through the vehicle computer equipment according to the instruction information issued by the user on the need to use the vehicle-mounted drone for exploration or search and rescue. In the posture voice control mode, the VR device collects the user's posture image and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle computer equipment to move. In the automatic cruise mode, based on the user's head movement information collected by the VR device, the vehicle-mounted drone is controlled to move at the cruising altitude within the cruising space. The vehicle computer device or VR device displays the image collected by the vehicle-mounted drone. The posture voice control of the VR device improves the accuracy of the control of the vehicle-mounted drone, so as to accurately identify the user's intention, thereby improving the user's experience of using the vehicle-mounted drone.

[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 One of the flow charts of a vehicle-mounted drone control method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the layout of a VR device provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of an automatic cruise of a vehicle-mounted drone provided in an embodiment of the present application;

[0044] Figure 4 The second flowchart of a vehicle-mounted drone control method provided in an embodiment of the present application;

[0045] Figure 5 This is one of the structural schematic diagrams of a vehicle-mounted drone control system provided in an embodiment of the present application;

[0046] Figure 6 The second structural diagram of a vehicle-mounted drone control system provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of a vehicle machine device provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a VR device provided in an embodiment of the present application;

[0049] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0051] Research has found that at present, the conventional control method of vehicle-mounted drones generally requires users to control the takeoff of the drone through the vehicle's large screen and control the drone through a handle. The images transmitted back by the drone are displayed on the vehicle's large screen. If the handle is not operated proficiently, there is a possibility of misoperation, increasing the risk of the vehicle-mounted drone crashing. In addition, the images transmitted by the vehicle-mounted drone are generally displayed on the vehicle's central control screen, and the user needs to look at the central control screen to operate the vehicle-mounted drone. If the viewing angle is incorrect, it will affect the user's control experience.

[0052] Based on this, an embodiment of the present application provides a method for controlling a vehicle-mounted drone. According to the instruction information issued by the user on the need to use the vehicle-mounted drone for exploration or search and rescue, the control mode of the vehicle-mounted drone is set by the vehicle-mounted device. In the posture voice control mode, the VR device collects the user's posture image and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move. In the automatic cruise mode, based on the user's head movement information collected by the VR device, the vehicle-mounted drone is controlled to move at the cruising altitude within the cruising space. The vehicle-mounted device or VR device displays the image collected by the vehicle-mounted drone. Through the posture voice control of the VR device, the accuracy of the vehicle-mounted drone control is improved, so as to accurately identify the user's intention, thereby improving the user's experience of using the vehicle-mounted drone.

[0053] See also Figure 1 , Figure 1 This is one of the flow charts of a vehicle-mounted drone control method provided in an embodiment of the present application. Figure 1 As shown in , the vehicle-mounted drone control method provided in the embodiment of the present application is applied to a vehicle-mounted drone control system, wherein the vehicle-mounted drone control system includes a vehicle machine device and a VR device, and the method includes:

[0054] S101. In response to obtaining instruction information from a user indicating the need to use a vehicle-mounted drone for exploration or search and rescue, the vehicle-mounted device determines a control mode of the vehicle-mounted drone based on the instruction information; wherein the control modes include a posture voice control mode, an automatic cruise mode, and a handle control mode.

[0055] It should be noted that in the scenario of the embodiment of the present application, the user is in the cockpit of a vehicle; the vehicle is equipped with a vehicle-mounted drone; the cockpit of the vehicle is equipped with vehicle-mounted equipment; and the VR device is set on a seat in the cockpit.

[0056] For example, in an adventure scenario, assuming that the vehicle enters a nature park, the driver and passengers can wear VR equipment and switch the vehicle to the panoramic camera system to view the surrounding natural scenery or ferocious animals. They can also start the vehicle's onboard drone and operate the drone through automatic cruising or autonomous control. They can use VR equipment to experience the natural scenery in the distance in an immersive way. They can also use infrared thermal imagers to detect the presence of biological species in the dense jungle, thereby determining the scope of exploration and operating the onboard drone to approach the corresponding area for detailed exploration.

[0057] In the example scenario of search and rescue, suppose a natural disaster occurs somewhere that makes it inconvenient for vehicles to travel, or the specific location of the missing person cannot be determined. Through the VR equipment worn by the driver and passengers, and by switching the vehicle to drone control and viewing mode, the on-board drone is controlled to explore a larger area. The drone camera can be used to roughly determine the location of the person or accident, and the infrared thermal imager can be used to determine whether there is life breath of people or other creatures, thereby improving the search and rescue scope and efficiency. At the same time, the same vehicle can be equipped with one or more on-board drones and their corresponding VR equipment to further increase the scope of search and rescue.

[0058] Among them, the vehicle-mounted drone establishes a wireless communication connection with the vehicle-mounted device; the vehicle-mounted device establishes a communication connection with the VR device, and the communication connection may include but is not limited to WIFI wireless connection and USB wired connection.

[0059] For details, please refer to Figure 2 , Figure 2 This is a schematic diagram of the layout of a VR device provided in an embodiment of the present application. Figure 2 As shown in , the VR device is arranged on the seat pillow; the signal line and power supply line of the VR device are arranged behind the seat back and connected to the vehicle equipment.

[0060] In this step, during the specific implementation, first, when the user needs to use the vehicle-mounted drone for actions including exploration or search and rescue, the user inputs the instruction information for setting the control mode of the vehicle-mounted drone through the interactive setting module of the vehicle-mounted device; then, the vehicle-mounted device responds to obtaining the instruction information and determines the control mode of the vehicle-mounted drone based on the control mode information corresponding to the instruction information; finally, the vehicle-mounted drone is set to the determined control mode.

[0061] In the embodiment of the present application, the vehicle-mounted drone is a drone device that can be carried by a vehicle and take off from the vehicle when needed.

[0062] Among them, the control modes include posture voice control mode, automatic cruise mode and handle control mode.

[0063] Here, the posture voice control mode refers to the control of posture behavior and voice commands by the user through the VR device, and the posture voice control command is sent to the vehicle-mounted drone by the vehicle-mounted device to achieve the technical effect of the user controlling the movement of the vehicle-mounted drone through the VR device; the automatic cruise mode refers to the mode in which the vehicle-mounted drone automatically cruises within the set range according to the set cruise parameters, including cruise parameters such as cruise space range and cruise altitude; the handle control mode refers to the mode in which the user controls the vehicle-mounted drone through the remote control handle configured for the vehicle-mounted drone.

[0064] S102: In response to determining that the control mode is the posture voice control mode, the VR device collects the posture image of the user and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move based on the posture image and the voice command.

[0065] In this step, after determining that the control mode of the vehicle-mounted drone is set to the posture voice control mode, first, the posture monitoring module and the voice acquisition module set in the VR device respectively collect the user's posture images and send them to the vehicle-mounted device; then, the vehicle-mounted device recognizes the user's head movement information or hand movement information, and the action indication information in the voice command; finally, based on the action combination corresponding to the head movement information or hand movement information and the action indication information, the first control command information for controlling the vehicle-mounted drone is determined to control the vehicle-mounted drone to move.

[0066] Among them, the posture image may include a head movement image or a hand movement image of the user; further, the vehicle-mounted device may recognize the head movement information or the hand movement information of the user.

[0067] In one implementation of the present application, during specific implementation, step S102 may include:

[0068] S1021. In response to determining that the control mode is the posture voice control mode, a posture monitoring module provided in the VR device collects a posture image of the user, and a voice collection module provided in the VR device collects a voice command issued by the user, so as to send the posture image and the voice command to the vehicle device.

[0069] In an embodiment of the present application, a posture monitoring module arranged in the VR device collects the posture image of the user; wherein, the posture monitoring module may include a driver monitoring system module (Driver Monitoring System, DMS) with a camera, and the DMS system module can be used to monitor the status of the driver and passengers to improve driving safety, and can detect the behavior and physiological state of the driver and passengers by using sensors such as cameras, such as attention level, fatigue level, and whether they are distracted; the voice collection module is provided with a microphone device.

[0070] Here, the posture image and voice command will be sent to the vehicle machine device through the communication connection established between the VR device and the vehicle machine device.

[0071] S1022: The vehicle-mounted device identifies head movement information or hand movement information in the received posture image, and identifies movement instruction information in the received voice command.

[0072] In this step, a preset posture recognition technology is used to identify head movement information or hand movement information in the posture image; wherein the posture recognition technology may include an open source platform framework based on Mediapipe, and support high-accuracy head movement, gesture and five-finger tracking.

[0073] Furthermore, a preset speech recognition model is used to identify action instruction information in the received voice command; wherein, the speech recognition model may include a Whisper model based on the OPENAI platform.

[0074] S1023: Determine first control command information for controlling the vehicle-mounted drone based on the action combination corresponding to the head action information or the hand action information and the action instruction information.

[0075] In the embodiment of the present application, the action combination can be specifically set according to the actual needs of the user and the actual application scenario of the vehicle-mounted drone to adapt to the user's behavioral habits and scenario requirements.

[0076] Exemplarily, in the action combination corresponding to the hand action information and the action instruction information, the first control command information for controlling the vehicle-mounted drone is determined as shown in the following table.

[0077]

[0078]

[0079] Exemplarily, in the action combination corresponding to the head action information and the action indication information, the first control command information for controlling the vehicle-mounted drone is determined as shown in the following table.

[0080] Head movement information Action instruction information First control command information Turn your head to the left Voice "left" Drone flying left Turn your head to the right Voice "right" Drone flying right Head facing forward and nodding Voice "Forward" Drone flying forward Head facing forward and shaking head Voice "backward" Drone flying backward Head facing forward and tilted up Voice "upward" Drone Ascends Head facing forward Voice "downward" Drone descent Keep your head facing forward and still Voice "stop" Drone hovering in the air

[0081] S1024: Based on the first control command information, control the vehicle-mounted drone to move.

[0082] In this step, based on the drone control method indicated by the first control command information, the vehicle-mounted drone is controlled to move by the vehicle-mounted drone through a wireless connection with the vehicle-mounted drone.

[0083] S103, in response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains the cruise space range and the cruise altitude set by the user, and based on the user's head movement information collected by the VR device, controls the vehicle-mounted drone to move within the cruise space range at the cruise altitude.

[0084] In an embodiment of the present application, after setting the control mode of the vehicle-mounted drone to the automatic cruise mode, the user can set the cruise space range and cruise altitude through the vehicle-mounted equipment, and can also set whether the vehicle-mounted drone should record video or infrared thermal imaging information.

[0085] Here, the vehicle-mounted drone will automatically avoid obstacles when in automatic cruise mode to avoid accidents. The obstacle avoidance mode uses commonly used ultrasonic or laser radar for identification.

[0086] For example, see Figure 3 , Figure 3 This is a schematic diagram of an automatic cruise of a vehicle-mounted drone provided in an embodiment of the present application. Figure 3 As shown in , when the vehicle-mounted drone starts to automatically cruise from the starting point in the cruising space range set by the user, if it encounters an obstacle, the vehicle-mounted drone will avoid the obstacle to obtain the actual cruising space range.

[0087] In one implementation of the present application, during specific implementation, step S103 may include:

[0088] S1031. In response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains the cruise space range and the cruise altitude set by the user, and based on the cruise space range and the cruise altitude, determines the cruise path of the vehicle-mounted drone using a preset simulated annealing method.

[0089] In an embodiment of the present application, the cruising path of the vehicle-mounted drone is based on the logic of a preset simulated annealing algorithm to achieve a better solution for the cruising route.

[0090] Among them, the principle of the simulated annealing algorithm is derived from the principle of solid annealing, that is, the solid is heated to a sufficiently high temperature and then allowed to cool. When the solid is heated, the particles inside the solid become disordered as the temperature rises, so that its internal energy increases. When the solid is cooled, the particles gradually become ordered and reach an equilibrium state at each temperature. Finally, it reaches the ground state at room temperature and the internal energy is reduced to a minimum.

[0091] Furthermore, according to the Metropolis criterion, the probability that a particle tends to equilibrium at temperature T is: e(-ΔE / (kT)), where E is the internal energy at temperature T; ΔE is its change; k is the Boltzmann constant; here, solid annealing is used to simulate the combinatorial optimization problem, the internal energy E is simulated as the objective function value f, and the temperature T evolves into the control parameter t, thus obtaining the simulated annealing algorithm for solving the combinatorial optimization problem.

[0092] Specifically, starting from the initial solution i and the initial value t of the control parameter, the iteration of "generate new solution → calculate the objective function difference → accept or discard" is repeated for the current solution, and the t value is gradually decayed. The current solution when the algorithm terminates is the approximate optimal solution. This is a heuristic random search process based on the Monte Carlo iterative solution method. The annealing process is controlled by the cooling schedule (CoolingSchedule), including the initial value t of the control parameter and its decay factor Δt, the number of iterations L at each t value, and the stopping condition S.

[0093] In this way, through the simulated annealing algorithm, a new cruise path can be randomly generated in the solution space, and whether to accept the new solution is determined based on the objective function (i.e., the total cruise distance). The algorithm accepts a solution that is worse than the current solution with a certain probability, thereby avoiding falling into the local optimum and ultimately approaching the global optimal solution.

[0094] S1032: Control the vehicle-mounted drone to perform automatic cruising along the cruising path.

[0095] S1033. In response to the head movement information of the user collected by the VR device during the automatic cruising process, determine the second control command information for controlling the lens orientation of the vehicle-mounted drone based on the head movement information.

[0096] In this step, when the vehicle-mounted drone is in the process of automatic cruising, the gyroscope and accelerometer built into the VR device can sense and collect the head movement information of the car owner, and then achieve the effect of controlling the direction of the drone's lens by actions such as looking up, lowering and turning the head.

[0097] Furthermore, based on the head movement information, second control command information for controlling the lens direction of the vehicle-mounted drone is determined; in addition, when the vehicle-mounted drone is in automatic cruise mode, the control mode can also be switched by pressing the function button of the VR device. When switching to the head control mode, the VR device will issue a voice reminder.

[0098] S1034: Based on the second control command information, control the lens of the vehicle-mounted drone to change its direction during the automatic cruising process.

[0099] In an embodiment of the present application, when the head movement information is looking up, the camera motor of the VR device continues to rotate slowly upward, and the head return motor of the VR device stops rotating. If the user keeps looking up, the motor of the VR device stops when it reaches the maximum rotation angle.

[0100] When the head movement information is lowering the head, the camera motor of the VR device continues to rotate slowly downward, and the head return motor of the VR device stops rotating. If the user keeps lowering his head, the motor of the VR device stops when it reaches the maximum rotation angle.

[0101] When the head movement information is turning left, the camera motor of the VR device continues to rotate slowly toward the left, and the head return motor of the VR device stops rotating. If the user keeps turning his head left, the motor of the VR device stops when it reaches the maximum rotation angle.

[0102] When the head movement information is turning right, the camera motor of the VR device continues to rotate slowly to the right, and the head return motor of the VR device stops rotating. If the user keeps turning his head right, the motor of the VR device stops when it reaches the maximum rotation angle.

[0103] S104: While the vehicle-mounted drone is moving, the vehicle-mounted device or the VR device displays the image captured by the vehicle-mounted drone.

[0104] In this step, the images collected by the vehicle-mounted drone during movement can be transmitted to the VR device through the vehicle equipment for display.

[0105] Specifically, the captured images are displayed on the VR device through the 3D technology corresponding to the preset binocular vision algorithm; wherein, the basic principle of the binocular vision algorithm is to shoot the same scene from different angles through two cameras to obtain two images, and calculate the depth information of the object based on the position deviation between the corresponding points in the two images to achieve a 3D display effect.

[0106] Furthermore, the VR device can also switch to the panoramic image corresponding to the vehicle-mounted device to display the image captured by the vehicle-mounted drone, and can also perform operations including switching control modes through the panoramic image corresponding to the vehicle-mounted device; in addition, the vehicle-mounted drone also supports a remote mode, that is, the vehicle-mounted device is transmitted to other VR devices located at the far end of the vehicle through the network for display.

[0107] Based on this, the present application exemplarily provides examples of vehicle-mounted drone control in two scenarios, including a search and rescue personnel scenario and a landscape exploration scenario.

[0108] As an example of vehicle-mounted drone control in a search and rescue scenario, first, the vehicle slowly drives in the search and rescue area, and the vehicle-mounted drone and VR device are started through the vehicle-mounted device; then, the control mode of the vehicle-mounted drone is confirmed through the vehicle-mounted device, and the VR device is selected to display the collected images, for example, infrared thermal imaging is recorded in the attitude voice control mode; after that, the images collected by the vehicle-mounted drone are transmitted back to the vehicle-mounted device in real time through the wireless module, and transmitted to the VR device via USB or WiFi, and the infrared thermal imaging is automatically recorded and stored in the vehicle-mounted device; finally, the real-time thermal imaging image can be opened through the vehicle-mounted device to assist fellow search and rescue personnel in making a preliminary judgment on the location of the personnel, and control the vehicle-mounted drone for further search until the location is confirmed for actual rescue.

[0109] As an example of vehicle-mounted drone control in a landscape exploration scenario, first, the vehicle is parked on a road at the edge of a natural landscape, and the vehicle-mounted drone and VR device are started through the vehicle-mounted equipment; then, the control mode of the vehicle-mounted drone is confirmed through the vehicle-mounted equipment, and the VR device is selected to display the collected images; after that, the gyroscope of the VR device is used to control the direction of travel of the vehicle-mounted drone, and the vehicle-mounted drone is set to automatic cruise mode, and the automatic cruise altitude and cruise space range of the drone are set, and infrared thermal imaging is turned off; finally, the image of the vehicle-mounted drone is transmitted back to the vehicle-mounted equipment in real time through the wireless module, and transmitted to the VR device through USB or WiFi, so that the surrounding landscape can be explored by rotating the head of the VR device.

[0110] Optional, see Figure 4 , Figure 4 This is a second flow chart of a vehicle-mounted drone control method provided in an embodiment of the present application. Figure 4As shown in, in addition to the vehicle-mounted drone control method described in steps S101 to S104, the embodiment of the present application also includes step S105. Specifically, step S105 is used to illustrate the vehicle-mounted drone control method when the control mode of the vehicle-mounted drone is the handle control mode, so as to improve the comprehensiveness of the vehicle-mounted drone control.

[0111] S105: In response to determining that the control mode is the handle control mode, the remote control handle device obtains a control instruction issued by the user using the remote control handle device, and controls the vehicle-mounted drone to move based on the control instruction.

[0112] Wherein, the vehicle-mounted drone control system also includes a remote control handle device.

[0113] In this step, the user sends a control instruction to the vehicle-mounted drone through the remote control handle device configured for the vehicle-mounted drone, so as to control the vehicle-mounted drone to move according to the control instruction.

[0114] The vehicle-mounted drone control method provided in the embodiment of the present application sets the control mode of the vehicle-mounted drone through the vehicle-mounted device according to the instruction information issued by the user on the need to use the vehicle-mounted drone for exploration or search and rescue. In the posture voice control mode, the VR device collects the user's posture image and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move. In the automatic cruise mode, based on the user's head movement information collected by the VR device, the vehicle-mounted drone is controlled to move at the cruising altitude within the cruising space. The vehicle-mounted device or the VR device displays the image collected by the vehicle-mounted drone. The posture voice control of the VR device improves the accuracy of the vehicle-mounted drone control, so as to accurately identify the user's intention, thereby improving the user's experience of using the vehicle-mounted drone.

[0115] See also Figure 5 , Figure 6 , Figure 5 This is one of the structural schematic diagrams of a vehicle-mounted drone control system provided in an embodiment of the present application. Figure 6 This is a second structural diagram of a vehicle-mounted drone control system provided in an embodiment of the present application. Figure 5 As shown in , the vehicle-mounted drone control system 10 includes:

[0116] The vehicle-mounted device 110 is used to, in response to obtaining the indication information issued by the user that the vehicle-mounted drone needs to be used for exploration or search and rescue, determine the control mode of the vehicle-mounted drone based on the indication information; wherein the control mode includes a posture voice control mode, an automatic cruise mode and a handle control mode; control the vehicle-mounted drone to move based on the posture image and the voice command; in response to determining that the control mode is the automatic cruise mode, obtain the cruise space range and the cruise altitude set by the user, and control the vehicle-mounted drone to move within the cruise space range at the cruise altitude based on the user's head movement information collected by the VR device; during the movement of the vehicle-mounted drone, display the image collected by the vehicle-mounted drone;

[0117] The VR device 120 is used for collecting the user's posture image and the voice command issued by the user in response to determining that the control mode is the posture voice control mode; collecting the user's head movement information in response to determining that the control mode is the automatic cruise mode; and displaying the image collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

[0118] Further, such as Figure 6 As shown in , the vehicle-mounted drone control system 10 also includes:

[0119] The remote control handle device 130 is used to obtain a control instruction issued by the user using the remote control handle device in response to determining that the control mode is the handle control mode, and control the vehicle-mounted drone to move based on the control instruction.

[0120] For further information, see Figure 7 , Figure 7 This is a schematic diagram of the structure of a vehicle equipment provided by an embodiment of the present application. Figure 7 As shown in , the vehicle machine device 110 includes:

[0121] The interactive setting module 111 is used to, in response to obtaining the instruction information issued by the user that the vehicle-mounted drone needs to be used for exploration or search and rescue, determine the control mode of the vehicle-mounted drone based on the instruction information; wherein the control mode includes a posture voice control mode, an automatic cruise mode and a handle control mode; in response to determining that the control mode is the automatic cruise mode, obtain the cruise space range and cruise altitude set by the user;

[0122] The drone control module 112 is used to control the vehicle-mounted drone to move based on the posture image and the voice command; based on the user's head movement information collected by the VR device, control the vehicle-mounted drone to move at the cruising altitude within the cruising space;

[0123] The first display module 113 is used to display the images collected by the vehicle-mounted drone when the vehicle-mounted drone is in motion.

[0124] For further information, see Figure 8 , Figure 8 This is a schematic diagram of the structure of a VR device provided in an embodiment of the present application. Figure 8 As shown in , the VR device 120 includes:

[0125] The posture monitoring module 121 is used for collecting the posture image of the user in response to determining that the control mode is the posture voice control mode; and collecting the head movement information of the user in response to determining that the control mode is the automatic cruise mode;

[0126] A voice collection module 122, for collecting a voice command issued by a user in response to determining that the control mode is a gesture voice control mode;

[0127] The second display module 123 is used to display the images collected by the vehicle-mounted drone when the vehicle-mounted drone is in motion.

[0128] The vehicle-mounted drone control system provided in the embodiment of the present application sets the control mode of the vehicle-mounted drone through the vehicle-mounted device according to the instruction information issued by the user on the need to use the vehicle-mounted drone for exploration or search and rescue. In the posture voice control mode, the VR device collects the user's posture image and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move. In the automatic cruise mode, based on the user's head movement information collected by the VR device, the vehicle-mounted drone is controlled to move at the cruising altitude within the cruising space. The vehicle-mounted device or VR device displays the image collected by the vehicle-mounted drone. The posture voice control of the VR device improves the accuracy of the vehicle-mounted drone control, so as to accurately identify the user's intention, thereby improving the user's experience of using the vehicle-mounted drone.

[0129] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown in , the electronic device 900 includes a processor 910 , a memory 920 and a bus 930 .

[0130] The memory 920 stores machine-readable instructions executable by the processor 910. When the electronic device 900 is running, the processor 910 communicates with the memory 920 via the bus 930. When the machine-readable instructions are executed by the processor 910, the above-mentioned Figure 1 as well as Figure 4The specific implementation of the steps of the vehicle-mounted drone control method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0131] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 4 The specific implementation of the steps of the vehicle-mounted drone control method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0137] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A vehicle-mounted drone control method, characterized in that: The method is applied to a vehicle-mounted UAV control system, wherein the vehicle-mounted UAV control system includes a vehicle-mounted device and a VR device, and the method includes: The vehicle-mounted device determines, in response to obtaining instruction information from a user indicating that the vehicle-mounted drone needs to be used for exploration or search and rescue, a control mode of the vehicle-mounted drone based on the instruction information; wherein the control mode includes a posture voice control mode, an automatic cruise mode, and a handle control mode; In response to determining that the control mode is the posture voice control mode, the VR device collects the posture image of the user and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move based on the posture image and the voice command; In response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains the cruise space range and the cruise altitude set by the user, and controls the vehicle-mounted drone to move within the cruise space range at the cruise altitude based on the user's head movement information collected by the VR device; During the movement of the vehicle-mounted drone, the vehicle-mounted device or the VR device displays the images collected by the vehicle-mounted drone.

2. The method according to claim 1, characterized in that The vehicle-mounted drone control system further includes a remote control handle device, and the method further includes: In response to determining that the control mode is the handle control mode, the remote control handle device obtains a control instruction issued by the user using the remote control handle device, and controls the vehicle-mounted drone to move based on the control instruction.

3. The method according to claim 1, characterized in that In response to determining that the control mode is the posture voice control mode, the VR device collects the posture image of the user and the voice command issued by the user, and the vehicle-mounted drone is controlled by the vehicle-mounted device to move based on the posture image and the voice command, including: In response to determining that the control mode is the posture voice control mode, a posture monitoring module disposed in the VR device collects a posture image of the user, and a voice collection module disposed in the VR device collects a voice command issued by the user, so as to send the posture image and the voice command to the vehicle machine device; The vehicle-mounted device recognizes head movement information or hand movement information in the received posture image, and recognizes movement instruction information in the received voice command; Determine first control command information for controlling the vehicle-mounted drone based on the head motion information or the hand motion information and a motion combination corresponding to the motion instruction information; Based on the first control command information, the vehicle-mounted drone is controlled to move.

4. The method according to claim 1, characterized in that: In response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains the cruise space range and the cruise altitude set by the user, and based on the user's head movement information collected by the VR device, controls the vehicle-mounted drone to move within the cruise space range at the cruise altitude, including: In response to determining that the control mode is the automatic cruise mode, the vehicle-mounted device obtains a cruise space range and a cruise altitude set by a user, and determines a cruise path of the vehicle-mounted UAV by using a preset simulated annealing method based on the cruise space range and the cruise altitude; Controlling the vehicle-mounted UAV to perform automatic cruising according to the cruising path; In response to the head movement information of the user collected by the VR device during the automatic cruising, based on the head movement information, determining the second control command information for controlling the lens orientation of the vehicle-mounted drone; Based on the second control command information, the lens of the vehicle-mounted drone is controlled to change its orientation during the automatic cruising process.

5. A vehicle-mounted drone control system, characterized in that: The system comprises: A vehicle-mounted device, for, in response to obtaining indication information from a user indicating the need to use a vehicle-mounted drone for exploration or search and rescue, determining a control mode of the vehicle-mounted drone based on the indication information; wherein the control modes include a posture voice control mode, an automatic cruise mode, and a handle control mode; controlling the vehicle-mounted drone to move based on a posture image and a voice command; in response to determining that the control mode is the automatic cruise mode, obtaining a cruise space range and a cruise altitude set by the user, and controlling the vehicle-mounted drone to move within the cruise space range at the cruise altitude based on the user's head movement information collected by a VR device; and displaying images collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone; The VR device is used for collecting the user's posture image and the voice command issued by the user in response to determining that the control mode is the posture voice control mode; collecting the user's head movement information in response to determining that the control mode is the automatic cruise mode; and displaying the image collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

6. The system according to claim 5, characterized in that The system further comprises: The remote control handle device is used to obtain a control instruction issued by a user using the remote control handle device in response to determining that the control mode is the handle control mode, and control the vehicle-mounted drone to move based on the control instruction.

7. The system according to claim 5, characterized in that The vehicle equipment includes: An interactive setting module, for, in response to obtaining instruction information from a user indicating the need to use a vehicle-mounted drone for exploration or search and rescue, determining a control mode of the vehicle-mounted drone based on the instruction information; wherein the control mode includes a gesture voice control mode, an automatic cruise mode, and a handle control mode; in response to determining that the control mode is the automatic cruise mode, obtaining a cruise space range and a cruise altitude set by the user; The drone control module is used to control the vehicle-mounted drone to move based on the posture image and voice command; based on the user's head movement information collected by the VR device, control the vehicle-mounted drone to move at the cruising altitude within the cruising space; The first display module is used to display images collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

8. The system according to claim 5, characterized in that The VR device includes: a posture monitoring module, for collecting a posture image of a user in response to determining that the control mode is a posture voice control mode; and collecting head movement information of the user in response to determining that the control mode is an automatic cruise mode; a voice collection module, for collecting a voice command issued by a user in response to determining that the control mode is a gesture voice control mode; The second display module is used to display the images collected by the vehicle-mounted drone during the movement of the vehicle-mounted drone.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the vehicle-mounted drone control method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the vehicle-mounted drone control method according to any one of claims 1 to 4 are executed.

Images