Unmanned aerial vehicle automatic driving method and system, medium and equipment
Through multimodal information collection and encoding, the world model and brain-like neural circuit network are used to generate autonomous driving control information, which solves the problems of poor migration and relying on artificial engineering in the existing technology, and realizes the efficiency of drones in complex scenarios Autonomous driving and adaptability.
Patent Information
- Application Number
- CN202411002574.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-08
AI Technical Summary
The existing autonomous driving model relies on artificial engineering, has poor migration, is difficult to adapt to new environments and new tasks, and the modular method is cumbersome, making it difficult to achieve the development requirements of general artificial intelligence.
Multimodal information acquisition and encoding are adopted, driving environment features are extracted through deep residual networks and convolutional neural networks, and input them into world models and brain-like neural circuit networks to generate autonomous driving control information to realize autonomous driving control of drones.
It improves the migration and adaptability of drones, reduces dependence on artificial engineering, can make reasonable decisions and actions in complex scenarios, improves the safety and reliability of autonomous driving, and adapts to the development needs of artificial intelligence.
Smart Images

Figure CN118915787A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to an unmanned aerial vehicle autonomous driving method and system, medium, and equipment. Background Art
[0002] At present, the modular approach is widely used in autonomous driving models, but this approach has many disadvantages. It is highly dependent on manual engineering and involves tedious work such as labeling individual modules and configuring cross-modules. In the face of new environments or new tasks, this approach requires manual redesign and algorithm upgrades, resulting in poor transferability and difficulty in adapting to the development requirements of general artificial intelligence. Summary of the invention
[0003] In response to at least one of the above technical problems, an embodiment of the present invention provides a method and system, medium, and device for automatic driving of a drone.
[0004] According to a first aspect, a human-machine automatic driving method provided by an embodiment of the present invention includes:
[0005] Collect multi-modal driving environment information;
[0006] Encoding the driving environment information to obtain multimodal input features suitable for a world model;
[0007] Inputting the multimodal input features into the world model to obtain driving environment dynamics information;
[0008] Inputting the driving environment dynamics information into the brain-like neural circuit network to obtain the UAV automatic driving control information;
[0009] The drone is automatically controlled according to the drone automatic driving control information.
[0010] In one embodiment, the collecting of multi-modal driving environment information includes:
[0011] Collect image data of the driving environment through the drone camera;
[0012] Wind speed data and sound wave data in the driving environment are collected through a variety of sensors.
[0013] In one embodiment, encoding the driving environment information to obtain multimodal input features suitable for a world model includes:
[0014] Extracting driving environment features from the image data through a deep residual network, and encoding the driving environment features to obtain image features corresponding to the driving environment;
[0015] Encoding the wind speed data to obtain wind speed characteristics of the driving environment;
[0016] Encoding the sound wave data to obtain the sound wave characteristics of the driving environment;
[0017] The image feature, the wind speed feature and the sound wave feature are combined to form the multimodal input feature.
[0018] In one embodiment, the method further comprises:
[0019] Inputting the driving environment dynamics information into a pre-trained convolutional neural network prediction model to obtain multi-modal driving environment prediction information within a preset time in the future;
[0020] The automatic driving process of the UAV is adjusted according to the driving environment prediction information.
[0021] In one embodiment, the method further comprises:
[0022] Inputting the driving environment dynamics information into a pre-trained convolutional neural network detection model to obtain a detection result of whether a preset target exists in the driving environment;
[0023] According to the detection result, it is known that the preset target exists, and the drone is controlled to perform a preset action during the automatic driving process; wherein the preset action is an action taken for the preset target.
[0024] In one embodiment, the brain-like neural circuit network is a network model established by pre-simulating the Drosophila nervous system.
[0025] In one embodiment, the world model is specifically a model obtained by fitting the environmental dynamics posterior distribution information and the environmental dynamics prior information through the multimodal input features and the driving environment dynamics information, and training with minimizing the gap between the environmental dynamics posterior distribution information and the environmental dynamics prior information as the training goal.
[0026] According to a second aspect, an unmanned aerial vehicle automatic driving system provided by an embodiment of the present invention includes:
[0027] An information collection module, used to collect multi-modal driving environment information;
[0028] An information encoding module, used for encoding the driving environment information to obtain multimodal input features suitable for a world model;
[0029] A first input module, used for inputting the multimodal input features into the world model to obtain driving environment dynamics information;
[0030] The second input module is used to input the driving environment dynamics information into the brain-like neural circuit network to obtain the UAV automatic driving control information;
[0031] The driving control module is used to control the drone automatically according to the drone automatic driving control information.
[0032] In one embodiment, the information acquisition module is specifically used to: collect image data of the driving environment through a drone camera; and collect wind speed data and sound wave data in the driving environment through a variety of sensors.
[0033] In one embodiment, the information encoding module is specifically used to: extract driving environment features from the image data through a deep residual network, and encode the driving environment features to obtain image features corresponding to the driving environment; encode the wind speed data to obtain wind speed features of the driving environment; encode the sound wave data to obtain sound wave features of the driving environment; and combine the image features, the wind speed features, and the sound wave features to form the multimodal input features.
[0034] In one embodiment, the system may further include:
[0035] The environment prediction module is used to: input the driving environment dynamics information into a pre-trained convolutional neural network prediction model to obtain multimodal driving environment prediction information within a preset time in the future; and adjust the automatic driving process of the drone according to the driving environment prediction information.
[0036] In one embodiment, the system may further include:
[0037] The target detection module is used to: input the driving environment dynamics information into a pre-trained convolutional neural network detection model to obtain a detection result of whether a preset target exists in the driving environment; based on the detection result, it is known that the preset target exists, and control the drone to perform a preset action during the automatic driving process; wherein the preset action is an action taken for the preset target.
[0038] In one embodiment, the brain-like neural circuit network is a network model established by pre-simulating the Drosophila nervous system.
[0039] In one embodiment, the world model is specifically a model obtained by fitting the environmental dynamics posterior distribution information and the environmental dynamics prior information through the multimodal input features and the driving environment dynamics information, and training with minimizing the gap between the environmental dynamics posterior distribution information and the environmental dynamics prior information as the training goal.
[0040] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.
[0041] According to a fourth aspect, a computing device provided by an embodiment of the present invention includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided by the first aspect is implemented.
[0042] The unmanned aerial vehicle automatic driving method, system, medium, and device provided in the embodiment of the present invention collect multimodal driving environment information, encode the driving environment information to obtain multimodal input features suitable for the world model, and then input the multimodal input features into the world model to obtain driving environment dynamics information, and then input the driving environment dynamics information into the brain-like neural circuit network to obtain unmanned aerial vehicle automatic driving control information, and finally perform automatic driving control on the unmanned aerial vehicle according to the unmanned aerial vehicle automatic driving control information. In the field of automatic driving, the world model can better cope with various complex scenarios by modeling the characteristics of the real environment, thereby helping the unmanned aerial vehicle to make reasonable decisions and actions, which can be seen to have brought new breakthroughs and possibilities to the field of automatic driving. Since the embodiment of the present invention does not adopt a modular method, it is not highly dependent on artificial engineering. When facing new tasks or new environments, the unmanned aerial vehicle can also adapt flexibly. Therefore, the method provided in the embodiment of the present invention has high mobility and can adapt to the development needs of artificial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A schematic diagram of a flow chart of an automatic driving method for a drone in one embodiment of the present invention;
[0044] Figure 2 1 is a structural block diagram of a drone autopilot system in one embodiment of the present invention. DETAILED DESCRIPTION
[0045] In a first aspect, an embodiment of the present invention provides a method for autonomous driving of a drone, see Figure 1 The method comprises the following steps S110 to S150:
[0046] S110, collecting multi-modal driving environment information;
[0047] In one embodiment, the collecting of multi-modal driving environment information in S110 may include:
[0048] Collect image data of the driving environment through the drone camera;
[0049] Wind speed data and sound wave data in the driving environment are collected through a variety of sensors.
[0050] It can be seen that the image data of the drone's driving environment can be collected through the drone camera, and the wind speed data and sound wave data in the driving environment can be collected through the sensor.
[0051] That is, the multimodal driving environment information includes image data, wind speed data, and sound wave data of the driving environment.
[0052] S120, encoding the driving environment information to obtain multimodal input features suitable for a world model;
[0053] In one embodiment, the encoding of the driving environment information in S120 to obtain multimodal input features suitable for a world model includes S121 to S124:
[0054] S121, extracting driving environment features from the image data through a deep residual network, and encoding the driving environment features to obtain image features corresponding to the driving environment;
[0055] That is, the deep residual network is used to extract features from the image data, and then the extracted driving environment features are encoded to obtain image features.
[0056] S122, encoding the wind speed data to obtain wind speed characteristics of the driving environment;
[0057] S123, encoding the sound wave data to obtain the sound wave characteristics of the driving environment;
[0058] S124. Combining the image feature, the wind speed feature, and the sound wave feature to form the multimodal input feature.
[0059] It can be seen that the multimodal input features include image features, wind speed features and sound wave features. The multimodal input features are suitable for the world model, which facilitates the subsequent input of the multimodal input features into the world model.
[0060] S130, inputting the multimodal input features into the world model to obtain driving environment dynamics information;
[0061] Among them, the world model is an emerging concept in the field of artificial intelligence. It provides us with a new way to understand and simulate the real world. The model is mainly built by computers or artificial intelligence systems to simulate and express the real world. It is a comprehensive and integrated method to describe the environment. Through the processing of perceptual information and data modeling, it can achieve accurate abstraction and simulation of objects, scenes, actions and other elements.
[0062] The realization of the world model mainly relies on the steps of perception data processing, data modeling and learning, and testing and simulation. The world model is used to convert the above multimodal input features into understandable information about the driving environment dynamics. These data can include images, wind speed, sound waves and other forms. Through the processing and analysis of data, the world model can better understand the entities and events in the environment.
[0063] Among them, the driving environment dynamics information can also be understood as effective traffic information of the driving environment.
[0064] S140, inputting the driving environment dynamics information into the brain-like neural circuit network to obtain the drone automatic driving control information;
[0065] It can be seen that after inputting the driving environment dynamics information into the brain-like neural circuit network, the drone automatic driving control information can be obtained.
[0066] The brain-like neural circuit network can be a network model established by simulating the Drosophila nervous system in advance, that is, the brain-like neural circuit network is a network established by simulating the Drosophila neural network.
[0067] One of the most studied networks in neuroscience is the brain of the fruit fly, especially the part called the mushroom body. This part can analyze sensory inputs such as odor, temperature, humidity, and visual data so that the fruit fly can learn to distinguish between friendly and dangerous stimuli. Many neural circuits in its nervous system are modeled as a four-layer hybrid topology. The embodiments of the present invention imitate its neural circuits to establish a brain-like neural circuit network.
[0068] S150: Perform automatic driving control on the drone according to the automatic driving control information of the drone.
[0069] That is, after obtaining the drone's automatic driving control information, the drone's automatic driving can be controlled according to the drone's automatic driving control information.
[0070] In one embodiment, the method provided by the embodiment of the present invention may further include:
[0071] Inputting the driving environment dynamics information into a pre-trained convolutional neural network prediction model to obtain multi-modal driving environment prediction information within a preset time in the future;
[0072] The automatic driving process of the UAV is adjusted according to the driving environment prediction information.
[0073] That is to say, the driving environment dynamics information is input into the convolutional neural network prediction model, and the role of the convolutional neural network prediction model is to predict the multimodal driving environment information in the future time period, such as the two-dimensional image of the driving environment, wind speed, sound waves and other information. After obtaining the driving environment prediction information, the automatic driving process of the UAV can be appropriately adjusted to ensure that the flight trend of the UAV under the current automatic driving control meets the driving environment in the future time period.
[0074] In one embodiment, the method provided by the embodiment of the present invention may further include:
[0075] Inputting the driving environment dynamics information into a pre-trained convolutional neural network detection model to obtain a detection result of whether a preset target exists in the driving environment;
[0076] According to the detection result, it is known that the preset target exists, and the drone is controlled to perform a preset action during the automatic driving process; wherein the preset action is an action taken for the preset target.
[0077] That is to say, the driving environment dynamics information is input into the convolutional neural network detection model, and the function of the convolutional neural network detection model is to detect whether there is a preset target in the driving environment. After obtaining the detection result, if the detection result is that there is a preset target, the drone is controlled to perform a preset action, such as irradiating the preset target with a laser, spraying liquid on the preset target, attacking the preset target, etc.
[0078] In one embodiment, the world model is specifically a model obtained by fitting the environmental dynamics posterior distribution information and the environmental dynamics prior information through the multimodal input features and the driving environment dynamics information, and training with minimizing the gap between the environmental dynamics posterior distribution information and the environmental dynamics prior information as the training goal.
[0079] It can be seen that the training goal of the world model during the training process is to minimize the gap between the posterior distribution information of the environmental dynamics and the prior information of the environmental dynamics. The posterior distribution information of the environmental dynamics and the prior information of the environmental dynamics are obtained by fitting the multimodal input features and the driving environment dynamics information. The posterior distribution information of the environmental dynamics is the probability distribution calculated after the result is obtained, and the prior information of the environmental dynamics is the probability distribution calculated before the result is obtained. By narrowing the gap between the two, the calculation of the result can be made more and more accurate.
[0080] It can be seen that the embodiment of the present invention uses drone cameras and sensors to obtain multi-modal driving environment information, and obtains effective traffic information of the driving environment based on the generative world model. The fruit fly neural network is simulated to establish a brain-like neural circuit network, and the effective traffic information of the driving environment is input into the brain-like neural circuit network to obtain the control output of the autonomous driving.
[0081] The embodiment of the present invention can collect and process a large amount of driving environment data in real time during the execution of tasks by the unmanned aerial vehicle automatic driving system based on the world model. These data can not only be used to improve and optimize system performance, but also provide valuable references for research and applications in other fields. Moreover, since the world model can simulate various environments and scenarios, the unmanned aerial vehicle automatic driving method based on the world model can better adapt to the mission requirements of different scenarios. Whether it is cargo transportation in an urban environment or exploration missions in remote areas, the embodiment of the present invention can respond flexibly and show strong adaptability. In addition, the world model simulates the real environment, allowing the unmanned aerial vehicle to more accurately perceive and predict the surrounding situation. This capability enables the unmanned aerial vehicle to make reasonable decisions and actions in a complex and changing environment, thereby greatly improving its safety and reliability.
[0082] In the field of autonomous driving, the world model provides more accurate perception and prediction capabilities for autonomous driving control. By modeling the characteristics of the real environment, it is possible to better cope with various complex scenarios and make reasonable decisions and actions. The application of this method has brought new breakthroughs and possibilities to the field of autonomous driving. Since the embodiment of the present invention does not adopt a modular approach, it is not highly dependent on artificial engineering. When faced with new tasks or new environments, the drone can also adapt flexibly. Therefore, the method provided by the embodiment of the present invention has high portability and can adapt to the development needs of artificial intelligence.
[0083] In a second aspect, an embodiment of the present invention provides an unmanned aerial vehicle automatic driving system, see Figure 2 , the system 100 comprises:
[0084] An information collection module 110, for collecting multi-modal driving environment information;
[0085] An information encoding module 120, configured to encode the driving environment information to obtain multimodal input features suitable for a world model;
[0086] A first input module 130, configured to input the multimodal input features into the world model to obtain driving environment dynamics information;
[0087] The second input module 140 is used to input the driving environment dynamics information into the brain-like neural circuit network to obtain the drone automatic driving control information;
[0088] The driving control module 150 is used to perform automatic driving control on the drone according to the automatic driving control information of the drone.
[0089] In one embodiment, the information acquisition module is specifically used to: collect image data of the driving environment through a drone camera; and collect wind speed data and sound wave data in the driving environment through a variety of sensors.
[0090] In one embodiment, the information encoding module is specifically used to: extract driving environment features from the image data through a deep residual network, and encode the driving environment features to obtain image features corresponding to the driving environment; encode the wind speed data to obtain wind speed features of the driving environment; encode the sound wave data to obtain sound wave features of the driving environment; and combine the image features, the wind speed features, and the sound wave features to form the multimodal input features.
[0091] In one embodiment, the system may further include:
[0092] The environment prediction module is used to: input the driving environment dynamics information into a pre-trained convolutional neural network prediction model to obtain multimodal driving environment prediction information within a preset time in the future; and adjust the automatic driving process of the drone according to the driving environment prediction information.
[0093] In one embodiment, the system may further include:
[0094] The target detection module is used to: input the driving environment dynamics information into a pre-trained convolutional neural network detection model to obtain a detection result of whether a preset target exists in the driving environment; based on the detection result, it is known that the preset target exists, and control the drone to perform a preset action during the automatic driving process; wherein the preset action is an action taken for the preset target.
[0095] In one embodiment, the brain-like neural circuit network is a network model established by pre-simulating the Drosophila nervous system.
[0096] In one embodiment, the world model is specifically a model obtained by fitting the environmental dynamics posterior distribution information and the environmental dynamics prior information through the multimodal input features and the driving environment dynamics information, and training with minimizing the gap between the environmental dynamics posterior distribution information and the environmental dynamics prior information as the training goal.
[0097] It is understandable that the explanation of the relevant contents, specific implementation methods, beneficial effects, examples, etc. in the system provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.
[0098] In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.
[0099] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.
[0100] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0101] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.
[0102] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0103] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0104] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.
[0105] In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any embodiment of the specification.
[0106] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.
[0107] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0108] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0109] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for automatic driving of a drone, characterized in that: include: Collect multi-modal driving environment information; Encoding the driving environment information to obtain multimodal input features suitable for a world model; Inputting the multimodal input features into the world model to obtain driving environment dynamics information; Inputting the driving environment dynamics information into the brain-like neural circuit network to obtain the UAV automatic driving control information; The drone is automatically controlled according to the drone automatic driving control information.
2. The method according to claim 1, characterized in that The collecting of multi-modal driving environment information includes: Collect image data of the driving environment through the drone camera; Wind speed data and sound wave data in the driving environment are collected through a variety of sensors.
3. The method according to claim 2, characterized in that The encoding of the driving environment information to obtain multimodal input features suitable for a world model includes: Extracting driving environment features from the image data through a deep residual network, and encoding the driving environment features to obtain image features corresponding to the driving environment; Encoding the wind speed data to obtain wind speed characteristics of the driving environment; Encoding the sound wave data to obtain the sound wave characteristics of the driving environment; The image feature, the wind speed feature and the sound wave feature are combined to form the multimodal input feature.
4. The method according to claim 1, characterized in that: Also includes: Inputting the driving environment dynamics information into a pre-trained convolutional neural network prediction model to obtain multi-modal driving environment prediction information within a preset time in the future; The automatic driving process of the UAV is adjusted according to the driving environment prediction information.
5. The method according to claim 1, characterized in that: Also includes: Inputting the driving environment dynamics information into a pre-trained convolutional neural network detection model to obtain a detection result of whether a preset target exists in the driving environment; According to the detection result, it is known that the preset target exists, and the drone is controlled to perform a preset action during the automatic driving process; wherein the preset action is an action taken for the preset target.
6. The method according to claim 1, characterized in that The brain-like neural circuit network is a network model established by pre-simulating the Drosophila nervous system.
7. The method according to claim 1, characterized in that The world model is specifically a model obtained by fitting the environmental dynamics posterior distribution information and the environmental dynamics prior information through the multimodal input features and the driving environment dynamics information, and trained with minimizing the gap between the environmental dynamics posterior distribution information and the environmental dynamics prior information as the training goal.
8. An unmanned aerial vehicle autopilot system, characterized in that: include: An information collection module, used to collect multi-modal driving environment information; An information encoding module, used for encoding the driving environment information to obtain multimodal input features suitable for a world model; A first input module, used for inputting the multimodal input features into the world model to obtain driving environment dynamics information; The second input module is used to input the driving environment dynamics information into the brain-like neural circuit network to obtain the UAV automatic driving control information; The driving control module is used to control the drone automatically according to the drone automatic driving control information.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.
10. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Unmanned aerial vehicle cooperation system based on intelligent unmanned aerial vehicle and multi-vehicle cooperation control method
CN119690134A
Environment interaction method, system, equipment and product of humanoid robot
CN121809538A
