Intelligent automobile automatic driving method and system based on space-ground integrated coordination

Abstract

The invention discloses a space-ground integrated cooperative intelligent automobile automatic driving method and system, and the method comprises the implementation steps: starting an automobile terminal, calibrating the positioning information of an automobile, and accessing a cloud service center through a communication double link; starting an automatic driving main system and a twin system, and dynamically switching an automatic driving mode into a primary mode, an intermediate mode and an advanced mode according to the vehicle state; if the mode is the primary mode, starting a basic automatic driving strategy and method in the primary mode; if the mode is the intermediate mode, starting a dual-system automatic driving strategy and method in the intermediate mode; if the mode is the advanced mode, starting a vehicle cloud collaborative automatic driving strategy and method in the advanced mode; and when the main system is abnormal and fails, starting the emergency driving method. Omnibearing space-ground integrated cooperative automatic driving in a complex driving environment is realized, the reliability of the system is enhanced, and efficient and safe vehicle control is realized through a multi-stage automatic driving mode.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and particularly relates to an intelligent vehicle autonomous driving method and system for space-ground integrated collaboration. Background Art

[0002] With the rapid development of information technology and artificial intelligence, autonomous driving technology has become a research hotspot and a major development direction in fields such as vehicle engineering and transportation engineering. An intelligent vehicle integrates multiple sensors and advanced algorithms to achieve the processes of perception, decision-making, and control, thereby enabling automatic driving. In recent years, the end-to-end autonomous driving model, as an innovative technical path, aims to directly integrate the processes of perception, decision-making, and control into a unified model through methods such as deep learning, simplify the complex modular design in traditional autonomous driving systems, and improve the response speed and overall performance of the system. It has been applied in single-vehicle intelligence.

[0003] Although single-vehicle intelligence is relatively mature and has achieved certain applications, single-vehicle sensors are easily limited by factors such as the field of view, installation location, and weather conditions, making it difficult to comprehensively and accurately perceive the dynamic environment around the vehicle. Therefore, vehicle-road collaborative autonomous driving technology has received increasing attention and development. However, roadside devices often have high costs and require road reconstruction, resulting in a relatively slow construction speed and low coverage rate. In contrast, low-earth orbit satellites have the characteristics of wide coverage, low cost, and rapid construction. They can provide real-time, high-precision positioning and navigation services for ground vehicles and support large-scale data transmission and processing, which provides a new foundation for vehicle-cloud collaborative autonomous driving technology.

[0004] The space-ground integrated collaborative autonomous driving system can significantly improve the comprehensiveness and accuracy of environmental perception, and enhance the robustness and response ability of the system by sharing information and collaborating between ground vehicles and low-earth orbit satellites. Low-earth orbit satellites can not only provide high-precision vehicle positioning and navigation information, but also monitor traffic flow, road conditions, and potential hazards in real time through wide-area coverage and feedback this information to ground vehicles to assist them in making more accurate path planning and decisions. In addition, the high-bandwidth and low-latency characteristics of the low-earth orbit satellite network enable the real-time transmission and processing of large-scale vehicle data, thus supporting more complex and efficient autonomous driving algorithms.

[0005] Based on the above background, the present invention designs an integrated space-ground collaborative intelligent vehicle autonomous driving method and system, which can not only make full use of the high-precision positioning and wide-area communication advantages of low-earth orbit satellites to improve the environmental perception ability and decision-making level of the autonomous driving system, but also enhance the flexibility and robustness of the overall system to ensure safe and efficient autonomous driving in various complex and dynamic traffic environments. The research and application of such models have broad market prospects and important scientific research value, and will promote the development of intelligent transportation systems towards a more intelligent and collaborative direction. Summary of the Invention

[0006] The present invention proposes an integrated space-ground collaborative intelligent vehicle autonomous driving method and system, which realizes efficient and safe vehicle control through multiple levels of autonomous driving modes. This method accesses the cloud service center through dual communication links of mobile communication and satellite communication to ensure the reliability and low latency of communication, realizes high-precision positioning and navigation of the vehicle through low-earth orbit satellites, realizes autonomous driving through the main system and the twin system, and switches between primary, intermediate, and advanced driving modes according to the vehicle state. In addition, an emergency driving method in case of abnormal failure of the main system is also designed. Overall, the present invention realizes all-round integrated space-ground collaborative autonomous driving in complex driving environments, enhances the redundancy and reliability of the system, and significantly improves the safety and efficiency of autonomous driving. To achieve the above object, the present invention adopts the following technical solutions:

[0007] An integrated space-ground collaborative intelligent vehicle autonomous driving method, the method comprising:

[0008] S1. The vehicle terminal is started, the vehicle positioning information is calibrated, and the cloud service center is accessed through the dual communication links;

[0009] S2. The autonomous driving main system and the twin system are started, and the autonomous driving mode is dynamically switched according to the vehicle state;

[0010] S3. If the autonomous driving mode is the primary mode, the basic autonomous driving strategy and method in the primary mode are started;

[0011] S4. If the autonomous driving mode is the intermediate mode, the dual-system autonomous driving strategy and method in the intermediate mode are started;

[0012] S5. If the autonomous driving mode is the advanced mode, the vehicle-cloud collaborative autonomous driving strategy and method in the advanced mode are started;

[0013] S6. If the main system fails abnormally, an emergency driving method is started.

[0014] Further explanation, the step S1 is specifically:

[0015] S11. The vehicle terminal is started, the in-vehicle components are self-checked, the in-vehicle sensors are initialized and corrected;

[0016] S12. The vehicle terminal activates the network interface, detects and accesses the mobile communication network and the satellite communication network, initially sends a data packet to determine the reachability of the communication dual links, accesses the cloud service center through the dual links, and calibrates the vehicle positioning information at the same time;

[0017] S13. The vehicle terminal and the cloud service center regularly send data packets to determine the reachability and latency of the communication dual links, select the link with the lowest latency as the main link to transmit data, and the other one is the communication backup link. When the main link is unreachable or the latency is too high, the identities of the communication backup link and the main link are swapped.

[0018] Further explanation, the specific steps of step S2 are as follows:

[0019] S21. Read the in-vehicle autonomous driving configuration file, start the autonomous driving main system, run the self-check data of the main system, ensure the availability and accuracy of basic autonomous driving, and enter the primary mode;

[0020] S22. After ensuring the stable operation of the autonomous driving main system, start the twin system, load the end-to-end network, run the self-check data of the twin system. If the self-check passes, enter the intermediate mode, and realize autonomous driving through the dual systems. Otherwise, stay in the primary mode and realize autonomous driving only relying on the main system;

[0021] S23. When the autonomous driving system is in the intermediate mode, during driving, try to obtain vehicle-cloud collaboration information through the communication dual links. If vehicle-cloud collaboration can be carried out and the links are unobstructed, enter the advanced mode. Otherwise, stay in the intermediate mode;

[0022] S24. In the primary mode, basic autonomous driving is only realized through the autonomous driving main system, which directly senses information from the vehicle itself and outputs control signals; in the intermediate mode, autonomous driving is realized through the autonomous driving main system and the twin system together. The main system generates basic decision results, and the twin system outputs global optimization results to the main system, and the system switch is realized through the risk mechanism; in the advanced mode, vehicle-cloud collaboration is introduced on the basis of the intermediate mode to further guide the twin system to generate advanced global optimization results.

[0023] Further explanation, the specific steps of step S3 are as follows:

[0024] S31. In the primary mode, the vehicle terminal only realizes basic autonomous driving, obtains environmental perception information through in-vehicle sensors, including visual data collected by cameras and point cloud data collected by lidar sensors, and performs high-precision vehicle positioning and navigation through low-earth orbit satellites;

[0025] S32. The main system processes the environmental perception information using the visual encoder and the point cloud encoder respectively to obtain the visual BEV feature and the point cloud BEV feature, and uses a multi-modal feature fusion algorithm to fuse the visual BEV feature and the point cloud BEV feature to obtain a fused feature;

[0026] S33. The main system performs basic target detection based on the fused feature to obtain the surrounding target categories and positions, and calculates the surrounding target distances and aggregation densities therefrom;

[0027] S34. The main system selects a driving route based on low-earth orbit satellite positioning and navigation, evaluates the vehicle risk range according to the surrounding target distances and aggregation densities, and thus performs decision-making path planning and outputs waypoints;

[0028] S35. The main system calculates the speed and direction required to reach the waypoint based on the vehicle kinematic model and the dynamic model, and converts them into a lateral control signal and a longitudinal control signal.

[0029] Further explanation, the specific content of step S4 is as follows:

[0030] S41. In the intermediate mode, the vehicle terminal uses the main system and the twin system to achieve dual-system autonomous driving. The main system undertakes the basic perception decision control task, obtains environmental perception information through in-vehicle sensors, performs high-precision vehicle positioning and navigation through low-earth orbit satellites, and transmits the data to the twin system. The twin system uses an end-to-end network to output the global optimization result, and the two systems achieve switching through a risk mechanism;

[0031] S42. Under the risk mechanism, the main system processes and fuses the environmental perception information to obtain a fused feature, calculates the surrounding target distances and aggregation densities based on this, further evaluates the vehicle risk range, generates a risk index in combination with the vehicle pose and speed, and determines the dual-system switching timing according to the risk index;

[0032] S43. If the risk index is lower than the threshold, the twin system is intervened. The twin system adopts a modular end-to-end network and regularly obtains a dynamically updated vectorized map from the cloud service center through a communication dual link, which includes traffic signs and lane lines and other traffic elements in the field environment where the vehicle terminal is located, so as to achieve more accurate decision-making planning and support lane-level decision-making planning at complex intersections;

[0033] S44. The twin system uses the fusion features of the main system as input for object detection and tracking. Then, based on the high-precision positioning of the vehicle provided by the low-earth orbit satellite, it calculates the projection matrix from its own coordinate system to the global coordinate system, projects the detected objects to the correct positions on the map, predicts and generates the future risk range, thereby conducts decision-making path planning, and outputs waypoints to the main system. The main system calculates the required speed and direction based on the vehicle kinematic model and dynamic model, and finally converts them into control signals;

[0034] S45. If the risk index is higher than the threshold, it indicates that the current road conditions are relatively complex. At this time, further combine the vehicle speed, the field map and historical experience to judge the risk critical decision interval. If the interval is greater than the decision-making delay of the twin system, the twin system will continue to perform autonomous driving. Otherwise, it indicates that the decision-making delay of the twin system will increase the driving risk under the current road conditions. At this time, it switches back to the main system for autonomous driving, which directly conducts decision-making path planning based on the risk range, outputs waypoints, calculates the required speed and direction based on the vehicle kinematic model and dynamic model, and generates control signals accordingly;

[0035] S46. In the intermediate mode, whether the twin system intervenes or not, the main system of the vehicle terminal can independently achieve basic autonomous driving, and the structure is relatively simple, which can respond quickly in real time. In contrast, the twin system can further achieve end-to-end optimization and output better decision-making results, but it has a certain calculation delay and the real-time performance is lower than that of basic autonomous driving. Therefore, it mainly intervenes in the main system when the risk is relatively low.

[0036] Further explanation, the specific steps of step S5 are as follows:

[0037] S51. In the advanced mode, the vehicle terminal uses the main system and the twin system to achieve dual-system autonomous driving, and further introduces vehicle-cloud collaboration. The cloud can obtain more generalized global traffic information and the driving requirements of each vehicle terminal, thereby conducts global path advanced planning, provides collaborative decision-making guidance for the vehicle terminal, and realizes high-level autonomous driving;

[0038] S52. In addition to completing various tasks in the intermediate mode, the main system further transmits the environmental perception data and high-precision vehicle positioning data to the cloud service center through the main link;

[0039] S53, the cloud service center clusters and groups multiple automobile terminals, and performs data preprocessing on the latest data of each automobile terminal within the group. First, the point cloud data of each automobile terminal is extracted respectively, and mapped to the global coordinate system according to the positioning data of the automobile to achieve preliminary fusion of the point cloud, and then the field point cloud BEV features are generated through the point cloud encoder. After that, the visual data of each automobile terminal is extracted, and the field visual BEV features are generated based on the vehicle positioning and visual encoder. Finally, the multimodal feature fusion algorithm is used to fuse the field point cloud BEV features and the field visual BEV features to obtain the field fusion features, and the field map is dynamically updated at the same time;

[0040] S54. The cloud service center detects and tracks field targets based on the field fusion features, and calibrates the field target positions through low-orbit satellites to predict the target trajectory and generate the future risk range of the field;

[0041] S55, the cloud service center makes a high-dimensional optimal decision based on the driving needs of each vehicle terminal in the group, the future risk range of the field, and the traffic flow optimization goal, and realizes the global path advance planning, and then transmits the advance path planning results and the future risk range of the field back to each vehicle terminal through the main link;

[0042] S56. The vehicle terminal receives the advanced path planning results and future risk range of the field from the cloud service center through the twin system, and based on this, corrects its own decision results to achieve better decision path planning, and outputs waypoints to the main system, calculates the speed and direction based on the vehicle kinematic model and dynamic model, and finally outputs the control signal;

[0043] S57. If the vehicle terminal suddenly enters a high-risk section, the main system will be switched to full driving, and the decision-making path planning will be made directly according to the risk range, waypoints will be output, speed and direction will be calculated, and control signals will be generated. At the same time, the main system will continue to transmit data to the cloud service center to re-realize high-level autonomous driving in vehicle-cloud collaboration when exiting the high-risk section. If the vehicle terminal is in a high-risk section for a long time and exceeds the time waiting threshold, the main system will stop transmitting data to the cloud service center, freeing up communication bandwidth and reducing communication pressure within the field.

[0044] To further illustrate, the step S6 is specifically as follows:

[0045] S61. In all driving modes, the vehicle terminal uses the main system as the underlying architecture for automatic driving. In abnormal conditions, such as system crashes and system upgrades, the vehicle terminal should still have driving capabilities and provide a takeover option when the system recovers.

[0046] S62. Under various driving modes, the controller is independent of the main system. It can accept two-way signal inputs from the main system and manual operations, and monitor the functions of the main system through a supervision mechanism.

[0047] S63. In the supervision mechanism, perception test data is input into the main system at regular intervals, and the output control test signal is received. If the control test signal is normally emitted and within the normal range, it indicates that the main system functions normally and the autonomous driving task can continue. Otherwise, it indicates that the main system is in an abnormal condition and it is difficult to continue the autonomous driving task. At this time, when the controller detects the corresponding situation, it reminds the user to directly intervene in driving through an alarm sound, and at the same time uses the manual operation signal to perform permission override.

[0048] S64. After the user directly intervenes in driving, the user can directly output manual operation signals to the controller through components such as the accelerator, brake, and steering wheel, and thus continue to control the vehicle terminal.

[0049] S65. After the main system resumes its function, it starts to send control test signals to the controller again. The controller then starts to receive two-way signals from the main system and manual operations, and provides the user with an autonomous driving switching option through the in-vehicle system. If the user switches back to the autonomous driving mode, the main system takes over the driving state. Otherwise, the user operation state is continued, thus achieving safer driving.

[0050] The present invention also proposes an intelligent vehicle autonomous driving system with space-earth integrated collaboration, including a vehicle terminal, a cloud service center, and a communication dual link; the vehicle terminal can transmit information to the cloud service center through the communication dual link.

[0051] The vehicle terminal includes an autonomous driving system and a manual driving system. The autonomous driving system and the manual driving system are supervised and switched by the controller. When the controller supervises that the autonomous driving system is normal, the vehicle is controlled by the autonomous driving system. When the controller supervises that the autonomous driving system is abnormal, the vehicle is controlled by the manual driving system.

[0052] The autonomous driving system includes a main system and a twin system. The main system and the twin system dynamically switch the autonomous driving mode to a primary mode, an intermediate mode, and an advanced mode according to the vehicle state. The primary mode only realizes basic autonomous driving through the main system, which directly senses information from the vehicle itself and outputs control signals; the intermediate mode realizes autonomous driving through the cooperation of the main system and the twin system. The main system generates basic decision results, and the twin system outputs global optimization results to the main system, and the system switch is realized through a risk mechanism; the advanced mode performs vehicle-cloud collaboration on the basis of the intermediate mode. The cloud service center further guides the twin system to generate advanced global optimization results and transmits them to the main system, and the main system outputs control signals.

[0053] The manual driving system enables the driver to directly control the accelerator, brake, and steering wheel;

[0054] The communication dual-link includes a mobile communication network and a satellite communication network. The vehicle terminal and the cloud service center periodically send data packets to determine the reachability and latency of the communication dual-link, and select the link with the lowest latency as the primary link for data transmission, while the other is the communication backup link. When the primary link is unreachable or has too high latency, the identities of the communication backup link and the primary link are swapped.

[0055] Furthermore, the main system and the twin system dynamically switch the autonomous driving mode to the primary mode, intermediate mode, and advanced mode according to the vehicle state, and this switching process adopts the content from S2 to S5 above.

[0056] Furthermore, the autonomous driving system and the manual driving system are supervised and switched by a controller, and this supervision mechanism is implemented according to the content of S6 above.

[0057] Advantages of the present invention:

[0058] 1. The present invention accesses the cloud service center through the communication dual-link of mobile communication and satellite communication, ensuring the redundancy and stability of communication. In the case where the primary link is unreachable or has too high latency, the backup link can take over in a timely manner to ensure the continuous operation of the external communication system;

[0059] 2. The present invention designs three autonomous driving modes: primary, intermediate, and advanced, which can be dynamically switched according to the real-time vehicle state and road conditions, ensuring that the optimal driving strategy can be provided in driving environments with different levels of complexity, and improving the adaptability and flexibility of the system;

[0060] 3. Through the design of the risk mechanism, the present invention can preferentially rely on the main system for quick decision-making in complex road conditions with high real-time requirements, and introduce the twin system for in-depth optimization in cases with lower risks, thus effectively balancing the relationship between real-time response and decision optimization, and improving the performance of the overall system;

[0061] 4. The present invention utilizes multi-modal sensor fusion and high-precision positioning technology, combined with global data processing in the cloud, to achieve accurate target detection, trajectory prediction, and risk assessment, making the autonomous driving system more intelligent and forward-looking in path planning, and significantly improving the safety and efficiency of driving;

[0062] 5. The present invention ensures that when the main system fails abnormally, the controller can monitor in a timely manner and prompt the user to intervene in driving, ensuring that the vehicle has the ability to drive safely under any circumstances, thereby improving the fault tolerance, safety, and reliability of the system. Description of the Drawings

[0063] Figure 1 Schematic diagram of the overall method flow of the present invention;

[0064] Figure 2 Schematic diagram of the specific operation process of the automatic driving system of the present invention;

[0065] Figure 3 Schematic diagram of the safe driving system of the present invention; Specific implementation manners

[0066] The present invention will be described in detail below in conjunction with the embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, method, or functional transformation made by those of ordinary skill in the art based on these embodiments is included in the protection scope of the present invention.

[0067] As Figure 1 shown, the integrated space-ground collaborative intelligent vehicle automatic driving method of the present invention includes the following steps:

[0068] S1. The vehicle terminal is started, the vehicle positioning information is calibrated, and the cloud service center is accessed through the communication dual-link;

[0069] S2. The automatic driving main system and the twin system are started, and the automatic driving mode is dynamically switched according to the vehicle state, including the primary mode, the intermediate mode, and the advanced mode;

[0070] S3. If the automatic driving mode is the primary mode, the basic automatic driving strategy and method in the primary mode are started;

[0071] S4. If the automatic driving mode is the intermediate mode, the dual-system automatic driving strategy and method in the intermediate mode are started;

[0072] S5. If the automatic driving mode is the advanced mode, the vehicle-cloud collaborative automatic driving strategy and method in the advanced mode are started;

[0073] S6. If the main system fails abnormally, an emergency driving method is started.

[0074] Further explanation, the specific content of step S1 is:

[0075] The vehicle terminal is started, the vehicle positioning information is calibrated, and the cloud service center is accessed through the communication dual-link;

[0076] S11. The vehicle terminal is started, the in-vehicle components are self-checked, and the in-vehicle sensors are initialized and corrected;

[0077] S12. The vehicle terminal turns on the network interface, detects and accesses the mobile communication network and the satellite communication network, initially sends a data packet to determine the reachability of the communication dual-link, accesses the cloud service center through the dual-link, and calibrates the vehicle positioning information at the same time;

[0078] S13. The vehicle terminal and the cloud service center regularly send data packets to determine the reachability and latency of the communication dual links, select the link with the lowest latency as the main link for data transmission, and the other as the communication backup link. When the main link is unreachable or the latency is too high, the identities of the communication backup link and the main link are swapped.

[0079] Combined with Figure 2 As shown in

[0080] Further explanation, the specific step S2 is as follows:

[0081] Start the autonomous driving main system and the twin system, and dynamically switch the autonomous driving mode according to the vehicle state.

[0082] S21. Read the in-vehicle autonomous driving configuration file, start the autonomous driving main system, and run the self-check data of the main system to ensure the availability and accuracy of basic autonomous driving, and enter the primary mode.

[0083] S22. After ensuring the stable operation of the autonomous driving main system, start the twin system, load the end-to-end network, and run the self-check data of the twin system. If the self-check passes, enter the intermediate mode, and realize autonomous driving through the dual systems. Otherwise, stay in the primary mode and realize autonomous driving only relying on the main system.

[0084] S23. When the autonomous driving system is in the intermediate mode, try to obtain vehicle-cloud collaboration information through the communication dual links during driving. If vehicle-cloud collaboration is possible and the link is unobstructed, enter the advanced mode. Otherwise, stay in the intermediate mode.

[0085] S24. In the primary mode, basic autonomous driving is only realized through the autonomous driving main system, which directly outputs control signals from the vehicle's perception information. In the intermediate mode, autonomous driving is realized through the joint action of the autonomous driving main system and the twin system. The main system generates basic decision results, and the twin system outputs global optimization results to the main system, and the system switch is realized through the risk mechanism. In the advanced mode, vehicle-cloud collaboration is introduced on the basis of the intermediate mode to further guide the twin system to generate advanced global optimization results.

[0086] Further explanation, the specific step S3 is as follows:

[0087] Basic autonomous driving strategies and methods in the primary mode;

[0088] S31. In the primary mode, the vehicle terminal only realizes basic autonomous driving, obtains environmental perception information through in-vehicle sensors, including visual data collected by cameras and point cloud data collected by lidar sensors, and performs high-precision vehicle positioning and navigation through low-earth orbit satellites.

[0089] S32. The main system processes the environmental perception information using the visual encoder and the point cloud encoder respectively to obtain the visual BEV feature and the point cloud BEV feature, and uses a multi-modal feature fusion algorithm to fuse the visual BEV feature and the point cloud BEV feature to obtain a fused feature;

[0090] S33. The main system performs basic target detection based on the fused feature to obtain the surrounding target categories and positions, and calculates the surrounding target distances and aggregation densities therefrom;

[0091] S34. The main system selects a driving route based on low-earth orbit satellite positioning and navigation, evaluates the vehicle risk range according to the surrounding target distances and aggregation densities, and thus performs decision path planning and outputs waypoints;

[0092] S35. The main system calculates the speed and direction required to reach the waypoint based on the vehicle kinematic model and the dynamic model, and converts them into a lateral control signal and a longitudinal control signal;

[0093] Further explanation: The specific content of step S4 is as follows:

[0094] Dual-system autonomous driving strategy and method in the intermediate mode;

[0095] S41. In the intermediate mode, the vehicle terminal uses the main system and the twin system to achieve dual-system autonomous driving. The main system undertakes the basic perception decision control task, obtains environmental perception information through in-vehicle sensors, performs high-precision vehicle positioning and navigation through low-earth orbit satellites, and transmits the data to the twin system. The twin system uses an end-to-end network to output a global optimization result, and the dual system realizes switching through a risk mechanism;

[0096] S42. Under the risk mechanism, the main system processes and fuses the environmental perception information to obtain a fused feature, calculates the surrounding target distances and aggregation densities based on this, further evaluates the vehicle risk range, generates a risk index by combining the vehicle pose and speed, and determines the dual-system switching timing according to this index;

[0097] S43. If the risk index is lower than the threshold, the twin system is intervened. The twin system adopts a modular end-to-end network and regularly obtains a dynamically updated vectorized map from the cloud service center through a communication dual link, which includes traffic signs and lane lines and other traffic elements in the field environment where the vehicle terminal is located, so as to achieve more accurate decision-making planning and support lane-level decision-making planning at complex intersections;

[0098] S44. The twin system uses the fusion features of the main system as input for object detection and tracking. Then, based on the high-precision positioning of the vehicle provided by the low-earth orbit satellite, it calculates the projection matrix from its own coordinate system to the global coordinate system, projects the detected objects to the correct positions on the map, predicts and generates the future risk range, thereby conducts decision-making path planning, and outputs waypoints to the main system. The main system calculates the required speed and direction based on the vehicle kinematic model and dynamic model, and finally converts them into control signals;

[0099] S45. If the risk index is higher than the threshold, it indicates that the current road conditions are relatively complex. At this time, further combine the vehicle speed, field map and historical experience to judge the risk critical decision interval. If the interval is greater than the decision-making delay of the twin system, continue to perform autonomous driving by the twin system. Otherwise, it indicates that the decision-making delay of the twin system will increase the driving risk under the current road conditions. At this time, switch back to the main system for autonomous driving. It directly conducts decision-making path planning based on the risk range, outputs waypoints, calculates the required speed and direction based on the vehicle kinematic model and dynamic model, and generates control signals accordingly;

[0100] S46. In the intermediate mode, whether the twin system intervenes or not, the main system of the vehicle terminal can independently achieve basic autonomous driving, and the structure is relatively simple, which can respond quickly and in real time. In contrast, the twin system can further achieve end-to-end optimization and output better decision-making results, but it has a certain calculation delay and the real-time performance is lower than that of basic autonomous driving. Therefore, it mainly intervenes in the main system when the risk is relatively low;

[0101] Further explanation, the specific steps of step S5 are as follows:

[0102] The vehicle-cloud collaborative autonomous driving strategy and method in the advanced mode;

[0103] S51. In the advanced mode, the vehicle terminal uses the main system and the twin system to achieve dual-system autonomous driving, and further introduces vehicle-cloud collaboration. The cloud can obtain more generalized global traffic information and the driving requirements of each vehicle terminal, thereby conducts global path advanced planning, provides collaborative decision-making guidance for the vehicle terminal, and realizes high-level autonomous driving;

[0104] S52. In addition to completing various tasks in the intermediate mode, the main system further transmits the environmental perception data and high-precision vehicle positioning data to the cloud service center through the main link;

[0105] S53, the cloud service center clusters and groups multiple automobile terminals, and performs data preprocessing on the latest data of each automobile terminal within the group. First, the point cloud data of each automobile terminal is extracted respectively, and mapped to the global coordinate system according to the positioning data of the automobile to achieve preliminary fusion of the point cloud, and then the field point cloud BEV features are generated through the point cloud encoder. After that, the visual data of each automobile terminal is extracted, and the field visual BEV features are generated based on the vehicle positioning and visual encoder. Finally, the multimodal feature fusion algorithm is used to fuse the field point cloud BEV features and the field visual BEV features to obtain the field fusion features, and the field map is dynamically updated at the same time;

[0106] S54. The cloud service center detects and tracks field targets based on the field fusion features, and calibrates the field target positions through low-orbit satellites to predict the target trajectory and generate the future risk range of the field;

[0107] S55, the cloud service center makes a high-dimensional optimal decision based on the driving needs of each vehicle terminal in the group, the future risk range of the field, and the traffic flow optimization goal, and realizes the global path advance planning, and then transmits the advance path planning results and the future risk range of the field back to each vehicle terminal through the main link;

[0108] S56. The vehicle terminal receives the advanced path planning results and future risk range of the field from the cloud service center through the twin system, and based on this, corrects its own decision results to achieve better decision path planning, and outputs waypoints to the main system, calculates the speed and direction based on the vehicle kinematic model and dynamic model, and finally outputs the control signal;

[0109] S57. If the vehicle terminal suddenly enters a high-risk section, it will switch to the main system for full driving, make decision-making path planning directly according to the risk range, output waypoints, calculate speed and direction, and generate control signals. At the same time, the main system will continue to transmit data to the cloud service center to re-realize high-level automatic driving in vehicle-cloud collaboration when exiting the high-risk section; if the vehicle terminal is in a high-risk section for a long time and exceeds the time waiting threshold, the main system will stop transmitting data to the cloud service center to release communication bandwidth and reduce communication pressure in the field;

[0110] To further illustrate, the step S6 is specifically as follows:

[0111] like Figure 3 As shown, the driving method when the main system fails abnormally;

[0112] S61. In all driving modes, the vehicle terminal uses the main system as the underlying architecture for automatic driving. In abnormal conditions, such as system crashes and system upgrades, the vehicle terminal should still have driving capabilities and provide a takeover option when the system recovers.

[0113] S62. Under various driving modes, the controller is independent of the main system. It can accept two-way control signal inputs from the main system and manual operations, and monitor the functions of the main system through a supervision mechanism.

[0114] S63. In the supervision mechanism, the controller periodically inputs sensing test data to the main system and receives the output control test signal. If the control test signal is normally issued and within the normal range, it indicates that the main system functions normally and the autonomous driving task can continue. Otherwise, it indicates that the main system is in an abnormal condition and it is difficult to continue the autonomous driving task. At this time, when the controller detects the corresponding situation, it reminds the user to directly intervene in driving through an alarm sound, and at the same time uses the manual operation signal to perform permission override.

[0115] S64. After the user directly intervenes in driving, they can directly output manual operation signals to the controller through components such as the accelerator, brake, and steering wheel, thereby continuing to control the vehicle terminal.

[0116] S65. After the main system resumes its functions, it starts to send control test signals to the controller again. The controller then starts to receive two-way signals from the main system and manual operations, and provides the user with an autonomous driving switching option through the in-vehicle system. If the user switches back to the autonomous driving mode, the main system takes over the driving state. Otherwise, the user operation state is continued, thus achieving safer driving.

[0117] The series of detailed descriptions listed above are only specific descriptions of the feasible implementation modes of the present invention, and they are not intended to limit the protection scope of the present invention. Any equivalent modes or changes made without departing from the technology created by the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration, characterized in that: include: S1. The car terminal starts, calibrates the vehicle positioning information, and accesses the cloud service center through a dual-link communication link; S2: Start the main autonomous driving system and the twin system, and dynamically switch the autonomous driving mode according to the vehicle status; S3. If the autonomous driving mode is the primary mode, start the basic autonomous driving strategy and method in the primary mode; S4. If the autonomous driving mode is the intermediate mode, the dual-system autonomous driving strategy and method in the intermediate mode is activated; S5. If the autonomous driving mode is the advanced mode, the vehicle-cloud collaborative autonomous driving strategy and method in the advanced mode is started; S6. If the main system of the automatic driving fails abnormally, initiate the emergency driving method.

2. The method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration according to claim 1, characterized in that: The step S1 specifically includes: S11, the vehicle terminal starts, the vehicle components self-check, and the vehicle sensors are initialized and calibrated; S12, the vehicle terminal opens the network interface, detects and accesses the dual link composed of the mobile communication network and the satellite communication network, sends a data packet for the first time to determine the accessibility of the communication dual link, accesses the cloud service center through the dual link, and calibrates the vehicle positioning information at the same time; S13. The vehicle terminal and the cloud service center periodically send data packets to determine the reachability and delay of the dual communication links, and select the link with the lowest delay as the main link to transmit data. The other is the communication backup link. When the main link is unreachable or the delay is too high, the identity of the communication backup link and the main link are swapped.

3. The method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration according to claim 1, characterized in that: The step S2 is specifically as follows: S21, read the vehicle's autonomous driving configuration file, start the autonomous driving main system, run the main system self-check data, ensure the availability and accuracy of basic autonomous driving, and enter the primary mode; S22. After ensuring that the main system of autonomous driving is running stably, start the twin system, load the end-to-end network, and run the self-check data of the twin system. If the self-check passes, enter the intermediate mode and realize autonomous driving through the dual systems. Otherwise, stay in the primary mode and realize autonomous driving only by relying on the main system. S23: When the autonomous driving system is in the intermediate mode, it attempts to obtain vehicle-cloud collaboration information through the dual communication link during driving. If vehicle-cloud collaboration is possible and the link is unobstructed, it enters the advanced mode, otherwise it stays in the intermediate mode. S24. In the primary mode, basic autonomous driving is achieved only through the autonomous driving main system, which directly senses information from the vehicle and outputs control signals. In the intermediate mode, autonomous driving is achieved through the autonomous driving main system and the twin system. The main system generates basic decision results, and the twin system outputs global optimization results to the main system, and system switching is achieved through a risk mechanism. In the advanced mode, vehicle-cloud collaboration is introduced on the basis of the intermediate mode to further guide the twin system to generate advanced global optimization results.

4. The method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration according to claim 3 is characterized in that: The specific implementation of step S3 includes: S31. In the primary mode, the vehicle terminal only realizes basic autonomous driving, obtains environmental perception information through on-board sensors, including visual data collected by the camera and point cloud data collected by the lidar sensor, and performs high-precision vehicle positioning and navigation through low-orbit satellites; S32, the main system uses the visual encoder and the point cloud encoder to process the environmental perception information respectively, obtains the visual BEV features and the point cloud BEV features, and uses the multimodal feature fusion algorithm to fuse the visual BEV features and the point cloud BEV features to obtain the fusion features; S33, the main system performs basic target detection based on the fusion features, obtains the category and position of the surrounding targets, and thereby calculates the distance and aggregation density of the surrounding targets; S34, the main system selects the driving route based on low-orbit satellite positioning and navigation, evaluates the vehicle risk range according to the surrounding target distance and aggregation density, and thus makes decision-making path planning and outputs waypoints; S35. The main system calculates the speed and direction required to reach the waypoint based on the vehicle kinematic model and dynamic model, and converts them into lateral control signals and longitudinal control signals.

5. The method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration according to claim 3 is characterized in that: The specific implementation of step S4 includes: S41. In the intermediate mode, the vehicle terminal uses the main system and the twin system to realize dual-system automatic driving. The main system undertakes the basic perception decision-making and control tasks, obtains environmental perception information through on-board sensors, performs high-precision vehicle positioning and navigation through low-orbit satellites, and transmits data to the twin system. The twin system uses an end-to-end network to output global optimization results, and the dual systems achieve switching through a risk mechanism. S42. Under the risk mechanism, the main system processes and fuses the environmental perception information to obtain fusion features, based on which the distance and aggregation density of surrounding targets are calculated, and the risk range of the vehicle is further evaluated. The risk index is generated in combination with the vehicle posture and speed, and the timing of dual system switching is determined based on the risk index; S43. If the risk index is lower than the threshold, the twin system is involved. The twin system adopts a modular end-to-end network and regularly obtains dynamically updated vector maps from the cloud service center through dual communication links. The map contains traffic signs and lane lines and other traffic elements in the field environment where the vehicle terminal is located, so as to achieve more accurate decision-making and planning, thereby supporting lane-level decision-making and planning at complex intersections; S44. The twin system uses the fusion features of the main system as input to detect and track objects. It then calculates the projection matrix from its own coordinate system to the global coordinate system based on the high-precision positioning of the vehicle provided by the low-orbit satellite, and projects the detected objects to the correct position on the map, thereby predicting and generating the future risk range, thereby making decision-making path planning, and outputting waypoints to the main system, which calculates the required speed and direction based on the vehicle kinematic model and dynamic model, and finally converts them into control signals. S45. If the risk index is higher than the threshold, it means that the current road conditions are relatively complicated. At this time, the risk critical decision interval is further judged in combination with the vehicle speed, field map and historical experience. If the interval is greater than the decision delay of the twin system, the twin system will continue to perform automatic driving. Otherwise, it indicates that the decision delay of the twin system under the current road conditions will increase the driving risk. At this time, it is switched back to the main system for automatic driving. It directly plans the decision path according to the risk range, outputs waypoints, and calculates the required speed and direction based on the vehicle kinematic model and dynamic model to generate control signals.

6. The method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration according to claim 3 is characterized in that: The specific implementation of step S5 includes: S51. In advanced mode, the vehicle terminal uses the main system and the twin system to achieve dual-system autonomous driving, and further introduces vehicle-cloud collaboration, using the cloud service center to obtain more general global traffic information and the driving needs of each vehicle terminal, so as to carry out global path advance planning and provide collaborative decision-making guidance for the vehicle terminal to achieve high-level autonomous driving; S52, in addition to completing various tasks in the intermediate mode, the main system further transmits the environmental perception data and high-precision vehicle positioning data to the cloud service center through the main link; S53, the cloud service center clusters and groups multiple automobile terminals, and performs data preprocessing on the latest data of each automobile terminal within the group. First, the point cloud data of each automobile terminal is extracted respectively, and mapped to the global coordinate system according to the positioning data of the automobile to achieve preliminary fusion of the point cloud, and then the field point cloud BEV features are generated through the point cloud encoder. After that, the visual data of each automobile terminal is extracted, and the field visual BEV features are generated based on the vehicle positioning and visual encoder. Finally, the multimodal feature fusion algorithm is used to fuse the field point cloud BEV features and the field visual BEV features to obtain the field fusion features, and the field map is dynamically updated at the same time; S54. The cloud service center detects and tracks field targets based on the field fusion features, and calibrates the field target positions through low-orbit satellites to predict the target trajectory and generate the future risk range of the field; S55, the cloud service center makes a high-dimensional optimal decision based on the driving needs of each vehicle terminal in the group, the future risk range of the field, and the traffic flow optimization goal, and realizes the global path advance planning, and then transmits the advance path planning results and the future risk range of the field back to each vehicle terminal through the main link; S56. The vehicle terminal receives the advanced path planning results and future risk range of the field from the cloud service center through the twin system, and based on this, corrects its own decision results to achieve better decision path planning, and outputs waypoints to the main system. The main system calculates the speed and direction based on the vehicle kinematic model and dynamic model, and finally outputs the control signal; S57. If the vehicle terminal suddenly enters a high-risk section, the main system will be switched to full driving, and the decision-making path planning will be made directly according to the risk range, waypoints will be output, speed and direction will be calculated, and control signals will be generated. At the same time, the main system will continue to transmit data to the cloud service center to re-realize high-level autonomous driving in vehicle-cloud collaboration when exiting the high-risk section. If the vehicle terminal is in a high-risk section for a long time and exceeds the time waiting threshold, the main system will stop transmitting data to the cloud service center, freeing up communication bandwidth and reducing communication pressure within the field.

7. The method for automatic driving of an intelligent vehicle with integrated earth-ground collaboration according to claim 1 or 3, characterized in that: The specific implementation of step S6 includes: S61. In all driving modes, the vehicle terminal uses the main system as the underlying architecture for automatic driving. In abnormal conditions, including system crashes and system upgrades, the vehicle terminal should still have driving capabilities and provide a takeover option when the system recovers. S62. In each driving mode, the controller is independent of the main system. It can accept bidirectional control signal input from the main system and manual operation, and monitor the main system function through a supervision mechanism; S63. In the supervision mechanism, the perception test data is input to the main system at regular intervals, and the control test signal outputted by the main system is received. If the control test signal is sent normally and is within the normal range, it indicates that the main system functions normally and can continue to perform the automatic driving task. Otherwise, it indicates that the main system is in an abnormal condition and it is difficult to continue the automatic driving task. At this time, the controller detects the corresponding situation and reminds the user to directly intervene in the driving through an alarm sound, and uses a manual operation signal to override the authority. S64, after the user directly intervenes in driving, the user directly outputs manual operation signals to the controller through the accelerator, brake, steering wheel and other components, thereby continuing to control the vehicle terminal; S65. After the main system recovers its function, it starts to send control test signals to the controller again. The controller then starts to receive two-way signals from the main system and manual operation, and provides the user with an automatic driving switching option through the vehicle system. If the user switches back to the automatic driving mode, the main system takes over the driving status. Otherwise, the user's operation status will continue to be maintained, thereby achieving safer driving.

8. The intelligent vehicle automatic driving system with integrated earth-ground collaboration is characterized by: It includes a car terminal, a cloud service center and a communication dual link; the car terminal can transmit information with the cloud service center through the communication dual link; The automobile terminal includes an automatic driving system and a manual driving system, wherein the automatic driving system and the manual driving system are switched under supervision of a controller, and when the controller detects that the automatic driving system is normal, the automatic driving system controls the vehicle, and when the controller detects that the automatic driving system is abnormal, the manual driving system controls the vehicle; The automatic driving system includes a main system and a twin system. The main system and the twin system dynamically switch the automatic driving mode to a primary mode, an intermediate mode and an advanced mode according to the vehicle state. The primary mode realizes basic automatic driving only through the main system, which directly senses information from the vehicle and outputs a control signal; the intermediate mode realizes automatic driving through the cooperation between the main system and the twin system. The main system generates basic decision results, and the twin system outputs global optimization results to the main system, and realizes system switching through a risk mechanism; the advanced mode performs vehicle-cloud collaboration on the basis of the intermediate mode, and the cloud service center further guides the twin system to generate advanced global optimization results and transmits them to the main system, and the main system outputs control signals; The manual driving system enables the driver to directly control the accelerator, brake and steering wheel; The dual communication link includes a mobile communication network and a satellite communication network. The automobile terminal and the cloud service center periodically send data packets to determine the reachability and delay of the dual communication links, and select the link with the lowest delay as the main link to transmit data. The other is the communication backup link. When the main link is unreachable or the delay is too high, the identity of the communication backup link and the main link are swapped.

9. The sky-ground integrated intelligent vehicle automatic driving system according to claim 8, characterized in that: The main system and the twin system dynamically switch the autonomous driving mode to primary mode, intermediate mode and advanced mode according to the vehicle status, and the switching process adopts the contents of S2 to S5 described in any one of claims 1-7.

10. The sky-ground integrated intelligent vehicle automatic driving system according to claim 8, characterized in that: The automatic driving system and the manual driving system are switched under supervision of a controller, and the supervision mechanism is implemented according to the content of S6 described in any one of claims 1-7.

Images