Driving control method, device and system for vehicle

Abstract

This specification provides a driving control method, device, and system for a vehicle. The method includes: obtaining environmental data of the current environment where the vehicle is located, and using a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; respectively inputting the environmental data and the high-risk driving content into an intelligent driving model, and obtaining a comprehensive driving control instruction output after the model's inference; and controlling the vehicle to drive according to the comprehensive driving control instruction.

Description

Technical Field

[0001] This application relates to the technical field of vehicles, and particularly to a driving control method, device and system for a vehicle. Background Art

[0002] A vehicle can implement intelligent driving functions based on an intelligent driving model (hereinafter referred to as an intelligent driving model for short). At present, the intelligent driving model is usually implemented based on rules and / or neural networks.

[0003] There are a large number of diverse driving scenarios in the real world, and the long-tail problem of driving scenarios is relatively prominent, that is, there are driving scenarios that are rare but have a large overall number and are relatively complex, and there are many long-tail cases (such as edge cases, corner cases, extreme cases, etc.).

[0004] The rule-based intelligent driving model needs to rely on pre-summarized and set logic and conditions to operate. Therefore, such models are usually only applicable to simple driving scenarios (such as starting, following, lane changing, braking, etc.) where corresponding rules can be manually extracted, and the driving effect in complex driving scenarios is poor or even it is difficult to implement the intelligent driving function, and it is often difficult to handle the above long-tail cases. The neural network-based intelligent driving model needs a large number of real driving samples for training to improve performance. However, due to the existence of the aforementioned long-tail problem, technicians can usually only collect driving cases in conventional driving scenarios as training samples, and cannot exhaust all driving cases in the real world, especially cannot obtain enough cases in long-tail scenarios. Therefore, the trained intelligent driving model often cannot accurately meet the intelligent driving requirements in the above long-tail scenarios and urgently needs to be improved. Summary of the Invention

[0005] In view of this, this application provides a driving control method, device and system for a vehicle, and solves the problems existing in the traditional intelligent driving model in the related art by introducing a risk prediction model.

[0006] Specifically, this application is implemented through the following technical solutions:

[0007] According to the first aspect of this application, a driving control method for a vehicle is provided, including:

[0008] Obtain the environmental data of the current environment where the vehicle is located, and use the risk prediction model to predict the high-risk driving factors in the current environment based on the environmental data and output the corresponding high-risk driving content;

[0009] Input the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtain the comprehensive driving control instruction output after the model inference;

[0010] Control the vehicle to travel according to the comprehensive driving control instruction.

[0011] According to a second aspect of the present application, there is provided a driving control device for a vehicle, including:

[0012] A data acquisition unit, configured to acquire environmental data of the current environment where the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content;

[0013] An instruction acquisition unit, configured to input the environmental data and the high-risk driving content into an intelligent driving model respectively, and acquire a comprehensive driving control instruction output after the inference of the model;

[0014] A vehicle control unit, configured to control the vehicle to travel according to the comprehensive driving control instruction.

[0015] According to a third aspect of the present application, there is provided an intelligent driving system, which includes a risk prediction model, an intelligent driving model and a vehicle control module, wherein,

[0016] The risk prediction model is used to predict high-risk driving factors in the current environment based on environmental data of the current environment where the vehicle is located and output corresponding high-risk driving content;

[0017] The intelligent driving model is used to perform inference based on the environmental data and the high-risk driving content, and output an inferred comprehensive driving control instruction;

[0018] The vehicle control module is used to control the vehicle to travel according to the comprehensive driving control instruction.

[0019] According to a fourth aspect of the present application, there is provided an electronic device, including:

[0020] A processor; a memory for storing instructions executable by the processor;

[0021] Wherein, the processor runs the executable instructions to implement the steps of the method as described in the first aspect above.

[0022] According to a fifth aspect of the present application, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method as described in the first aspect above are implemented.

[0023] According to a sixth aspect of the present application, there is provided a computer program and / or instruction, and when the computer program and / or instruction is executed by a processor, the steps of the method as described in the first aspect above are implemented.

[0024] The technical solutions provided by the present application can at least include the following beneficial effects:

[0025] Through the above embodiments, after obtaining the environmental data of the current environment where the vehicle is located, the risk prediction model can be first used to predict the high-risk driving factors in the current environment based on the environmental data and output the corresponding high-risk driving content; then the environmental data and the high-risk driving content are respectively input into the intelligent driving model, and the comprehensive driving control instruction output after the inference of the model is obtained; finally, the vehicle is controlled to drive according to the comprehensive driving control instruction.

[0026] Based on the traditional intelligent driving model, this solution introduces a new risk prediction model to participate in the composition of the risk prediction model, and uses this model to predict the high-risk driving factors in the current environment and output the corresponding high-risk driving content. It can be understood that based on the risk prediction / inference ability of the risk prediction model, the high-risk driving content extracted by this model can be regarded as the high-dimensional features related to driving in the current environment, while the intelligent driving model can extract the low-dimensional features related to driving in the current environment based on the environmental data. Therefore, the comprehensive driving control instruction output after the intelligent driving model infers according to the above high / low dimensional features actually comprehensively considers various multi-dimensional and various types of driving-related factors in the current environment, and thus can effectively cope with high-risk driving factors and avoid safety accidents, significantly improving the intelligent driving level of the vehicle.

[0027] It can be seen that the intelligent driving system of the present application makes up for the defects of the traditional intelligent driving model in incomplete or in-depth understanding of driving scenarios (especially long-tail or complex scenarios) by introducing a risk prediction model. With the powerful prediction / inference ability provided by the risk prediction model, whether in complex driving scenarios with more elements or rare driving scenarios (such as the aforementioned corner case, etc.), the new risk prediction model of the present application can effectively predict the possible risks in real time based on the environmental data obtained by the vehicle and generate the corresponding comprehensive driving control instructions to control the vehicle to effectively respond, so as to ensure that the vehicle can quickly and accurately respond to direct risks and potential risks in the above driving scenarios, effectively improving the safety and decision-making accuracy of vehicle intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the hardware architecture of a risk prediction model shown in an embodiment of the present application.

[0029] Figure 2 It is a schematic diagram of the software architecture of a risk prediction model shown in an embodiment of the present application.

[0030] Figure 3 It is a flowchart of a driving control method of a vehicle shown in an exemplary embodiment.

[0031] Figure 4 It is a schematic diagram of a multi-lane driving scenario shown according to an exemplary embodiment.

[0032] Figure 5 It is a schematic diagram of the environment where a vehicle is located shown according to an exemplary embodiment.

[0033] Figure 6 It is a schematic structural diagram of an electronic device shown according to an exemplary embodiment.

[0034] Figure 7 It is a block diagram of a driving control device for a vehicle shown according to an exemplary embodiment. Detailed implementation manners

[0035] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] It should be noted that: in other embodiments, the steps of the corresponding methods are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0037] In view of the technical problems existing in the related art, the present application proposes a brand-new intelligent driving system implemented based on a risk prediction model and a driving control method for a vehicle implemented by this system.

[0038] From a software perspective, the intelligent driving system includes a risk prediction model, an intelligent driving model, and a vehicle control module, where

[0039] The risk prediction model is used to predict high-risk driving factors in the current environment based on the environmental data of the current environment where the vehicle is located and output corresponding high-risk driving content;

[0040] The intelligent driving model is used to reason based on the environmental data and the high-risk driving content and output an inferred comprehensive driving control instruction;

[0041] The vehicle control module is used to control the vehicle to drive according to the comprehensive driving control instruction.

[0042] Among them, the intelligent driving model can be a rule-based driving model only (such models have clear rules and are suitable for dealing with deterministic simple driving scenarios when used alone), or it can also be a neural network-based driving model only (such models have strong adaptability and can handle some complex driving scenarios when used alone), or it can also be a hybrid driving model based on both rules and neural networks (such models have both the interpretability of rules and the adaptability of neural networks). The construction, training, and deployment methods of any of the above types of intelligent driving models can be referred to the records in related technologies, and the embodiments of the present application do not limit this.

[0043] The risk prediction model can be built based on any form of neural network framework and trained in a supervised or unsupervised manner. Taking supervised training as an example, sample data corresponding to driving cases can be obtained in advance. Any sample data contains sample environmental data collected during vehicle driving and corresponding sample driving risk content, and this sample driving risk content is used as the label of the sample environmental data. Through the above training, the risk prediction model can accurately and efficiently use environmental data to predict high-risk driving factors such as pedestrians, surrounding vehicles, and signs in the vehicle's environment and output the high-risk driving content corresponding to these factors. In one embodiment, the risk prediction model can adopt a Vision-Language Model (VLM), which is a multi-modal artificial intelligence model that combines the capabilities of Computer Vision (CV) and Natural Language Processing (NLP). It can simultaneously understand and process image (or video) and text information and establish associations between the two to achieve more complex tasks. Using VLM to implement the risk prediction model enables the model to relatively more accurately predict high-risk driving factors in the current environment from multiple dimensions based on multi-modal environmental data and output the high-risk driving content corresponding to these factors.

[0044] This application combines traditional intelligent driving models with risk prediction models such as VLM, giving full play to the respective advantages of both: based on the decision-making ability and safety guarantee of traditional intelligent driving models themselves, providing a fallback for intelligent driving strategies to ensure the lower limit of the performance and decision-making ability of the intelligent driving system; at the same time, by integrating the advantages of VLM in reasoning ability and multi-modal fusion, it can achieve a comprehensive understanding of the driving environment and intelligent decision-making support, thereby providing a higher-dimensional and further risk prediction for traditional intelligent driving models, effectively improving the upper limit of the decision-making ability and intelligent level of the intelligent driving system, and greatly enhancing the comprehensive performance of the system.

[0045] In addition, the intelligent driving implemented by the intelligent driving system described in this application can be assisted driving or autonomous driving, and this application does not limit the level of this intelligent driving (such as L3, L4, etc.). However, it should be noted that the intelligent driving implemented by this system should comply with the relevant laws, regulations and standards of the corresponding countries and regions (such as the sales place and / or the place of use of the vehicle), and provide corresponding operation entrances for relevant personnel (such as drivers, etc.) for them to choose to authorize or refuse to use.

[0046] The specific working methods of the above-mentioned various models and modules can be found in the description of the embodiments below, and will not be elaborated here.

[0047] Figure 1 is a schematic diagram of the hardware architecture of an intelligent driving system shown in an embodiment of this application. As Figure 1 shown, from a hardware perspective, this system can only include vehicle 11, or can also include vehicle 11 and server 13 at the same time. If the intelligent driving system only includes vehicle 11, at this time, the risk prediction model, the intelligent driving model and the vehicle control module are all deployed in vehicle 11, such as all deployed in the domain controller 111 of vehicle 11. Exemplarily, the risk prediction model and the intelligent driving model can be deployed in the domain controller of the intelligent driving domain of vehicle 11, and the vehicle control module can be deployed in the domain control of the body domain and / or the power domain of vehicle 11. If the intelligent driving system includes vehicle 11 and server 13 at the same time, at this time, at least one of the risk prediction model and the intelligent driving model can be deployed in server 13, and the vehicle control module can be deployed in vehicle 11 (such as in the aforementioned domain controller), which will not be elaborated.

[0048] In addition to the aforementioned domain control, corresponding information collection devices can also be installed on vehicle 11. According to the collected data / installation location, the above-mentioned information collection devices can be divided into in-vehicle collection devices and out-of-vehicle collection devices. Exemplarily, the in-vehicle collection devices can include at least one of in-vehicle cameras, in-vehicle microphones, in-vehicle biosensors (which can be used to identify various biometric features such as the body temperature, fingerprints, irises, etc. of in-vehicle occupants 12), in-vehicle odor sensors, etc. The in-vehicle collection devices can include at least some of the devices in the DMS (Driver Monitoring System, driver monitoring system) that detect the attention and emotions of the driver. The out-of-vehicle collection devices can include at least one of visual cameras, millimeter wave radars, lidars, infrared sensors, out-of-vehicle microphones, etc. Among them, the above-mentioned information collection devices can be installed at different positions of the vehicle according to their functions. Taking out-of-vehicle sensors as an example, such as Figure 1As shown, the visual camera 112 is assembled at the front of the vehicle, the visual camera 113 is assembled at the rear of the vehicle, and the lidar 114 is assembled at a position near the front of the roof, which will not be elaborated here. Of course, the embodiments of the present application do not limit the specific parameters such as the type, quantity, size, and assembly position of the above information collection devices.

[0049] In addition, the vehicle exterior collection device may further include a wireless communication module, a navigation and positioning module, etc. The wireless communication module may include at least one of a radio frequency module (such as 3G, 4G, and / or 5G modules, etc.), a BLE (Bluetooth) module, a Wi-Fi module, an NFC (Near Field Communication) module, a NearLink module, etc.; the navigation and positioning module may include obtaining navigation and positioning related information of the vehicle by using systems such as GPS (Global Positioning System), Beidou, GLONASS (Global Navigatsion Satellite System), the "Galileo" satellite navigation system, the quasi-zenith satellite system, etc.

[0050] In addition to directly collecting information inside and outside the vehicle through the information collection module assembled on the vehicle 11 itself, the vehicle 11 can also communicate with the electronic devices used by the vehicle occupants 12 to obtain the environmental data directly collected by these electronic devices. For example, the vehicle 11 can establish a wireless connection with the electronic device through the aforementioned wireless communication module and receive the information inside and outside the vehicle (such as images of external obstacles, videos for showing traffic congestion outside the vehicle, etc.) and the occupant's physical state information (such as heartbeat, body temperature, blood sugar / oxygen content, etc.) reported after collection. Exemplarily, the electronic devices may include, but are not limited to, mobile phones, PCs (Personal Computers), workstations, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc. One or more embodiments of the present application do not limit this.

[0051] In addition, the vehicle can establish a network connection with the remote server 13 through the aforementioned wireless communication module to perform data interaction with the server 13. For example, if the risk prediction model is deployed in the server 13, the vehicle 11 can send the environmental data collected by itself to the server 13 and receive the high-risk driving content output by the model after predicting high-risk driving factors using the environmental data. For another example, if the risk prediction model and the intelligent driving model are deployed in the server 13, the vehicle 11 can send the environmental data collected by itself to the server 13 and receive the comprehensive driving control instructions output after the intelligent driving model infers based on the environmental data and the high-risk driving content (output by the risk prediction model based on the prediction of environmental data). Among them, the server 13 can be a physical server including an independent host, or can also be a virtual server, cloud server, etc. hosted by a host cluster. In addition, the embodiments of the present application do not limit the number, type, and specific interaction method with the vehicle of the server 13. As for the network 10 for interaction between the vehicle 11 and the server 13, it can be specifically selected to use a corresponding type of wireless network to implement communication based on the communication methods supported by the corresponding devices, and this specification does not limit this.

[0052] In addition, the vehicle (such as vehicle 11) described in the present application, in terms of its functional form, can be a pickup truck, sedan, SUV (Sport Utility Vehicle), RV, truck, etc.; in terms of its power form, it can be a fuel vehicle or a new energy vehicle (such as a hybrid vehicle, pure electric vehicle, hydrogen energy vehicle, methanol energy vehicle, etc.), and the present invention does not limit the specific form of the vehicle. In addition, the occupant 12 in the cockpit can be the driver sitting in the driver's seat, or can also be at least one passenger sitting in other positions, and the present application also does not limit the number of members and their sitting positions.

[0053] Figure 2 It is a schematic software architecture diagram of a risk prediction model shown in the embodiments of the present application, as Figure 2 shown, the system includes a risk prediction model, an intelligent driving model, and a vehicle control module. The environmental data obtained by the on-vehicle information collection device can be simply divided into basic environmental data and extended environmental data. Among them, the intelligent driving model can infer based on the input extended environmental data such as out-of-vehicle image data, out-of-vehicle sound and light signal data, and in-vehicle occupant behavior data to predict high-risk driving factors, and then output the corresponding high-risk driving content to the intelligent driving module; while the risk prediction model can infer based on the input basic environmental data and the high-risk driving content, and output the corresponding comprehensive driving control instructions to the vehicle control module, so that the latter controls the vehicle to drive according to the instructions.

[0054] In addition, the intelligent driving system may further include a data storage module for collecting relevant data during the operation of the intelligent driving system (such as the aforementioned environmental data, comprehensive driving control instructions, etc.) and generating a recording file in multimedia form accordingly. For the specific process, reference may be made to the embodiments below, which will not be elaborated here for the time being.

[0055] Figure 3 is a flowchart of a driving control method for a vehicle shown in an embodiment of the present application. This method is applied to the aforementioned intelligent driving system. As Figure 3 shown, this method includes the following steps 302 to 306.

[0056] Step 302: Obtain the environmental data of the current environment where the vehicle is located, and use a risk prediction model to predict the high-risk driving factors in the current environment based on the environmental data and output the corresponding high-risk driving content.

[0057] Step 304: Input the environmental data and the high-risk driving content into an intelligent driving model respectively, and obtain the comprehensive driving control instructions output after the model inference.

[0058] First of all, it should be noted that the current environment where the vehicle of the present application is located is not limited to the external environment of the vehicle (such as the road where the vehicle is located, the parking lot, etc.), but may also include the internal environment of the vehicle (such as the position, behavior, state, etc. of the passengers in the vehicle). After the vehicle is powered on and / or the engine is started, information collection devices such as sensors installed on the vehicle can start to collect the corresponding environmental data for the current environment in real time. After the intelligent driving system starts to run (such as the driver triggers the opening of the intelligent driving function), the intelligent driving system can obtain the above environmental data in real time and use this data for subsequent processing to control the vehicle to drive. Of course, at least part of the above information collection devices can also only collect environmental data in real time after the intelligent driving system starts to run, and do not collect environmental data when the system is not started, so as to avoid ineffective collection of environmental data to save resources such as the power and storage of the vehicle.

[0059] In the embodiments of the present application, the high-risk driving factors in the current environment may include factors that may pose potential safety hazards to the normal driving of the vehicle or even cause traffic accidents. For example, when the vehicle (hereinafter referred to as the host vehicle) is driving normally in the current lane, the vehicle in front in the same lane may suddenly brake, resulting in the host vehicle rear-ending the vehicle in front; the vehicle behind may accelerate and rear-end the host vehicle; other vehicles in adjacent lanes may change lanes and cause scratches with the host vehicle, etc. These vehicles are all high-risk driving factors for the host vehicle. For another example, when the vehicle is driving in the current lane without a guardrail, two-wheeled vehicles (such as bicycles, electric scooters, etc.) driving on the adjacent non-motorized lane, children playing and chasing on the sidewalk, etc. may suddenly enter the current lane and collide with the host vehicle. Therefore, they are also high-risk driving factors for the host vehicle. For such risk factors, the intelligent driving system of the present application can avoid them by controlling the vehicle to drive according to the comprehensive driving control instruction, so as to avoid safety accidents as much as possible.

[0060] Alternatively, the high-risk driving factors in the current environment may also include factors that may cause a decrease in the driving efficiency of the vehicle, an increase in driving energy consumption, and / or violent driving actions, etc. For example, when the host vehicle is driving normally in the current lane, the vehicle in front in the same lane may be driving slowly (i.e., the driving speed is significantly lower than the speed limit of the current lane), and there are no other vehicles in the adjacent fast lane (i.e., the adjacent lane with a speed limit not lower than the current lane) or there are no other vehicles within the lane change distance (i.e., the distance that can ensure the host vehicle to complete the lane change operation normally). In this case, if the host vehicle continues to follow the vehicle in front slowly, it will obviously limit the driving speed of the host vehicle and increase the driving time / fuel consumption. At this time, the vehicle in front can be regarded as a high-risk driving factor for the host vehicle. For another example, when the host vehicle is queuing up to wait for a straight-through at the intersection ahead in the current lane, if the traffic light has turned green but the vehicle in front in the current lane is severely congested and driving slowly, and there are no vehicles (or fewer vehicles) in the right lane and straight-through is allowed, then if the host vehicle continues to creep or even stop and wait behind the vehicle in front, it will obviously limit the driving speed of the host vehicle and increase the driving time / fuel consumption. At this time, the vehicle in front can also be regarded as a high-risk driving factor for the host vehicle. For such risk factors, the intelligent driving system of the present application can avoid them by controlling the vehicle to drive according to the comprehensive driving control instruction, so as to improve the driving efficiency as much as possible and achieve smooth / low-energy consumption driving.

[0061] The environmental data is at least one type of data collected for the current environment and capable of characterizing the environment, such as the external vehicle images / videos collected by an external vehicle vision camera, the point cloud data detected by a lidar, the honking sound of the following vehicle collected by an external vehicle microphone, the expressions of occupants such as the driver detected by an internal vehicle camera, the voices of occupants such as the driver inside the vehicle collected by an internal vehicle microphone, etc., which will not be elaborated here. Based on the above current environment, the risk prediction model can predict whether there are high-risk driving factors in the current environment and extract the high-risk driving content of these factors, such as the expected speed and braking probability of the vehicle ahead, the expected trajectory and lane-changing probability of the vehicle in the adjacent lane within a certain period in the future, the expected trajectory and intrusion probability of the bicycle traveling on the front non-motor vehicle lane, etc., which will not be elaborated here.

[0062] In one embodiment, since both the risk prediction model and the intelligent driving model in the intelligent driving system can receive environmental data and process them separately, and the environmental data received by the two may be the same or different, for the sake of easy distinction, the environmental data input into the risk prediction model can be called basic environmental data, and the environmental data input into the intelligent driving model can be called extended environmental data. Therefore, the risk prediction model can predict high-risk driving factors in the current environment based on the extended environmental data and output the corresponding high-risk driving content, while the intelligent driving system can input the basic environmental data and the high-risk driving content into the intelligent driving model for it to reason based on the two and output the final comprehensive driving control instruction. In this way, different environmental data can be accurately input into the corresponding models, which helps the risk prediction model achieve more accurate risk prediction and helps the intelligent driving model generate more accurate and comprehensive control instructions.

[0063] It should be noted that the environmental data collected by any information collection device may be only input into the risk prediction model (such as Figure 2 the external vehicle waveform data collected by the millimeter-wave radar shown, the external vehicle point cloud data collected by the lidar, etc.), may also be only input into the intelligent driving model (such as Figure 2 the internal vehicle images collected by the internal vehicle vision camera shown, the external vehicle sound signals collected by the external vehicle microphone, the internal vehicle sound signals collected by the internal vehicle microphone, etc.), or may be respectively input into the risk prediction model and the intelligent driving model (such as Figure 2 the external vehicle images collected by the external vehicle vision camera shown, the map data obtained by the vehicle, the current positioning data, etc.). Which data is input into the above two models respectively (that is, which model the collected any environmental data is input into) can be determined according to actual needs during the model training and use phases, and the embodiments of the present application do not limit this.

[0064] As mentioned above, the risk prediction model and / or the intelligent driving model can be deployed and run on a server in the cloud (such as Figure 1In the server 11) shown, it can also be deployed and run locally on the vehicle. It can be understood that if it is deployed on the server, the rich computing / storage / network and other resources of the server can be fully utilized to run a full-scale model with rich functions and a large number of parameters / size, so as to achieve an intelligent driving solution with high accuracy and rich functions. If it is deployed locally on the vehicle, a relatively lightweight in-vehicle model obtained after optimizing and compressing the full-scale model (such as model pruning, model quantization, knowledge distillation, etc.) can be deployed, so as to effectively shorten the data transmission time while ensuring the inference accuracy and basic functions, thereby greatly accelerating the inference speed of the intelligent driving system and reducing the safety hazards that may be caused by control delay or even signal loss. Of course, when the resources carried by the vehicle are sufficient, the full-scale model can also be deployed and run locally on the vehicle, and the embodiments of the present application do not limit this.

[0065] It can be understood that if both the risk prediction model and the intelligent driving model are deployed locally on the vehicle, various environmental data can be directly input into the corresponding models. If both the risk prediction model and the intelligent driving model are deployed in the server, the environmental data can be uniformly sent to the server and the comprehensive driving control instructions returned by the latter can be received. If the risk prediction model is deployed in the server and the intelligent driving model is deployed locally on the vehicle, the extended environmental data can be sent to the server, and the basic environmental data and the high-risk driving content returned by the server can be input into the intelligent driving model; conversely, if the risk prediction model is deployed locally on the vehicle and the intelligent driving model is deployed in the server, the extended environmental data can be input into the intelligent driving model, and the basic environmental data and the high-risk driving content output by the model can be sent to the server together, and the comprehensive driving control instructions returned by the server can be received, which will not be elaborated. Of course, the specific deployment locations of the above models can be reasonably set according to actual situations such as latency requirements and resource requirements, and the embodiments of the present application do not limit this. The following embodiments will be described by taking both models being deployed locally on the vehicle as an example.

[0066] In one embodiment, the intelligent driving model can predict high-risk driving factors based on the type of environmental data and output corresponding high-risk driving content. For example, when the environmental data includes external vehicle image data, this data can be input into the risk prediction model to obtain the high-risk driving content of the external high-risk driving factors output after the model's inference. For example, the model can identify high-risk vehicles that may change lanes outside the vehicle based on the external images or videos collected by the external visual camera, and output the expected trajectory and lane-changing probability of the vehicle, identify pedestrians who may run a red light and output their red-light running probability and driving trajectory at the intersection, identify children who may enter the current lane and output their intrusion probability and walking trajectory after intrusion, etc. Another example is when the environmental data includes external vehicle sound and light signal data, this data can be input into the risk prediction model to obtain the high-risk driving content of the external high-risk driving factors output after the model's inference. For example, the model can predict possible obstacles or car accidents on the road ahead based on the light signals of surrounding vehicles collected by the external visual camera and / or the honking signals of surrounding vehicles collected by the external microphone, and output the impact probability on the host vehicle, predict possible special road conditions such as fog / ice / potholes on the road ahead and output the impact degree on the host vehicle's trajectory, predict possible yielding requirements of the following vehicle (such as special vehicles) and output the yielding direction and urgency, etc. Another example is when the environmental data includes in-vehicle occupant behavior data, this data can be input into the risk prediction model to obtain the high-risk driving content of the in-vehicle high-risk driving factors output after the model's processing. For example, the model can predict possible collisions between the host vehicle and the object ahead based on the facial expressions and body movements of the driver collected by the in-vehicle visual camera and / or the voices of the driver or occupants collected by the in-vehicle microphone, and output the collision position and probability, predict possible physical abnormalities of the host vehicle driver and output possible abnormal vehicle controls and their probabilities, etc. Through the above methods, the risk prediction model can fully and accurately predict the possible high-risk driving factors in the current environment and output corresponding high-risk driving content, which helps the subsequent risk prediction model to output more accurate driving strategies.

[0067] It should be noted that the above-mentioned various in-vehicle and out-of-vehicle information may belong to the personal sensitive information of users / occupants or the sensitive information of relevant organizations / institutions. The above-mentioned information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by users or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0068] In one embodiment, the intelligent driving model may include a perception processing module, a path planning module, and an instruction generation module (such as Figure 2 As shown). The perception processing module is used to extract basic features of the input environmental data (such as the aforementioned basic environmental data), the path planning module is used to plan the vehicle's driving path at least according to the basic features, and the instruction generation module is used to generate a driving control instruction for the driving path so as to control the vehicle to travel along the path.

[0069] For the above-mentioned multiple modules, when high-risk driving content is input into the intelligent driving model, the risk prediction model can input different forms of content into different modules in a targeted manner to achieve corresponding intelligent driving performance. For example, the high-risk driving content output by the risk prediction model may include predicted semantic information, and the information can be input into the aforementioned path planning module at this time, so that the module determines the target path that matches the high-risk driving factor from the various candidate paths predicted based on the environmental data, and enables the instruction generation module to output a comprehensive driving control instruction for the target path. Among them, the predicted semantic information can be in any form that can be recognized by the path planning module, such as text, feature vector, token, etc., and the embodiment of the present application does not limit this. After being processed by the path planning module, the above-mentioned predicted voice information will affect the weight value (that is, the possibility of being selected) of each candidate path, thereby affecting the target path. In this solution, the path planning module can generate multiple candidate paths for the vehicle at the current moment (i.e., the paths that the vehicle can choose to travel after the current moment) based on the aforementioned basic environmental data according to its own rules / logic, and the risk prediction model directly inputs the predicted semantic information (related to the risk factors at the current moment) extracted by itself into the path planning module, so that the module can select the target path that can avoid the above risk factors as much as possible from multiple candidate paths in combination with the predicted semantic information, thereby improving the safety and driving quality during intelligent driving. In this way, risk prediction results can be introduced in the process of path planning by the intelligent driving model, so that the path planning results can effectively avoid risks.

[0070] For another example, the high-risk driving content output by the risk prediction model may include a prediction control signal. At this time, this signal can be input into the aforementioned instruction generation module, so that the instruction generation module adjusts the driving control instruction for the target path according to the prediction control signal and outputs an adjusted comprehensive driving control instruction, where the target path and its driving control instruction are predicted based on the environmental data. In this solution, the perception processing module first extracts the basic features of the current environment based on the environmental data (such as the aforementioned basic environmental data). Therefore, the target path planned by the path planning module and the driving control instruction generated by the instruction generation module for this path can both be regarded as being obtained based on the environmental data. On this basis, after receiving the prediction control signal output by the risk prediction model, the instruction generation module can directly generate a comprehensive driving control instruction for the target path according to this signal, so as to fully reflect the influence of the high-risk driving factors corresponding to the prediction control signal on the driving control instruction. In this way, the risk prediction result can be directly introduced when the intelligent driving model generates a control instruction for the target path, thereby significantly shortening the influence process of the risk prediction result on the final output instruction of this model and enhancing the influence intensity, which helps to achieve emergency control of the vehicle in scenarios with a relatively high degree of risk severity and improve the response accuracy and timeliness for serious risks.

[0071] In one embodiment, the intelligent driving system can predict possible high-risk driving content based on the driving conditions of other vehicles around the vehicle, and control the vehicle to avoid it in advance to improve the safety of intelligent driving, or control the vehicle to actively change the current driving state to actively respond to the above factors, thereby improving the quality of intelligent driving. For example, when the environmental data includes the current driving states of other vehicles around the vehicle, the high-risk driving content may include the expected driving trajectory information of the other vehicles. Among them, the current driving state of any other vehicle may include the vehicle's current speed, current acceleration, current driving direction, the distance from the host vehicle, light signals (such as turn signal, hazard warning signal, etc.). Based on this, the risk prediction model can predict the expected driving trajectory information of other vehicles, such as whether to change lanes into the host vehicle's lane and its expected lane-changing trajectory, whether to brake and the acceleration and braking time when braking. Further, the comprehensive driving control instruction output by the risk prediction model can be a targeted following instruction, lane-changing instruction, deceleration instruction, etc. Taking the lane-changing instruction as an example, at this time, the vehicle can be controlled to change lanes into the adjacent lane according to this lane-changing instruction to avoid the above risks.

[0072] It is understandable that when the traffic volume in the lanes on both sides is higher than the speed of the vehicle in front; or when the traffic volume in the adjacent lanes is small and the vehicle in front maintains the current speed for a long time, the traditional intelligent driving model will often control the vehicle to continue to follow the vehicle and drive slowly, so the driving experience is not flexible enough, which may lead to low driving efficiency. However, through the above method, the intelligent driving system can comprehensively analyze many factors such as the speed, acceleration, distance from the vehicle and traffic volume of other vehicles on both sides, and predict the expected driving conditions of surrounding vehicles in the future, so as to generate lane change trajectories while ensuring safety; further, it can evaluate the timing and suitability of lane change for different lane change trajectories, and then make comprehensive decisions and decisively issue corresponding lane change instructions to control the vehicle to complete the lane change safely and efficiently.

[0073] like Figure 4 As shown, it is assumed that the vehicle V1 is following the vehicle V2 in front of it in lane R2 (i.e., following the vehicle), there is a vehicle V2 in the left lane R1 in front of the vehicle on the left, a vehicle V3 in the right lane R2 in the right rear of the vehicle on the right, and a vehicle V4 in the right lane R2 in front of the vehicle on the right. It is assumed that the path planning module in the risk prediction model running locally on V1 plans three candidate paths: the path L1 for changing lanes to the left, the path L2 for continuing to follow the vehicle, and the path L3 for changing lanes to the right. It should be noted that the path planning module described in the embodiment of the present application is used to plan the driving path of the vehicle, which can be a complete path from the departure point to the destination determined in combination with map, navigation and other information, or it can be a path corresponding to simple operations such as turning, changing lanes, and avoiding during driving; in other words, the embodiment of the present application does not limit the length and degree of refinement (i.e., granularity) of the driving path planned by the intelligent driving model. If the risk prediction model predicts that lane V4 will change lanes into lane R2, the vehicle V1 can be controlled to slow down (at this time, L2 is selected as the target path) or change lanes to R1 along path L1 (at this time, L1 is selected as the target path) to avoid rear-ending vehicle V4 that changes lanes to R2.

[0074] For another example, when the environmental data includes vehicle congestion information in the first adjacent lane of the current lane (which can be any lane adjacent to the current lane), the high-risk driving content may include expected lane-changing information of other vehicles in the first adjacent lane changing into the current lane. In this case, the vehicle control module can control the vehicle to decelerate according to the deceleration instruction issued by the intelligent driving model, or control the vehicle to change lanes into the second adjacent lane of the current lane according to the lane-changing instruction issued by the model. The intelligent driving system predicts the congestion situation ahead in the adjacent lane by continuously observing data such as the driving states of other surrounding vehicles, and through in-depth reasoning about the surrounding scenarios, predicts the speeds, accelerations, and lane-changing behaviors of other vehicles, enabling the system to predict in advance the possible risks of the vehicles ahead, so as to take measures such as decelerating in advance or changing lanes to avoid. For example, when sudden congestion occurs in an adjacent lane, even if there are no obvious lane-changing actions, turn signals, or intrusions into the own lane by the vehicles in the front area of that lane, the system will predict that there may be vehicles changing lanes (i.e., intruding) into the own lane ahead, which may cause the vehicle ahead to brake suddenly. If the own vehicle still travels at the current speed, there is a risk of rear-ending. In this case, the vehicle can be controlled to actively reduce the speed, maintain a safe distance from the vehicle ahead, or choose to change lanes to the other lane. As Figure 4 shown, if it is predicted that congestion is about to occur ahead of vehicle V4 in the right lane R3, the own vehicle V1 can be controlled to change lanes to the left lane R1 along path L2 to avoid possible congestion in lane R2 or sudden braking of the vehicle V2 ahead.

[0075] In one embodiment, considering that there may be high-risk driving factors only at some positions in the space around the vehicle (i.e., the risk prediction model may only identify high-risk driving factors at some positions), obviously the safety / inefficiency risks of the objects / factors at these positions are lower than those at other positions. Therefore, in order to more accurately predict the possible risk behaviors of the high-risk driving factors at these positions in the future, more processing resources can be allocated to these positions. For example, when the risk prediction model identifies the high-risk driving factors, the intelligent driving system can control the on-vehicle information acquisition device to collect environmental data related to the high-risk driving factors at a higher frame rate (such as increasing the sampling frame rate from 10 fps to 30 fps), so as to be able to process the high-risk driving factors at a faster frame rate and reduce the prediction inaccuracy caused by processing delay. Of course, this solution requires that the information acquisition device can operate at different frame rates and its operating frame rate can be controlled and adjusted by the intelligent driving system.

[0076] To ensure that the environmental data collected at a higher frame rate can be smoothly and effectively processed to obtain more accurate processing results, more storage, computing, and other resources can also be allocated to the aforementioned risk prediction model and intelligent driving model. Additionally, the corresponding number of resources can be scheduled in real time according to the number, type, and data volume of high-risk driving factors for the above models to use, so as to improve the efficient and full utilization of the vehicle's local resources. Moreover, when high-risk driving factors are predicted, the driver can be reminded by means of highlighting or voice broadcasting, etc.

[0077] Step 306, control the vehicle to travel according to the comprehensive driving control instruction.

[0078] As described above, based on the environmental data collected and the aforementioned high-risk driving content output by the risk prediction model, the intelligent driving model can perform reasoning and output the corresponding comprehensive driving control instruction. At this time, the vehicle control module can control the vehicle to travel according to this instruction.

[0079] As can be seen from the foregoing embodiments, in this solution, on the basis of the traditional intelligent driving model, a new risk prediction model is introduced to participate in the formation of the risk prediction model. This model is used to predict the high-risk driving factors in the current environment and output the corresponding high-risk driving content. It can be understood that based on the risk prediction / reasoning ability of the risk prediction model, the high-risk driving content extracted by this model can be regarded as the high-dimensional features related to driving in the current environment, while the intelligent driving model can extract the low-dimensional features related to driving in the current environment based on the environmental data. Therefore, the comprehensive driving control instruction output by the intelligent driving model after reasoning according to the above high / low-dimensional features actually comprehensively considers various multi-dimensional and various types of driving-related factors in the current environment, and thus can effectively cope with high-risk driving factors to avoid safety accidents, significantly improving the intelligent driving level of the vehicle.

[0080] It can be seen that the intelligent driving system of this application makes up for the defect that the traditional intelligent driving model has an incomplete or in-depth understanding of driving scenarios (especially long-tail or complex scenarios) by introducing a risk prediction model. With the powerful prediction / reasoning ability provided by the risk prediction model, whether in a complex driving scenario with more elements or a rare driving scenario (such as the aforementioned corner case, etc.), the new risk prediction model of this application can effectively predict the possible risks in real time based on the environmental data obtained by the vehicle and generate the corresponding comprehensive driving control instruction to control the vehicle to effectively respond, so as to ensure that the vehicle can quickly and accurately respond to direct risks and potential risks in the above driving scenarios, effectively improving the safety and decision-making accuracy of the vehicle's intelligent driving.

[0081] In one embodiment, when the high-risk driving factors include high-risk personnel on at least one side in front of the vehicle driving direction and / or associated personnel in a high-risk area on the at least one side, the high-risk driving content may include the expected trajectory and movement probability of the high-risk personnel. At this time, the vehicle driving can be controlled according to comprehensive driving control instructions of corresponding types. For example, the vehicle can be controlled to stop and wait according to a waiting instruction, to drive at a low speed according to a slow-down instruction, and / or to decelerate according to a deceleration instruction. Among them, at least one side in front of the vehicle driving direction can be the left side and / or the right side of the current lane where the vehicle is currently located (hereinafter referred to as the current lane). The high-risk personnel are those with a probability of traffic violations / breaches (such as running a red light, not stopping to give way, etc.) greater than a probability threshold. For example, when the vehicle is waiting to go straight through an intersection, high-risk professional practitioners (such as food delivery riders, couriers, etc.) and / or special personnel (such as children, the elderly, etc.) appearing at the intersection have a relatively high probability of running a red light, invading road markings, etc., so the risk is relatively high. And the associated personnel in the high-risk area can be school-associated children (that is, slow down when approaching a school ahead), hospital-associated patients, nursing home-associated elderly people, etc. Of course, the above-mentioned high-risk personnel and high-risk areas can be reasonably defined and set according to the actual situation of the physical world, and the embodiments of the present application do not limit this. If the above-mentioned high-risk personnel are detected or predicted, it is defaulted that they may pose a certain degree of potential safety hazard to the vehicle driving. Therefore, at this time, the intelligent driving model can predict and output the expected trajectory and corresponding movement probability of the high-risk personnel, and make a comprehensive decision according to the interaction between the expected trajectory and the driving trajectory of the vehicle (in a future period of time).

[0082] Exemplarily, if a rolling ball is detected ahead and its rolling trajectory intersects with the vehicle's trajectory at a certain future moment, it can be inferred that there may be children invading the current lane, and the vehicle can be controlled to decelerate to avoid collision. For another example, if the map shows that there is a school or kindergarten on the right side of the road ahead, at this time, multi-modal data such as map information, the vehicle's driving state, and visual images can be combined to determine that the school gate is facing the current lane where the vehicle is located. At this time, it is possible to detect in real time and focus on whether there are pedestrians crossing the road at the school gate, and change lanes in advance to a lane that is relatively farther away from the school gate.

[0083] In one embodiment, when the risk prediction model identifies high-risk driving factors in the current environment, the system can also output risk warning information for the high-risk driving factors to the vehicle occupants, so that the vehicle occupants (especially the driver) can timely learn about these factors, enhance their perception of the current environment, and facilitate the occupants to respond to risks in advance. Among them, the above-mentioned risk warning information can be output to the user in any suitable form such as a display screen, playing audio (such as playing a warning voice through the in-vehicle microphone), and / or controlling vibration (such as vibrating the steering wheel). Taking the display screen as an example, a screen containing risk warning information can be displayed through an in-vehicle screen or a HUD (Head-Up Display) device. For example, a complete bird's-eye view (BEV) screen of the space where the vehicle is currently located can be displayed on the vehicle's center control screen or instrument panel screen, or some key risk warning information can be displayed through the HUD. Among them, the above-mentioned screen can include many contents such as the current state of the vehicle, surrounding lanes / vehicles / pedestrians / marking lines, etc., for the user to view.

[0084] Taking the bird's-eye view screen as an example, as Figure 5 shown, this screen displays each content in a three-dimensional form. For example, the vehicle 501 itself is shown in the middle of the screen, and the surrounding vehicles, road elements (lane lines, zebra crossings, traffic lights, etc.), and nearby people (such as the person riding an electric bicycle in the left rear, the child riding a scooter / roller skates on the outside of the road in the left front, the baby walking on the zebra crossing ahead, etc.) are shown according to their actual spatial position relationships. At this time, the traffic light is green. According to the driving direction of the vehicle itself and the driving direction and actions of the child riding the roller skates, the intelligent driving system can predict that the child is likely to slide into the intersection and run a red light. Therefore, it is determined as a high-risk driving factor at this time. In this regard, the child can be highlighted with a red indication box, and prompt text such as "There is a child riding a roller skates, please pay attention to safety" can be prominently displayed through a text box, which will not be elaborated here.

[0085] In one embodiment, if the aforementioned high-risk person is located at the intersection in front of the vehicle, the environmental data may include at least one of the signal light state at the intersection (such as red / green light state, remaining duration of the red light, etc.), camera state (whether there is a camera, camera position, whether the camera is turned on, etc.), and / or traffic controller state (such as the position of the traffic police, traffic police gestures, etc.). The above data can be collected through the in-vehicle information collection device, or directly obtained from the map or navigation information. The embodiments of the present application do not limit this.

[0086] Taking a food delivery rider commonly seen at intersections as an example, traditional intelligent driving models can usually only predict the rider's trajectory based on their current motion state. If the food delivery rider stops and does not move at the current moment, it is difficult to judge their driving intention, so a collision may occur with the rider after the host vehicle enters the intersection. The risk prediction model of this solution can, when detecting the appearance of a food delivery rider in the waiting area of the intersection, further predict their trajectory and the probability of them crossing the road or running a red light. The following is an explanation in combination with specific scenarios:

[0087] Scenario 1: When it is recognized that the traffic signal corresponding to the lane where the host vehicle is located turns green, if the extension line of the front of the food delivery rider intersects with the driving trajectory line of the host vehicle (i.e., the vehicle) in the middle part of the road, and other vehicles in the adjacent lane in the same direction as the host vehicle have not started (still in a stopped state), at this time, the risk prediction model may infer that the probability of the food delivery rider running a red light is 90%. Therefore, it will control the host vehicle to remain stationary and wait for the food delivery rider to pass before starting.

[0088] Scenario 2: When it is recognized that the traffic signal corresponding to the lane where the host vehicle is located turns green, if the extension line of the front of the food delivery rider intersects with the driving trajectory line of the host vehicle in the middle part of the road, and at least one other vehicle in the adjacent lane in the same direction as the host vehicle has started, at this time, the risk prediction model may infer that the probability of the food delivery rider running a red light drops to 20%. Therefore, the vehicle can be controlled to start, and at the same time, the system can lock the food delivery rider and continuously predict their next move (such as high-frequency collection of environmental data for this food delivery rider), that is, focus on the movement intention and direction of this food delivery rider to brake in time to avoid a collision.

[0089] Scenario 3: When it is recognized that there is a traffic police officer standing inside the intersection (such as beside the food delivery rider) (at this time, the food delivery rider usually will not run a red light), regardless of whether the front of the food delivery rider intersects with the driving trajectory of the host vehicle, the risk prediction model will infer that it is almost impossible for the food delivery rider to run a red light (that is, the probability of running a red light is extremely low). Therefore, the vehicle can be controlled to drive normally according to the traffic signal or the traffic police officer's gesture.

[0090] Scenario 4: When it is recognized that there is a traffic monitoring camera above the food delivery rider, if the extension line of the front of the food delivery rider intersects with the driving trajectory line of the host vehicle in the middle part of the road, then the risk prediction model infers that the probability of the food delivery rider running a red light is about 50%. At this time, the vehicle can be controlled to start and slowly enter the intersection, and lock the food delivery rider and continuously predict their next move during the driving process, which will not be elaborated here.

[0091] In another embodiment, if the high-risk personnel include children, the environmental data may include at least one of the first personal behaviors of the child (i.e., the behaviors of the child himself / herself, such as walking, running, etc.), the interaction behaviors between the child and the accompanying adult (such as walking side by side, holding hands, the pedestrian carrying / holding the child, waving to call the child, etc.), and the second personal behaviors of the accompanying adult (such as looking down at the mobile phone, walking fast, etc.). The above data can be collected by an in-vehicle information collection device.

[0092] Taking the scenario of child safety behavior analysis as an example, when the vehicle is about to pass through a crosswalk or drive on a road near the sidewalk, after identifying a child, the risk prediction model can further determine whether there is an adult nearby and further reason and predict based on the behaviors of the child and / or the adult to determine whether the child will suddenly rush into the current lane where the self-vehicle is located. Among them, in this solution, the adult and the child can be comprehensively judged through many dimensions such as height, body posture, and gesture. The specific judgment method in the embodiments of the present application is not limited. The following is an illustration in combination with specific scenarios:

[0093] Scenario 1: If it is identified that an adult is holding a child's hand, looking down at the child and (expressions and actions, etc.) showing no urgency (relaxed and not in a hurry), the risk prediction model can infer that the probability of the child running into the current lane is extremely low (such as zero). At this time, the self-vehicle can be controlled to drive normally.

[0094] Scenario 2: If it is identified that an adult is holding a child's hand, but the adult's eyes are looking across the intersection and showing a relatively urgent (flustered and anxious) expression, the risk prediction model can infer that the adult (has something urgent) may take the child to run a red light and rush into the current lane. At this time, the probability of the adult and the child crossing the road may be 50%. Therefore, the self-vehicle can be controlled to decelerate and stop waiting; if the adult does not have the action of running a red light, then control the self-vehicle to start and drive normally through the crosswalk.

[0095] Scenario 3: If it is identified that there is no adult beside the child, the child can be locked as the key target of concern, and its action intention can be analyzed through its behaviors (such as body movements). If it is predicted that the child has the intention to run, the probability of running a red light can reach 80%. At this time, the self-vehicle can be controlled to stop waiting; if the child does not have the action of crossing, then control the self-vehicle to pass slowly; if the child already has the action of crossing, then continue to wait until the child passes in front of the self-vehicle and then start and drive through the crosswalk.

[0096] Scenario 4: If it is recognized that an adult is present but not holding the child's hand, and the child shows signs of running towards the opposite side of the intersection and the adult fails to block in front of the child in time, the risk prediction model can estimate that the probability of the child running a red light is 80%. At this time, the vehicle can be controlled to stop and wait; after both the child and the adult have passed in front of the vehicle, it can start driving again. If it is recognized that the adult successfully blocks the child from running a red light (such as immediately reaching out to grab and block in front of the child after discovery), the probability of the child running a red light can be corrected to zero, and at this time, the vehicle can be controlled to start driving normally.

[0097] In one embodiment, when the environmental data input into the risk prediction model can include sound signals (such as the honking of other vehicles around the vehicle, the shouts of people in the vehicle, etc.) and / or light signals (such as the turn signal of other vehicles, the hazard warning signal, the high beam flash signal, the fog light signal, etc.) emitted by other vehicles around the vehicle, the high-risk driving content output by the model may include reminder information or blocking information from the surrounding vehicles. Specifically, the vehicle can use an external microphone to detect the surrounding sound wave signals in real time. When the honking of a nearby vehicle is detected, the duration and frequency of the honking can be analyzed; it can also use an external visual camera to detect the light signals emitted by other vehicles in real time, and predict / infer the reminders (such as reminding that the road ahead is narrowing, reminding that there is a vehicle breakdown or an obstacle ahead, prompting the vehicle to accelerate, etc.) and blocking information (such as blocking the vehicle from changing lanes or turning, etc.) sent by the surrounding vehicles to the vehicle based on the above signals. At this time, the comprehensive driving control instruction output by the risk prediction model can be at least one of the following instructions, and the vehicle control module can control the vehicle to drive accordingly according to this instruction: such as controlling the vehicle to stop changing lanes and keep driving in the current lane according to the stop changing lane instruction, controlling the vehicle to accelerate according to the acceleration instruction, controlling the vehicle to decelerate according to the deceleration instruction, controlling the vehicle to drive closer to the side of the current lane away from the other vehicle according to the pull-over driving instruction, controlling the vehicle to pull over (it can pull over to the left or right according to local regulations) and stop according to the pull-over and stop instruction, etc.

[0098] Still taking Figure 4Taking the scenario shown as an example, assume that when the host vehicle V1 is changing lanes to the right lane R2 along the trajectory L3 (i.e., the target trajectory at this time) (such as from turning on the right turn signal to the right turn signal turning off after the lane change is completed), if the driver of the vehicle V4 in the right front (a truck with a wide field of vision for the driver) observes that there is a lane change to the left on the front road or there is a malfunctioning vehicle parked, they may turn on the hazard lights (to alert the following vehicles that there is an abnormality ahead), and / or, if the driver of the vehicle V5 in the right rear wants to prevent the host vehicle V1 from changing lanes to the right because it is relatively close to the vehicle V1, they may honk the horn or quickly flash the high beam lights. At this time, the intelligent driving system running in the host vehicle V1 can predict that there may be risks in continuing to change lanes based on the above light signals and / or honking sounds, so it can control the vehicle to stop changing lanes and keep driving within the lane R2 (i.e., continue to follow the vehicle V2) or change lanes to the lane R1 along the trajectory L1 to stay away from the danger. Assume that the driving speed of the host vehicle V2 is lower than the speed limit specified for the lane R2, and there are no other vehicles and possible risks ahead. At this time, if it detects that the following vehicle V1 honks the horn or quickly flashes the high beam lights, the intelligent driving system running in the host vehicle V2 can predict the intention of the following vehicle V1 to urge the host vehicle V2 to accelerate based on this, and at this time, it can control the host vehicle V2 to accelerate according to the acceleration instruction. Assume that the current vehicle speed of the host vehicle V2 is higher than that of the vehicle V4 in the right front, and it is recognized that the vehicle V4 is loaded with heavy goods / hazardous chemicals. When the host vehicle V2 overtakes the vehicle V4 within the lane R2, the intelligent driving system can control the host vehicle V2 to drive closer to the left side of the lane R2 to stay as far away from the vehicle V4 as possible, avoiding causing adverse interference to the driving of the vehicle V4 and resulting in abnormalities; and / or, it can also accelerate to quickly overtake the vehicle V4 to minimize the parallel driving time with this vehicle and reduce the possibility of danger occurring. Assume that when the host vehicle V2 is driving in the vehicle R2, the vehicle V4 in the right front turns on the hazard lights and slowly stops in the lane R3, or the vehicle V4 is parked in the lane R3 and its door is opened at a small angle. At this time, the intelligent driving system can predict that the vehicle V4 may suddenly open the door and cause a collision (i.e., encounter a "door-opening collision") when the host vehicle V2 passes by the vehicle V4. Therefore, the intelligent driving system can control the host vehicle V2 to drive closer to the left side of the lane R2 to stay as far away from the vehicle V4 as possible, or even control the host vehicle V2 to change lanes to the left lane R1 to avoid the possible collision risk in advance.

[0099] In one embodiment, when the environmental data includes the behavioral state data of the vehicle occupants, the intelligent driving system can control the vehicle to travel according to one of the following comprehensive driving control instructions. For example, if the high-risk driving content includes the vehicle's speeding information, the vehicle can be controlled to decelerate according to the deceleration instruction, so as to control the deceleration according to the occupant's behavior when the vehicle is speeding to avoid danger. If the high-risk driving content includes the vehicle's low-speed information, the vehicle can be controlled to accelerate according to the acceleration instruction, so as to control the acceleration according to the occupant's behavior when the vehicle speed is too low to improve the driving efficiency. If the high-risk driving content includes the vehicle's trajectory deviation information, the vehicle can be controlled to change lanes according to the lane change instruction or to turn according to the turning instruction, so as to control the vehicle to slightly steer according to the occupant's behavior when the vehicle is about to or has crossed the line to avoid violations or accidents; or to control the vehicle to return to the navigation path or continue to travel along a new path when the vehicle deviates from the navigation path, etc.

[0100] Exemplarily, the intelligent driving system can predict the driver's intention based on the driver's expression, voice, movement, and / or body state information (such as heart rate, body temperature, blood glucose / oxygen content, etc.), and perform corresponding control on the vehicle. For example, the in-vehicle vision camera can track the driver's eye movement in real time and record the number of gazes at a certain side of the vehicle (such as the right or left side) per unit time and the duration of each gaze; if it is statistically found that the number of gazes and the duration at a certain side meet the corresponding thresholds (such as the number of gazes is not less than 5 times and the duration of each gaze is not less than 200 ms, etc.), it is speculated that the lane on that side may be the area of interest to the driver. In this regard, if it is determined that there is sufficient space for lane change on that side, the vehicle can be controlled to immediately change lanes (because the driver usually observes the rearview mirror on that side multiple times before changing lanes to the other side to confirm safety); or, if it is determined that there are high-risk driving factors on that side, the vehicle can be controlled to decelerate or even stop, etc. In addition, it is also possible to predict whether a traffic accident has occurred on that side in combination with navigation information, etc., and control the vehicle to stay as far away from the accident location as possible when an accident occurs ahead (such as when an accident occurs in front of the current lane, switch from the current lane to other lanes in advance to avoid congestion), etc., which will not be elaborated here.

[0101] Among them, the risk level corresponding to the high-risk driving content output by the risk prediction model can also be obtained, and the risk level is input into the intelligent driving model, so that the control intensity included in the comprehensive driving control instruction output by the model is positively correlated with the risk level (that is, output a comprehensive driving control instruction with a control intensity positively correlated with the risk level). For example, by analyzing the expressions and voices of the driver and the co-driver, the dangerous scenarios currently faced by the vehicle and their urgency can be comprehensively judged, and corresponding comprehensive driving control instructions can be output to control the vehicle to respond. Adopting corresponding-level instructions to control the vehicle's driving according to the urgency of high-risk driving factors helps to ensure the smooth and smooth driving of the vehicle when the urgency is relatively low, improving the riding experience of the passengers; while achieving fast and precise vehicle control when the urgency is relatively high to avoid risks as much as possible and improve driving safety. For specific details, please refer to the following scenarios:

[0102] Scenario 1: If the co-driver makes a sound of "There are many cars ahead, be careful", and at the same time the driver's facial features are "slightly frowning, lips slightly closed", the system will predict that the driver is only slightly worried about the road conditions ahead, but there is no need for emergency braking. At this time, the braking force in the deceleration instruction can be set to 10% of the maximum force (that is, output a deceleration instruction including a braking force of 10% of the maximum force), and the vehicle can be controlled to perform slight braking accordingly to slow down slowly.

[0103] Scenario 2: If the co-driver makes a sound of "Quick brake, about to crash", and at the same time the driver's facial features are "eyes wide open, mouth slightly open, expression tense", the system will predict that the driver is very alert at this time, and the vehicle is still within the controllable range. At this time, the braking force in the deceleration instruction can be set to 70% of the maximum force (that is, output a deceleration instruction including a braking force of 70% of the maximum force), and the vehicle can be controlled to perform stronger braking accordingly to slow down as soon as possible.

[0104] Scenario 3: If the driver makes a shrill scream of "Ah, about to crash, quick brake", and at the same time the driver's facial expression is extremely panicked, the system will predict that the driver is extremely tense at this time and the dangerous situation is very urgent (that is, the vehicle is about to collide). At this time, the braking force in the deceleration instruction can be set to the maximum force (that is, output a deceleration instruction including a braking force of 100% of the maximum force), and the vehicle can be controlled to perform full braking accordingly to strive for the maximum degree of braking to a stop.

[0105] In one embodiment, multimedia resources may be playing during vehicle driving, such as music being played by the in-vehicle audio, a slideshow being played on the co-pilot screen, a video being played on the rear-row screen, etc. At this time, when the risk prediction model identifies that there are high-risk driving factors in the current environment, if the degree of danger is relatively minor (such as the degree of danger is lower than the threshold), the resource can continue to be played; if the degree of danger is relatively serious (such as the degree of danger is not lower than the threshold), at this time, if the multimedia video continues to be played, the occupants may not be in the mood to watch / listen, and may even interfere with the driver's driving of the vehicle. Therefore, the playback volume of the multimedia resource unit can be controlled to decrease, or even the multimedia resource can be directly paused. Exemplarily, if the vehicle drives from a wide main road into a narrow alley with many pedestrians / obstacles, when it is predicted that there are many and serious risk factors around the vehicle, the volume of the music can be controlled to decrease or the music can be directly paused / turned off, without the driver's manual operation, so that the driver can concentrate on driving and avoid being distracted and causing accidents.

[0106] In one embodiment, the intelligent driving system can also use the risk prediction model to determine the sensitive factors in the current environment based on the environmental data, and control the vehicle to perform preset actions for the sensitive factors to isolate or eliminate the sensitive factors. For example, if it is identified that a sprinkler truck is approaching the own vehicle, the vehicle windows and / or sunroof can be controlled to automatically close to prevent the water mist after the sprinkler truck passes from entering the vehicle (isolating the water mist). For another example, if the dust / haze / smoke in the outside air is relatively serious or it is predicted that the vehicle is about to enter a section with relatively serious dust / haze / smoke, the vehicle windows, sunroof and / or external circulation can be controlled to automatically close to prevent poor-quality air from entering the cockpit and affecting the occupants' riding experience (isolating dust / haze / smoke, etc.). For another example, if it is identified that the oncoming vehicle has its high beams on and the driver of the own vehicle squints or can emit a voice reflecting their mood during night driving, the high beams can be automatically controlled to flash to prompt the oncoming vehicle to turn off the high beams (i.e., eliminating the oncoming high beams) and improve the safety of passing each other.

[0107] In one embodiment, the intelligent driving system can also identify high-risk events during the driving of the vehicle, such as lane-changing events with too close a distance to adjacent vehicles, emergency braking (i.e., hard braking) events, braking events that cause the intelligent driving state to exit, collision events (such as rear-ending, being rear-ended, scraping, hitting, etc.), skidding events (possibly caused by factors such as rain / snow / ice on the road), etc., which will not be elaborated here. The occurrence of the above high-risk events may be due to poor control of the vehicle by the driver or the intelligent driving system. In this regard, a risk playback record in multimedia form can be generated for the high-risk event, so that the driver can improve their driving ability by viewing the record (where high-risk driving factors can be marked for easy understanding), or the record can be returned to the server as sample data for subsequent iterative training of the intelligent driving system, thereby improving the intelligent driving level of the system through real driving data.

[0108] In one embodiment, the intelligent driving system can also identify high-value areas passed by the vehicle during driving, such as preset locations like scenic spots and amusement parks, automatically identified high-quality scenery, or scenery that meets the personalized needs of the occupants (such as the user currently logged in to the in-vehicle computer). The videos taken by the vehicle when passing near the above areas may be high-value data for the user. Therefore, a travel playback record in multimedia form can be generated for the high-value area for the user to view. Exemplarily, high-value areas can be automatically identified during driving and automatically generated (or one-key generated by the user during driving or after parking) vlogs of beautiful scenery along the way to enhance the user's travel experience. Additionally, when the vehicle is associated with a preset account (such as a social account, cloud storage account, etc.), the user can also operate on the in-vehicle computer to achieve one-key upload (publish or store) of vlogs, etc., without exporting video data, further simplifying the user's operation.

[0109] Among them, the above risk playback record and travel playback record can be generated by the data storage module in the intelligent driving system (see Figure 2 ). Additionally, the above risk playback record and travel playback record can be in any data form, such as photos, videos, audios, and / or texts, etc., and the embodiments of the present application do not limit this.

[0110] It should be noted that many vehicle control methods (such as accelerating, decelerating, lane-changing, etc.) in the foregoing embodiments should comply with relevant laws, regulations, and standards of relevant countries and regions to ensure that the solutions described in the present application are implemented while conforming to relevant laws, regulations, and standards.

[0111] Corresponding to the embodiment of the driving control method of the vehicle, this specification also provides an embodiment of a driving control device of the vehicle.

[0112] Please refer to Figure 6 ,Figure 6 It is a hardware structure diagram of an electronic device where a driving control device of a vehicle is shown in an exemplary embodiment. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610. Of course, there may also be other hardware required for other services. One or more embodiments of this specification can be implemented in software. For example, the processor 602 reads the corresponding computer program from the non-volatile memory 610 into the memory 608 and then runs it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0113] Please refer to Figure 7 , Figure 7 It is a block diagram of a driving control device of a vehicle shown in an exemplary embodiment. The driving control device of the vehicle can be applied to Figure 6 the electronic device shown in

[0114] a data acquisition unit 701, configured to acquire environmental data of the current environment where the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content;

[0115] an instruction acquisition unit 702, configured to input the environmental data and the high-risk driving content into an intelligent driving model respectively, and acquire a comprehensive driving control instruction output after the model inference;

[0116] a vehicle control unit 703, configured to control the vehicle to drive according to the comprehensive driving control instruction.

[0117] Optionally, the environmental data includes basic environmental data and extended environmental data.

[0118] The data acquisition unit 701 is specifically configured to use the risk prediction model to predict high-risk driving factors in the current environment based on the extended environmental data and output corresponding high-risk driving content;

[0119] The instruction acquisition unit 702 is specifically configured to input the basic environmental data and the high-risk driving content into the intelligent driving model respectively.

[0120] Optionally, the data acquisition unit 701 is specifically configured to perform at least one of the following:

[0121] When the environmental data includes external vehicle image data, input the external vehicle image data into the risk prediction model, and obtain the high-risk driving content of the high-risk driving factors outside the vehicle output after the model inference;

[0122] When the environmental data includes external vehicle sound and light signal data, input the external vehicle sound and light signal data into the risk prediction model, and obtain the high-risk driving content of the high-risk driving factors outside the vehicle output after the model inference;

[0123] When the environmental data includes in-vehicle occupant behavior data, input the behavior state data into the risk prediction model, and obtain the high-risk driving content of the high-risk driving factors inside the vehicle output after the model processing.

[0124] Optionally, the intelligent driving model includes a perception processing module, a path planning module, and an instruction generation module cascaded in sequence. The instruction acquisition unit 702 is specifically configured to:

[0125] When the high-risk driving content includes predicted semantic information, input the predicted semantic information into the path planning module, so that the path planning module determines a target path that matches the high-risk driving factors from each candidate path predicted based on the environmental data, and enables the instruction generation module to output a comprehensive driving control instruction for the target path; and / or,

[0126] When the high-risk driving content includes a predicted control signal, input the predicted control signal into the instruction generation module, so that the instruction generation module adjusts the driving control instruction for the target path according to the predicted control signal, and outputs an adjusted comprehensive driving control instruction, where the target path and its driving control instruction are predicted based on the environmental data.

[0127] Optionally,

[0128] When the environmental data includes the current driving states of other vehicles around the vehicle, the high-risk driving content includes the expected driving trajectory information of the other vehicles. The vehicle control unit 703 is specifically configured to: control the vehicle to change lanes into the adjacent lane according to a lane change instruction;

[0129] When the environmental data includes traffic congestion information of vehicles in the first adjacent lane of the current lane, the high-risk driving content includes the expected lane change information of other vehicles in the first adjacent lane into the current lane. The vehicle control unit 703 is specifically configured to: control the vehicle to decelerate according to a deceleration instruction, or control the vehicle to change lanes into the second adjacent lane of the current lane according to a lane change instruction.

[0130] Optionally, when the high-risk driving factors include high-risk personnel on at least one side in front of the vehicle driving direction and / or associated personnel in the high-risk area on the at least one side, the high-risk driving content includes the expected trajectory and movement probability of the high-risk personnel, and the vehicle control unit 703 is specifically configured to perform one of the following:

[0131] Control the vehicle to stop and wait according to a waiting instruction, control the vehicle to travel at a low speed according to a slow-down instruction, and control the vehicle to decelerate according to a deceleration instruction.

[0132] Optionally,

[0133] If the high-risk personnel are located at the intersection in front of the vehicle, the environmental data includes at least one of the signal light state, camera state, and / or traffic controller state at the intersection;

[0134] If the high-risk personnel include children, the environmental data includes at least one of the first personal behavior of the children, the interaction behavior between the children and the accompanying adults, and the second personal behavior of the accompanying adults.

[0135] Optionally, when the environmental data includes sound signals and / or light signals emitted by other vehicles around the vehicle, and the high-risk driving content includes reminder information or blocking information of the surrounding vehicles, the vehicle control unit 703 is specifically configured to perform one of the following:

[0136] Control the vehicle to stop lane-changing and maintain driving in the current lane according to a stop lane-changing instruction, control the vehicle to accelerate according to an acceleration instruction, control the vehicle to decelerate according to a deceleration instruction, control the vehicle to drive on the side of the current lane away from the other vehicle according to a pull-over driving instruction, and control the vehicle to pull over and stop according to a pull-over stop instruction.

[0137] Optionally, when the environmental data includes the behavioral state data of the vehicle occupants, the vehicle control unit 703 is specifically configured to perform one of the following:

[0138] If the high-risk driving content includes the speeding information of the vehicle, control the vehicle to decelerate according to a deceleration instruction;

[0139] If the high-risk driving content includes the low-speed information of the vehicle, control the vehicle to accelerate according to an acceleration instruction;

[0140] If the high-risk driving content includes the trajectory deviation information of the vehicle, control the vehicle to change lanes according to a lane-changing instruction or control the vehicle to turn according to a turning instruction.

[0141] Optionally, it further includes a degree control unit 704 for:

[0142] Obtain the risk level corresponding to the high-risk driving content output by the risk prediction model, and input the risk level into the intelligent driving model, so that the control intensity included in the comprehensive driving control instruction is positively correlated with the risk level.

[0143] Optionally, it further includes a high-frame-rate sampling unit 705, which is used for:

[0144] When the risk prediction model identifies the high-risk driving factor, control the in-vehicle information acquisition device to collect environmental data related to the high-risk driving factor at a higher frame rate.

[0145] Optionally, it further includes a multimedia playback control unit 706, which is used for:

[0146] When the vehicle is playing multimedia resources, if it is determined that the danger level of the high-risk driving factor is not lower than the threshold, reduce the playback volume of the multimedia resources or pause playing the multimedia resources.

[0147] Optionally, it further includes a risk prompt unit 707, which is used for:

[0148] Output risk prompt information for the high-risk driving factor to the vehicle occupants.

[0149] Optionally, it further includes a playback generation unit 708, which is used for:

[0150] In the case of a high-risk event during the vehicle's driving process, generate a risk playback record in multimedia form for the high-risk event; and / or,

[0151] In the case of passing through a high-value area during the vehicle's driving process, generate a travel playback record in multimedia form for the high-value area.

[0152] Optionally, it further includes a sensitive factor processing unit 709, which is used for:

[0153] Use the risk prediction model to determine the sensitive factors in the current environment based on the environmental data, and control the vehicle to perform preset actions for the sensitive factors to isolate or eliminate the sensitive factors.

[0154] Optionally, the risk prediction model is a vision-language large model VLM; and / or, the intelligent driving model is a driving model based on rules and / or neural networks.

[0155] The specific implementation process of the functions and roles of each unit in the device can be specifically seen in the implementation process of the corresponding steps in the method, which will not be elaborated here.

[0156] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions of the method embodiments. The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions in this specification. Those of ordinary skill in the art can understand and implement them without creative efforts.

[0157] The systems, devices, modules or units illustrated in the embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email receiving and sending device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0158] In a typical configuration, a computer includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0159] The memory may include non-permanent memory in the computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0160] Computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of the storage media of a computer include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, disk storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media, such as modulated data signals and carrier waves.

[0161] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0162] The specific embodiments of this specification have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the" and "said" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0164] It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "upon" or "in response to determining".

[0165] The above description is only the preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included within the scope of protection of one or more embodiments of this specification.

[0166] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

Claims

1. A driving control method for a vehicle, characterized in that, Including: Obtain environmental data of the current environment where the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content, where the risk prediction model is a vision-language large model VLM; Among them, the use of the risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content includes at least one of the following: When the environmental data includes external vehicle image data, input the external vehicle image data into the risk prediction model, and obtain the high-risk driving content of the high-risk driving factors outside the vehicle output after the model's inference; When the environmental data includes external vehicle sound and light signal data, input the external vehicle sound and light signal data into the risk prediction model, and obtain the high-risk driving content of the high-risk driving factors outside the vehicle output after the model's inference; When the environmental data includes the behavior state data of the vehicle occupants, input the behavior state data into the risk prediction model, and obtain the high-risk driving content of the high-risk driving factors inside the vehicle output after the model's processing; Input the environmental data and the high-risk driving content into an intelligent driving model respectively, and obtain a comprehensive driving control instruction output after the model's inference, where the intelligent driving model is a driving model based on rules and / or neural networks; Control the vehicle to drive according to the comprehensive driving control instruction.

2. The method according to claim 1, wherein The environmental data includes basic environmental data and extended environmental data, and use the risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; Inputting the environmental data and the high-risk driving content into the intelligent driving model respectively includes: Using the risk prediction model to predict high-risk driving factors in the current environment based on the extended environmental data and output corresponding high-risk driving content; Input the basic environmental data and the high-risk driving content into the intelligent driving model respectively.

3. The method according to claim 1, characterized in that, The intelligent driving model includes a perception processing module, a path planning module, and an instruction generation module cascaded in sequence. The input of the high-risk driving content into the intelligent driving model includes: When the high-risk driving content includes predicted semantic information, input the predicted semantic information into the path planning module, so that the path planning module determines a target path that matches the high-risk driving factor from each candidate path predicted based on the environmental data, and enables the instruction generation module to output a comprehensive driving control instruction for the target path; and / or, When the high-risk driving content includes a predicted control signal, input the predicted control signal into the instruction generation module, so that the instruction generation module adjusts the driving control instruction for the target path according to the predicted control signal and outputs the adjusted comprehensive driving control instruction, where the target path and its driving control instruction are predicted based on the environmental data.

4. The method according to claim 1, wherein When the environmental data includes the current driving states of other vehicles around the vehicle, the high-risk driving content includes the expected driving trajectory information of the other vehicles, and controlling the vehicle to drive according to the comprehensive driving control instruction includes: controlling the vehicle to change lanes into the adjacent lane according to a lane-changing instruction; When the environmental data includes traffic congestion information of vehicles in the first adjacent lane of the current lane, the high-risk driving content includes the expected lane-changing information of other vehicles in the first adjacent lane changing into the current lane, and controlling the vehicle to drive according to the comprehensive driving control instruction includes: controlling the vehicle to decelerate according to a deceleration instruction, or controlling the vehicle to change lanes into the second adjacent lane of the current lane according to a lane-changing instruction.

5. The method according to claim 1, characterized in that, When the high-risk driving factors include high-risk personnel on at least one side in front of the driving direction of the vehicle, and / or associated personnel in high-risk areas on the at least one side, the high-risk driving content includes the expected trajectory and movement probability of the high-risk personnel, and controlling the vehicle to drive according to the comprehensive driving control instruction includes one of the following: Controlling the vehicle to stop and wait according to a waiting instruction, controlling the vehicle to drive at a low speed according to a slow-travel instruction, or controlling the vehicle to decelerate according to a deceleration instruction.

6. The method according to claim 5, wherein if the high-risk personnel are located at the intersection in front of the vehicle, the environmental data includes at least one of the signal light state, camera state, and / or traffic controller state at the intersection; if the high-risk personnel include children, the environmental data includes at least one of the first personal behavior of the children, the interaction behavior between the children and the accompanying adults, and the second personal behavior of the accompanying adults.

7. The method according to claim 1, wherein When the environmental data includes sound signals and / or light signals emitted by other vehicles around the vehicle, and the high-risk driving content includes reminder information or blocking information of the other vehicles, controlling the vehicle to drive according to the comprehensive driving control instruction includes one of the following: Controlling the vehicle to stop changing lanes and keep driving in the current lane according to a stop-lane-changing instruction, controlling the vehicle to accelerate according to an acceleration instruction, controlling the vehicle to decelerate according to a deceleration instruction, controlling the vehicle to drive closer to the side of the current lane away from the other vehicle according to a pull-over instruction, or controlling the vehicle to pull over and stop according to a pull-over-and-stop instruction.

8. The method according to claim 1, characterized in that, When the environmental data includes the behavior state data of the vehicle occupants, controlling the vehicle to drive according to the comprehensive driving control instruction includes one of the following: if the high-risk driving content includes the speeding information of the vehicle, controlling the vehicle to decelerate according to a deceleration instruction; if the high-risk driving content includes the low-speed information of the vehicle, controlling the vehicle to accelerate according to an acceleration instruction; if the high-risk driving content includes the trajectory deviation information of the vehicle, controlling the vehicle to change lanes according to a lane-changing instruction or controlling the vehicle to turn according to a turning instruction.

9. The method according to claim 8, wherein It further includes: Obtain the risk level corresponding to the high-risk driving content output by the risk prediction model, and input the risk level into the intelligent driving model so that the control intensity included in the comprehensive driving control instruction is positively correlated with the risk level.

10. The method according to claim 1, wherein It further includes: When the risk prediction model identifies the high-risk driving factor, control the in-vehicle information acquisition device to collect environmental data related to the high-risk driving factor at a higher frame rate.

11. The method according to claim 1, characterized in that, It further includes: When the vehicle is playing multimedia resources, if it is determined that the danger level of the high-risk driving factor is not lower than the threshold, reduce the playback volume of the multimedia resources or pause playing the multimedia resources.

12. The method according to claim 1, wherein It further includes: Output risk warning information for the high-risk driving factor to the vehicle occupants.

13. The method according to claim 1, characterized in that, It further includes: When there is a high-risk event during the vehicle driving process, generate a risk playback record in multimedia form for the high-risk event; and / or, When the vehicle passes through a high-value area during the driving process, generate a travel playback record in multimedia form for the high-value area.

14. The method according to claim 1, characterized in that, It further includes: Use the risk prediction model to determine sensitive factors in the current environment based on the environmental data, and control the vehicle to perform preset actions for the sensitive factors to isolate or eliminate the sensitive factors.

15. A driving control device for a vehicle, characterized in that, It includes: A data acquisition unit, configured to acquire environmental data of the current environment where the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content, and the risk prediction model is a vision-language large model VLM; Among them, the use of the risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content specifically includes at least one of the following: When the environmental data includes out-of-vehicle image data, input the out-of-vehicle image data into the risk prediction model, and obtain the high-risk driving content of the out-of-vehicle high-risk driving factor output after the model inference; When the environmental data includes out-of-vehicle sound and light signal data, input the out-of-vehicle sound and light signal data into the risk prediction model, and obtain the high-risk driving content of the out-of-vehicle high-risk driving factor output after the model inference; When the environmental data includes the behavior state data of the vehicle occupants, input the behavior state data into the risk prediction model, and obtain the high-risk driving content of the in-vehicle high-risk driving factor output after the model processing; An instruction acquisition unit, configured to input the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtain the comprehensive driving control instruction output after the model inference, and the intelligent driving model is a driving model based on rules and / or neural networks; A vehicle control unit, configured to control the vehicle to drive according to the comprehensive driving control instruction.

16. An intelligent driving system, characterized in that, The system includes a risk prediction model, an intelligent driving model and a vehicle control module, and the risk prediction model is a vision-language large model VLM, where, The risk prediction model is used to predict high-risk driving factors in the current environment based on environmental data of the vehicle and output corresponding high-risk driving content; Among them, the risk prediction model predicts high-risk driving factors in the current environment based on the environmental data and outputs corresponding high-risk driving content, specifically including at least one of the following: When the environmental data includes out-of-vehicle image data, reasoning is performed based on the input out-of-vehicle image data and high-risk driving content of out-of-vehicle high-risk driving factors is output; When the environmental data includes out-of-vehicle sound and light signal data, reasoning is performed based on the input out-of-vehicle sound and light signal data and high-risk driving content of out-of-vehicle high-risk driving factors is output; When the environmental data includes the behavior state data of in-vehicle occupants, reasoning is performed based on the input behavior state data and high-risk driving content of out-of-vehicle high-risk driving factors is output; The intelligent driving model is used to perform reasoning based on the environmental data and the high-risk driving content and output an inferred comprehensive driving control instruction, and the intelligent driving model is a driving model based on rules and / or neural networks; The vehicle control module is used to control the vehicle to travel according to the comprehensive driving control instruction.

17. An electronic device, comprising: A processor; A memory for storing processor-executable instructions; Among them, the processor realizes the method according to any one of claims 1-14 by running the executable instructions.

18. A computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1-14 are realized.

19. A computer program product, including a computer program and / or instructions, and when the computer program and / or instructions are executed by a processor, the steps of the method according to any one of claims 1-14 are realized.

Images