Vehicle driving control method, device and system

Abstract

The invention provides a driving control method, device and system of a vehicle. The method comprises the steps of obtaining environment data of a current environment where a vehicle is located, predicting high-risk driving factors in the current environment based on the environment data by using a risk prediction model, and outputting corresponding high-risk driving contents; the environment data and the high-risk driving content are input into an intelligent driving model, and a comprehensive driving control instruction output after reasoning of the model is obtained; and controlling the vehicle to run according to the comprehensive running control instruction.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle driving control method, device and system. Background Art

[0002] Vehicles can implement intelligent driving functions based on intelligent driving models (hereinafter referred to as intelligent driving models). At present, intelligent driving models are usually implemented based on rules and / or neural networks.

[0003] The number of driving scenarios in the real world is numerous and ever-changing, and the long-tail problem of driving scenarios is particularly prominent. That is, there are driving scenarios that are rare but numerous and relatively complex. There are many long-tail cases (such as edge cases, corner cases, extreme cases, etc.).

[0004] The rule-based intelligent driving model needs to rely on pre-summarized and set logic and conditions to run, so this type of model is usually only applicable to simple driving scenarios where the corresponding rules can be manually extracted (such as starting, following, changing lanes, braking, etc.), and the driving effect in complex driving scenarios is poor or even difficult to achieve intelligent driving functions, and it is often difficult to deal with the above-mentioned long-tail cases. The intelligent driving model based on neural networks requires a large number of real driving samples for training to improve performance, but due to the existence of the aforementioned long-tail problem, technicians can usually only collect driving cases in conventional driving scenarios as training samples, and cannot exhaust all driving cases in the real world, especially cannot obtain enough cases in long-tail scenarios. Therefore, the trained intelligent driving model often cannot accurately respond to the intelligent driving needs in the above-mentioned long-tail scenarios, and urgently needs to be improved. Summary of the invention

[0005] In view of this, the present application provides a vehicle driving control method, device and system, which solve the problems existing in the traditional intelligent driving model in the related technology by introducing a risk prediction model.

[0006] Specifically, the present application is implemented through the following technical solutions: According to a first aspect of the present application, a vehicle driving control method is provided, comprising: Acquire environmental data of the current environment in which the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; Inputting the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtaining the comprehensive driving control instructions output by the model after reasoning; The vehicle is controlled to travel according to the integrated travel control instruction.

[0007] According to a second aspect of the present application, a vehicle driving control device is provided, comprising: A data acquisition unit, used to acquire environmental data of the current environment in which the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; An instruction acquisition unit, used to input the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtain the comprehensive driving control instruction output by the model after reasoning; A vehicle control unit is used to control the driving of the vehicle according to the comprehensive driving control instruction.

[0008] According to a third aspect of the present application, an intelligent driving system is provided, the system comprising a risk prediction model, an intelligent driving model and a vehicle control module, wherein: The risk prediction model is used to predict high-risk driving factors in the current environment based on environmental data of the current environment in which the vehicle is located and output corresponding high-risk driving content; The intelligent driving model is used to perform reasoning based on the environmental data and the high-risk driving content, and output a comprehensive driving control instruction obtained by reasoning; The vehicle control module is used to control the vehicle driving according to the comprehensive driving control instructions.

[0009] According to a fourth aspect of the present application, an electronic device is provided, including: a processor; a memory for storing instructions executable by the processor; The processor implements the steps of the method described in the first aspect by running the executable instructions.

[0010] According to a fifth aspect of the present application, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0011] According to a sixth aspect of the present application, a computer program and / or instructions are provided, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0012] The technical solution provided by this application may at least include the following beneficial effects: Through the above embodiments, after obtaining the environmental data of the current environment in which the vehicle is located, the risk prediction model can be used to predict the high-risk driving factors in the current environment based on the environmental data and output the corresponding high-risk driving content; then the environmental data and the high-risk driving content are respectively input into the intelligent driving model, and the comprehensive driving control instructions output after the model reasoning are obtained; finally, the vehicle driving is controlled according to the comprehensive driving control instructions.

[0013] Based on the traditional intelligent driving model, this solution introduces a new risk prediction model to participate in the construction of the risk prediction model, and uses this model to predict high-risk driving factors in the current environment and output the corresponding high-risk driving content. It can be understood that based on the risk prediction / reasoning capabilities of the risk prediction model, the high-risk driving content extracted by the model can be regarded as high-dimensional features related to driving in the current environment, and the intelligent driving model can extract low-dimensional features related to driving in the current environment based on environmental data. Therefore, the comprehensive driving control instructions output by the intelligent driving model after reasoning based on the above-mentioned high / low dimensional features actually comprehensively consider the multi-dimensional and multi-type driving-related factors in the current environment, so it can effectively deal with high-risk driving factors to avoid safety accidents, and significantly improve the intelligent driving level of the vehicle.

[0014] It can be seen that the intelligent driving system of the present application makes up for the defects of the traditional intelligent driving model in incomplete or shallow understanding of driving scenarios (especially long-tail or complex scenarios) by introducing a risk prediction model. With the powerful prediction / reasoning capabilities provided by the risk prediction model, whether in complex driving scenarios with more elements or in rare driving scenarios (such as the aforementioned cone case, etc.), the new risk prediction model of the present application can predict the possible risks in real time and effectively based on the environmental data obtained by the vehicle and generate corresponding comprehensive driving control instructions to control the vehicle to respond effectively, thereby ensuring that the vehicle can quickly and accurately respond to direct risks and potential risks in the above-mentioned driving scenarios, effectively improving the safety of intelligent driving of the vehicle and the accuracy of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the hardware architecture of a risk prediction model shown in an embodiment of the present application.

[0016] Figure 2 It is a software architecture diagram of a risk prediction model shown in an embodiment of the present application.

[0017] Figure 3 It is a flowchart of a vehicle driving control method shown in an exemplary embodiment.

[0018] Figure 4is a schematic diagram of a multi-lane driving scenario according to an exemplary embodiment.

[0019] Figure 5 It is a schematic diagram showing an environment in which a vehicle is located according to an exemplary embodiment.

[0020] Figure 6 It is a schematic structural diagram of an electronic device shown in an exemplary embodiment.

[0021] Figure 7 It is a block diagram of a vehicle driving control device shown in an exemplary embodiment. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0024] In response to the technical problems existing in the related technologies, the present application proposes a new intelligent driving system based on a risk prediction model and a vehicle driving control method implemented by the system.

[0025] From the software perspective, the intelligent driving system includes a risk prediction model, an intelligent driving model and a vehicle control module, wherein: The risk prediction model is used to predict high-risk driving factors in the current environment based on environmental data of the current environment in which the vehicle is located and output corresponding high-risk driving content; The intelligent driving model is used to perform reasoning based on the environmental data and the high-risk driving content, and output a comprehensive driving control instruction obtained by reasoning; The vehicle control module is used to control the vehicle driving according to the comprehensive driving control instructions.

[0026] Among them, the intelligent driving model can be a driving model based only on rules (this type of model has clear rules and is suitable for processing deterministic simple driving scenarios when used alone), or it can be a driving model based only on neural networks (this type of model has strong adaptability and can handle some complex driving scenarios when used alone), or it can be a hybrid driving model based on both rules and neural networks (this type of model has both the interpretability of rules and the adaptability of neural networks). The construction, training and deployment methods of any of the above types of intelligent driving models can be referred to the records in the relevant technology, and the embodiments of the present application do not limit this.

[0027] The risk prediction model can be built based on any form of neural network framework and trained in a supervised or unsupervised manner. Taking supervised training as an example, sample data corresponding to the driving case can be obtained in advance, and any sample data contains sample environmental data collected during the vehicle driving process and the corresponding sample driving risk content, and the sample driving risk content is used as the label of the sample environmental data. Through the above training, the risk prediction model can accurately and efficiently use environmental data to predict high-risk driving factors such as pedestrians, surrounding vehicles, and signs in the vehicle's environment, and output high-risk driving content corresponding to these factors. In one embodiment, the risk prediction model can adopt a visual language model (Vision-Language Model, i.e. VLM), which is a multimodal artificial intelligence model that combines computer vision (Computer Vision, i.e. CV) and natural language processing (Natural Language Processing, i.e. NLP) capabilities, and can simultaneously understand and process image (or video) and text information, and establish associations between the two, thereby achieving more complex tasks. The risk prediction model is implemented using VLM, so that the model can predict the high-risk driving factors in the current environment relatively more accurately from multiple dimensions based on multimodal environmental data, and output the high-risk driving content corresponding to the factor.

[0028] This application combines the traditional intelligent driving model with risk prediction models such as VLM to give full play to their respective advantages: based on the decision-making ability and safety guarantee of the traditional intelligent driving model itself, the intelligent driving strategy is guaranteed to ensure the performance and decision-making ability of the intelligent driving system; at the same time, by integrating the advantages of VLM in reasoning ability and multimodal fusion, it can achieve a comprehensive understanding of the driving environment and intelligent decision-making support, thereby providing higher-dimensional and further risk prediction for the traditional intelligent driving model, effectively improving the upper limit of the decision-making ability and intelligence level of the intelligent driving system, and greatly improving the overall performance of the system.

[0029] In addition, the intelligent driving implemented by the intelligent driving system described in this application can be assisted driving or automatic driving, and this application does not limit the level of the intelligent driving (such as L3, L4, etc.). However, it should be noted that the intelligent driving implemented by the system should comply with the relevant laws, regulations and standards of the corresponding countries and regions (such as the place of sale and / or use of the vehicle), and provide relevant personnel (such as drivers, etc.) with corresponding operation entrances for them to choose to authorize or refuse to use.

[0030] The specific working methods of the above-mentioned models and modules can be found in the description of the embodiments below and will not be elaborated here.

[0031] Figure 1 is a schematic diagram of the hardware architecture of an intelligent driving system shown in an embodiment of the present application, such as Figure 1 As shown, from a hardware perspective, the system may include only vehicle 11, or may include both vehicle 11 and server 13. If the intelligent driving system includes only vehicle 11, then the risk prediction model, intelligent driving model, and vehicle control module are all deployed in vehicle 11, such as in the domain controller 111 of vehicle 11. For example, the risk prediction model and intelligent driving model may be deployed in the domain controller of the intelligent driving domain of vehicle 11, and the vehicle control module may be deployed in the domain controller of the body domain and / or power domain of vehicle 11. If the intelligent driving system includes both vehicle 11 and server 13, then at least one of the risk prediction model and intelligent driving model may be deployed in server 13, and the vehicle control module may be deployed in vehicle 11 (such as in the aforementioned domain controller), which will not be described in detail.

[0032] In addition to the aforementioned domain control, the vehicle 11 may also be equipped with corresponding information collection devices. According to the collected data / installation location, the above-mentioned information collection devices can be divided into in-vehicle collection devices and out-vehicle collection devices. Exemplarily, the in-vehicle collection device may include at least one of an in-vehicle camera, an in-vehicle microphone, an in-vehicle biosensor (which can be used to identify the body temperature, fingerprint, iris and other biometric features of the in-vehicle occupant 12), an in-vehicle odor sensor, etc. The in-vehicle collection device may include at least part of the device in the DMS (Driver Monitoring System) that detects the driver's attention and emotions. The out-vehicle collection device may include at least one of a visual camera, a millimeter-wave radar, a laser radar, an infrared sensor, an out-vehicle microphone, etc. Among them, the above-mentioned information collection device can be installed in different positions of the vehicle according to its function. Taking the out-vehicle sensor as an example, Figure 1As shown, the visual camera 112 is installed at the front of the vehicle, the visual camera 113 is installed at the rear of the vehicle, and the laser radar 114 is installed at the front of the roof, which will not be repeated. Of course, the embodiment of the present application does not limit the specific parameters such as the type, quantity, size, and installation position of the above-mentioned information collection device.

[0033] In addition, the off-vehicle data collection device may also include a wireless communication module and a navigation and positioning module. The wireless communication module may include at least one of a radio frequency module (such as a 3G, 4G and / or 5G module), a BLE (Bluetooth) module, a Wi-Fi module, an NFC (Near Field Communication) module, and a NearLink module; the navigation and positioning module may include using GPS (Global Positioning System), Beidou, GLONASS (Global Navigatsion Satellite System), Galileo satellite navigation system, quasi-zenith satellite system and other systems to obtain navigation and positioning related information of the vehicle.

[0034] In addition to directly collecting information inside and outside the vehicle through the information collection module installed by the vehicle 11 itself, the vehicle 11 can also communicate with the electronic device used by the occupant 12 in the vehicle to obtain the environmental data directly collected by the electronic device. For example, the vehicle 11 can establish a wireless connection with the electronic device through the aforementioned wireless communication module, and receive the information inside and outside the vehicle (such as images of obstacles outside the vehicle, videos for showing congestion outside the vehicle, etc.) and the occupant's physical state information (such as heartbeat, body temperature, blood sugar / blood oxygen content, etc.) reported after the collection. Exemplarily, the electronic device may include but is not limited to mobile phones, PCs (Personal Computers), workstations (Work Stations), PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc., and one or more embodiments of the present application are not limited to this.

[0035] In addition, the vehicle can establish a network connection with the remote server 13 through the aforementioned wireless communication module to interact with the server 13 for data. For example, if the risk prediction model is deployed in the server 13, the vehicle 11 can send the environmental data collected by itself to the server 13, and receive the high-risk driving content output by the model after predicting the high-risk driving factors using the environmental data. For another example, if the risk prediction model and the intelligent driving model are deployed in the server 13, the vehicle 11 can send the environmental data collected by itself to the server 13, and receive the comprehensive driving control instructions output by the intelligent driving model after reasoning based on the environmental data and the high-risk driving content (output by the risk prediction model based on the environmental data). Among them, the server 13 can be a physical server including an independent host, or it can also be a virtual server, cloud server, etc. hosted by a host cluster. In addition, the embodiment of the present application does not limit the number, type, specific interaction method with the vehicle, etc. of the server 13. As for the network 10 for interaction between the vehicle 11 and the server 13, the corresponding type of wireless network can be specifically selected to achieve communication based on the communication method supported by the corresponding device, and this specification does not limit this.

[0036] In addition, the vehicle described in the present application (such as vehicle 11) can be a pickup truck, a sedan, an SUV (Sport Utility Vehicle), a motorhome, a truck, etc. in terms of its functional form; it can be a fuel vehicle or a new energy vehicle (such as a hybrid vehicle, a pure electric vehicle, a hydrogen energy vehicle, a methanol energy vehicle, etc.) in terms of its power form. The present invention does not limit the specific form of the vehicle. In addition, the passenger 12 in the cabin can be a driver sitting in the driving seat, or can also be at least one passenger sitting in other positions. The present application also does not limit the number of members and their seating positions.

[0037] Figure 2 is a schematic diagram of the software architecture of a risk prediction model shown in an embodiment of the present application, such as Figure 2 As shown, the system includes a risk prediction model, an intelligent driving model and a vehicle control module. The environmental data acquired by the vehicle-mounted information acquisition device can be simply divided into basic environmental data and extended environmental data. Among them, the intelligent driving model can be inferred based on the input extended environmental data such as external image data, external sound and light signal data, and internal occupant behavior data to predict high-risk driving factors, and then output the corresponding high-risk driving content to the intelligent driving module; and the risk prediction model can be inferred based on the input basic environmental data and the high-risk driving content, and output the corresponding comprehensive driving control instructions to the vehicle control module, so that the latter can control the vehicle driving according to the instructions.

[0038] In addition, the intelligent driving system may also include a data storage module for collecting relevant data during the operation of the intelligent driving system (such as the aforementioned environmental data, integrated driving control instructions, etc.) and generating multimedia record files based on this. The specific process can be found in the embodiments below, which will not be repeated here.

[0039] Figure 3 is a flow chart of a vehicle driving control method shown in an embodiment of the present application, and the method is applied to the aforementioned intelligent driving system. Figure 3 As shown, the method includes the following steps 302-306.

[0040] Step 302, obtaining environmental data of the current environment in which the vehicle is located, and using a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content.

[0041] Step 304: input the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtain the comprehensive driving control instructions output by the model after reasoning.

[0042] First of all, it should be noted that the current environment of the vehicle described in this application is not limited to the external environment (such as the road or parking lot where the vehicle is located), but may also include the internal environment (such as the position, behavior, and status of the passengers in the vehicle). After the vehicle is powered on and / or the engine is started, the information collection devices such as sensors installed on the vehicle can start to collect corresponding environmental data in real time for the current environment. After the intelligent driving system starts running (such as the driver triggers the intelligent driving function), the intelligent driving system can obtain the above environmental data in real time, and use the data for subsequent processing to control the vehicle's driving. Of course, at least part of the above information collection devices can also only collect environmental data in real time after the intelligent driving system starts running, and do not collect environmental data when the system is not started, thereby avoiding invalid collection of environmental data to save the vehicle's power, storage and other resources.

[0043] In the embodiment of the present application, the high-risk driving factors in the current environment may include factors that may cause safety hazards to the normal driving of the vehicle or even cause traffic accidents. For example, when the vehicle (hereinafter referred to as the self-vehicle) is driving normally in the current lane, the front vehicle in the same lane may brake suddenly, causing the self-vehicle to rear-end the front vehicle, the rear vehicle may accelerate to rear-end the self-vehicle, and other vehicles in the adjacent lane may change lanes and cause scratches with the self-vehicle, etc. These vehicles are all high-risk driving factors for the self-vehicle. For another example, when the vehicle is driving in the current lane without a guardrail, two-wheeled vehicles (bicycles, battery vehicles, etc.) driving on the non-motorized vehicle lane next to it, children chasing and playing on the sidewalk, etc. may suddenly invade the current lane and collide with the self-vehicle, so they are also high-risk driving factors for the self-vehicle. For such risk factors, the intelligent driving system of the present application can avoid them by controlling the vehicle driving according to the comprehensive driving control instructions, so as to avoid safety accidents as much as possible.

[0044] Alternatively, the high-risk driving factors in the current environment may also include factors that may cause the vehicle's driving efficiency to decrease, driving energy consumption to increase, and / or driving action to be violent. For example, when the vehicle is driving normally in the current lane, the vehicle in front of the same lane may be driving slowly (i.e., the driving speed is significantly lower than the speed limit of the current lane), and there are no other vehicles in the adjacent fast lane (i.e., the speed limit is not lower than the adjacent lane of the current lane) or other vehicles within the lane change distance (i.e., the distance that can ensure that the vehicle can normally complete the lane change operation). At this time, if the vehicle continues to follow the vehicle in front and drive slowly, it will obviously limit the vehicle's driving speed and increase the driving time / fuel consumption. At this time, the vehicle in front can be regarded as a high-risk driving factor for the vehicle. For example, when the vehicle is waiting in line in the current lane to go straight through the intersection ahead, if the red light has turned green but the vehicle in front of the current lane is seriously congested and driving slowly, and there is no vehicle (or fewer vehicles) in the right lane and it is allowed to go straight, then if the vehicle continues to crawl or even stop and wait, it will obviously limit the vehicle's driving speed and increase the driving time / fuel consumption. At this time, the vehicle in front can also be regarded as a high-risk driving factor for the vehicle. For such risk factors, the intelligent driving system of the present application can avoid them by controlling the vehicle driving according to the comprehensive driving control instructions, thereby improving the driving efficiency as much as possible and achieving smooth / low-energy driving.

[0045] The environmental data is at least one type of data collected for the current environment and capable of representing the environment, such as the external images / videos collected by the external visual camera, the point cloud data detected by the laser radar, the horn of the rear vehicle collected by the external microphone, the expressions of the driver and other passengers detected by the internal camera, the voices of the driver and other passengers detected by the internal microphone, etc. Based on the above current environment, the risk prediction model can predict whether there are high-risk driving factors in the current environment, and extract the high-risk driving content of these factors, such as the expected speed and braking probability of the vehicle in front, the expected trajectory and lane change probability of the vehicles in the adjacent lanes in the future, and the expected trajectory and intrusion probability of the bicycles driving on the non-motorized lane ahead, etc., which will not be repeated.

[0046] In one embodiment, given that both the risk prediction model and the intelligent driving model in the intelligent driving system can receive environmental data and process them separately, and the environmental data received by the two may be the same or different, in order to facilitate distinction, the environmental data input into the risk prediction model can be referred to as basic environmental data, and the environmental data input into the intelligent driving model can be referred to as extended environmental data. Therefore, the risk prediction model can predict the high-risk driving factors in the current environment based on the extended environmental data and output the corresponding high-risk driving content, and the intelligent driving system can input the basic environmental data and the high-risk driving content into the intelligent driving model, so that it can infer based on the two and output the final comprehensive driving control instructions. In this way, different environmental data can be accurately input into the corresponding model, which helps the risk prediction model to achieve more accurate risk prediction and helps the intelligent driving model to generate more accurate and comprehensive control instructions.

[0047] It should be noted that the environmental data collected by any information collection device may only be input into the risk prediction model (such as Figure 2 The external waveform data collected by the millimeter-wave radar and the external point cloud data collected by the lidar as shown in the figure may also be input into the intelligent driving model only (such as Figure 2 The in-car images captured by the in-car visual camera, the outside sound signals captured by the outside microphone, the in-car sound signals captured by the in-car microphone, etc.) may also be input into the risk prediction model and the intelligent driving model (such as Figure 2 The external image of the vehicle collected by the external visual camera, the map data obtained by the vehicle, the current positioning data, etc.). What kind of data is input into the above two models respectively (that is, which model any collected environmental data is input into) can be determined according to actual needs during the model training and use stage, and the embodiment of the present application does not limit this.

[0048] As mentioned above, the risk prediction model and / or intelligent driving model can be deployed and run on a cloud server (such as Figure 1The server 11 shown can also be deployed and run locally in the vehicle. It is understandable that if deployed on a server, the server's rich computing / storage / network resources can be fully utilized to run a full model with rich functions and large parameter quantity / scale, thereby realizing a high-precision and feature-rich intelligent driving solution. If deployed locally in the vehicle, a relatively lightweight on-board model obtained after optimization and compression processing (such as model pruning, model quantization, knowledge distillation, etc.) of the full model can be deployed, thereby ensuring the reasoning accuracy and basic functions while effectively shortening the data transmission time, thereby greatly speeding up the reasoning speed of the intelligent driving system and reducing the safety hazards that may be caused by control delays or even signal loss. Of course, if the resources carried by the vehicle are sufficient, the full model can also be deployed and run locally in the vehicle, and the embodiments of the present application do not limit this.

[0049] It is understandable that if the risk prediction model and the intelligent driving model are both deployed locally in the vehicle, various environmental data can be directly input into the corresponding model. If the risk prediction model and the intelligent driving model are both deployed in the server, the environmental data can be sent to the server in a unified manner, and the comprehensive driving control instructions returned by the latter can be received. If the risk prediction model is deployed in the server and the intelligent driving model is deployed locally in the vehicle, the extended environmental data can be sent to the server, and the basic environmental data and the high-risk driving content returned by the server can be input into the intelligent driving model; conversely, if the risk prediction model is deployed locally in the vehicle and the intelligent driving model is deployed in the server, the extended environmental data can be input into the intelligent driving model, and the basic environmental data and the high-risk driving content output by the model can be sent to the server together, and the comprehensive driving control instructions returned by the server can be received, which will not be repeated. Of course, the specific deployment locations of the above-mentioned models can be reasonably set according to actual conditions such as latency requirements and resource requirements, and the embodiments of the present application do not limit this. The following embodiments are described as an example in which both models are deployed locally in the vehicle.

[0050] In one embodiment, the intelligent driving model can predict high-risk driving factors and output corresponding high-risk driving content according to the type of environmental data. For example, in the case where the environmental data includes external image data, the data can be input into the risk prediction model and the high-risk driving content of the high-risk driving factors outside the vehicle output after the model is inferred can be obtained. For example, the model can identify high-risk vehicles outside the vehicle that may change lanes and output the expected trajectory of the vehicle and its lane change probability based on the external image or video collected by the external visual camera, identify pedestrians who may run red lights and output their red light running probability and driving trajectory at the intersection, etc., identify children who may be introduced into the lane and output their intrusion probability and walking trajectory after intrusion, etc. For another example, in the case where the environmental data includes external sound and light signal data, the data can be input into the risk prediction model, and the high-risk driving content of the high-risk driving factors outside the vehicle output after the model is inferred can be obtained. For example, the model can predict possible obstacles or car accidents on the road ahead and output the probability of their impact on the vehicle, predict possible special road conditions such as fog / ice / pits / etc. on the road ahead and output the degree of their impact on the vehicle's trajectory, predict possible yielding requirements of vehicles behind (such as special vehicles) and output the yielding direction and urgency, etc., based on the light signals of surrounding vehicles collected by the external visual camera and / or the horn signals of surrounding vehicles collected by the external microphone, etc. For example, in the case where the environmental data includes the behavior data of the passengers in the vehicle, the data can be input into the risk prediction model, and the high-risk driving content of the high-risk driving factors in the vehicle output after processing by the model can be obtained. For example, the model can predict the possible collision between the vehicle and the object in front and output the collision position and probability, etc., based on the facial expressions and body movements of the driver collected by the internal visual camera and / or the voices of the driver or passengers collected by the internal microphone, etc., predict the possible physical abnormalities of the driver of the vehicle and output the possible control abnormalities of the vehicle and their probabilities, etc. Through the above method, the risk prediction model can fully and accurately predict the high-risk driving factors that may exist in the current environment and output the corresponding high-risk driving content, which will help the subsequent risk prediction model to output more accurate driving strategies.

[0051] It should be noted that the above-mentioned various inside and outside vehicle information may be personal sensitive information of users / passengers or sensitive information of relevant organizations / institutions. The above-mentioned information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0052] In one embodiment, the intelligent driving model may include a perception processing module, a path planning module, and an instruction generation module (such as Figure 2 As shown). The perception processing module is used to extract basic features of the input environmental data (such as the aforementioned basic environmental data), the path planning module is used to plan the vehicle's driving path at least according to the basic features, and the instruction generation module is used to generate a driving control instruction for the driving path so as to control the vehicle to travel along the path.

[0053] For the above-mentioned multiple modules, when high-risk driving content is input into the intelligent driving model, the risk prediction model can input different forms of content into different modules in a targeted manner to achieve corresponding intelligent driving performance. For example, the high-risk driving content output by the risk prediction model may include predicted semantic information, and the information can be input into the aforementioned path planning module at this time, so that the module determines the target path that matches the high-risk driving factor from the various candidate paths predicted based on the environmental data, and enables the instruction generation module to output a comprehensive driving control instruction for the target path. Among them, the predicted semantic information can be in any form that can be recognized by the path planning module, such as text, feature vector, token, etc., and the embodiment of the present application does not limit this. After being processed by the path planning module, the above-mentioned predicted voice information will affect the weight value (that is, the possibility of being selected) of each candidate path, thereby affecting the target path. In this solution, the path planning module can generate multiple candidate paths for the vehicle at the current moment (i.e., the paths that the vehicle can choose to travel after the current moment) based on the aforementioned basic environmental data according to its own rules / logic, and the risk prediction model directly inputs the predicted semantic information (related to the risk factors at the current moment) extracted by itself into the path planning module, so that the module can select the target path that can avoid the above risk factors as much as possible from multiple candidate paths in combination with the predicted semantic information, thereby improving the safety and driving quality during intelligent driving. In this way, risk prediction results can be introduced in the process of path planning by the intelligent driving model, so that the path planning results can effectively avoid risks.

[0054] For another example, the high-risk driving content output by the risk prediction model may include a prediction control signal. At this time, the signal may be input into the aforementioned instruction generation module, so that the instruction generation module adjusts the driving control instruction for the target path according to the prediction control signal, and outputs the adjusted comprehensive driving control instruction, wherein the target path and its driving control instruction are predicted based on the environmental data. In this scheme, the perception processing module first extracts the basic features of the current environment based on the environmental data (such as the aforementioned basic environmental data), so the target path planned by the path planning module and the driving control instruction generated by the instruction generation module for the path can be regarded as being obtained based on the environmental data. On this basis, after receiving the prediction control signal output by the risk prediction model, the instruction generation module can directly generate a comprehensive driving control instruction for the target path according to the signal, so as to fully reflect the influence of the high-risk driving factor corresponding to the prediction control signal on the driving control instruction. In this way, the risk prediction result can be directly introduced when the intelligent driving model generates a control instruction for the target path, thereby significantly shortening the influence process of the risk prediction result on the final output instruction of the model and enhancing the influence, which is helpful to realize the emergency control of the vehicle in the scenario with a high risk severity, and improve the accuracy and timeliness of the response to serious risks.

[0055] In one embodiment, the intelligent driving system can predict the possible high-risk driving content according to the driving conditions of other vehicles around the vehicle, and control the vehicle to avoid it in advance to improve the safety of intelligent driving, or control the vehicle to actively change the current driving state to actively respond to the above factors to improve the quality of intelligent driving. For example, in the case where the environmental data includes the current driving state of other vehicles around the vehicle, the high-risk driving content may include the expected driving trajectory information of the other vehicles. Among them, the current driving state of any other vehicle may include the current speed, current acceleration, current driving direction, distance from the vehicle, light signals (such as turn signal signals, double flash signal signals, etc.) of the vehicle, etc. Based on this, the risk prediction model can predict the expected driving trajectory information of other vehicles, such as whether to change lanes into the lane and its expected lane change trajectory, whether to brake and its acceleration and braking time when braking, etc. Further, the comprehensive driving control instructions output by the risk prediction model may be targeted following instructions, lane change instructions, deceleration instructions, etc. Taking the lane change instruction as an example, the vehicle can be controlled to change lanes into the adjacent lane according to the lane change instruction to avoid the above risks.

[0056] It is understandable that when the traffic volume in the lanes on both sides is higher than the speed of the vehicle in front; or when the traffic volume in the adjacent lanes is small and the vehicle in front maintains the current speed for a long time, the traditional intelligent driving model will often control the vehicle to continue to follow the vehicle and drive slowly, so the driving experience is not flexible enough, which may lead to low driving efficiency. However, through the above method, the intelligent driving system can comprehensively analyze many factors such as the speed, acceleration, distance from the vehicle and traffic volume of other vehicles on both sides, and predict the expected driving conditions of surrounding vehicles in the future, so as to generate lane change trajectories while ensuring safety; further, it can evaluate the timing and suitability of lane change for different lane change trajectories, and then make comprehensive decisions and decisively issue corresponding lane change instructions to control the vehicle to complete the lane change safely and efficiently.

[0057] like Figure 4 As shown, it is assumed that the vehicle V1 is following the vehicle V2 in front of it in lane R2 (i.e., following the vehicle), there is a vehicle V2 in the left lane R1 in front of the vehicle on the left, a vehicle V3 in the right lane R2 in the right rear of the vehicle on the right, and a vehicle V4 in the right lane R2 in front of the vehicle on the right. It is assumed that the path planning module in the risk prediction model running locally on V1 plans three candidate paths: the path L1 for changing lanes to the left, the path L2 for continuing to follow the vehicle, and the path L3 for changing lanes to the right. It should be noted that the path planning module described in the embodiment of the present application is used to plan the driving path of the vehicle, which can be a complete path from the departure point to the destination determined in combination with map, navigation and other information, or it can be a path corresponding to simple operations such as turning, changing lanes, and avoiding during driving; in other words, the embodiment of the present application does not limit the length and degree of refinement (i.e., granularity) of the driving path planned by the intelligent driving model. If the risk prediction model predicts that lane V4 will change lanes into lane R2, the vehicle V1 can be controlled to slow down (at this time, L2 is selected as the target path) or change lanes to R1 along path L1 (at this time, L1 is selected as the target path) to avoid rear-ending vehicle V4 that changes lanes to R2.

[0058] For another example, when the environmental data includes vehicle congestion information in the first adjacent lane of the current lane (which may be any lane adjacent to the current lane), the high-risk driving content may include expected lane change information of other vehicles in the first adjacent lane changing lanes into the current lane. At this time, the vehicle control module may control the vehicle to decelerate according to the deceleration instruction issued by the intelligent driving model, or control the vehicle to change lanes to the second adjacent lane of the current lane according to the lane change instruction issued by the model. The intelligent driving system predicts the congestion in front of the adjacent lane by continuously observing the driving status of other vehicles around, and predicts the speed, acceleration and lane change behavior of other vehicles by in-depth reasoning of the surrounding scenes, so that the system can predict the possible risks of the vehicle in front in advance, so as to take measures to decelerate or change lanes in advance. For example, when the adjacent lane suddenly becomes congested, even if the vehicle in the front area of ​​the lane does not have obvious lane change action, turn on the turn signal or invade into the lane, the system will predict that there may be a vehicle in front of the vehicle changing lanes (i.e. invading) the lane, which may cause the vehicle in front to brake urgently. If the vehicle still drives at the current speed, there is a risk of rear-end collision. In this case, you can control the vehicle to actively reduce speed, maintain a safe distance from the vehicle in front, or choose to change lanes to the other side. Figure 4 As shown, if it is predicted that congestion is about to occur in front of the vehicle V4 in the right lane R3, the vehicle V1 can be controlled to change lanes to the left lane R1 along the path L2 to avoid possible congestion in the lane R2 or possible sudden braking of the front vehicle V2.

[0059] In one embodiment, given that there may be only some locations in the space around the vehicle where high-risk driving factors exist (i.e., the risk prediction model may only identify high-risk driving factors located in some locations), it is clear that the safety / inefficiency risks of the objects / factors in these locations are lower than those in other locations. Therefore, in order to more accurately predict the risky behaviors that may be subsequently generated by high-risk driving factors in these locations, more processing resources can be allocated to these locations. For example, when the risk prediction model identifies the high-risk driving factors, the intelligent driving system can control the on-board information acquisition device to collect environmental data related to the high-risk driving factors at a higher frame rate (such as increasing the sampling frame rate from 10fps to 30fps), so that the high-risk driving factors can be processed at a faster frame rate, reducing the prediction inaccuracy that may be caused by processing delays. Of course, this solution requires that the information acquisition device can operate at different frame rates and its operating frame rate can be controlled and adjusted by the intelligent driving system.

[0060] In order to ensure that the environmental data collected at a higher frame rate can be processed smoothly and effectively to obtain more accurate processing results, more storage and computing resources can be allocated to the aforementioned risk prediction model and intelligent driving model. The corresponding amount of resources can be dispatched in real time for the above models according to the number, type and data volume of high-risk driving factors, so as to improve the efficient and full use of the vehicle's local resources. In addition, when a high-risk driving factor is predicted, the driver can be reminded by highlighting or voice broadcasting.

[0061] Step 306: Control the vehicle to travel according to the comprehensive travel control instruction.

[0062] As mentioned above, based on the collected environmental data and the aforementioned high-risk driving content output by the risk prediction model, the intelligent driving model can perform inference and output corresponding comprehensive driving control instructions. At this time, the vehicle control module can control the vehicle driving according to the instructions.

[0063] It can be seen from the above embodiments that this solution introduces a new risk prediction model on the basis of the traditional intelligent driving model to participate in the construction of the risk prediction model, and uses the model to predict the high-risk driving factors in the current environment and output the corresponding high-risk driving content. It can be understood that based on the risk prediction / reasoning capabilities of the risk prediction model, the high-risk driving content extracted by the model can be regarded as high-dimensional features related to driving in the current environment, and the intelligent driving model can extract low-dimensional features related to driving in the current environment based on environmental data. Therefore, the comprehensive driving control instructions output by the intelligent driving model after reasoning based on the above-mentioned high / low dimensional features actually comprehensively consider the multi-dimensional and multi-type driving-related factors in the current environment, so it can effectively deal with high-risk driving factors to avoid safety accidents, and significantly improve the intelligent driving level of the vehicle.

[0064] It can be seen that the intelligent driving system of the present application makes up for the defects of the traditional intelligent driving model in incomplete or shallow understanding of driving scenarios (especially long-tail or complex scenarios) by introducing a risk prediction model. With the powerful prediction / reasoning capabilities provided by the risk prediction model, whether in complex driving scenarios with more elements or in rare driving scenarios (such as the aforementioned cone case, etc.), the new risk prediction model of the present application can predict the possible risks in real time and effectively based on the environmental data obtained by the vehicle and generate corresponding comprehensive driving control instructions to control the vehicle to respond effectively, thereby ensuring that the vehicle can quickly and accurately respond to direct risks and potential risks in the above-mentioned driving scenarios, effectively improving the safety of intelligent driving of the vehicle and the accuracy of decision-making.

[0065] In one embodiment, when the high-risk driving factors include high-risk persons located at least one side ahead of the vehicle in the driving direction, and / or associated persons located in the high-risk area on at least one side, the high-risk driving content may include the expected trajectory and movement probability of the high-risk persons. At this time, the vehicle can be controlled to travel according to the corresponding type of comprehensive driving control instructions. For example, the vehicle can be controlled to stop and wait according to the waiting instruction, the vehicle can be controlled to travel at a low speed according to the slow-down instruction, and / or the vehicle can be controlled to decelerate according to the deceleration instruction. Among them, at least one side ahead of the vehicle in the driving direction can be the left and / or right side of the lane where the vehicle is currently located (hereinafter referred to as the current lane). The high-risk person is a person whose probability of traffic violations / violations (such as running a red light, not stopping to give way, etc.) is greater than the probability threshold, such as high-risk professional practitioners (such as food delivery riders, couriers, etc.) and / or special personnel (such as children, the elderly, etc.) appearing at the intersection when the vehicle is waiting to go straight through the intersection. The above-mentioned personnel have a high probability of running a red light, invading road markings, etc., and therefore have a high risk. The associated persons in the high-risk area may be children associated with the campus (i.e., please drive slowly in the school ahead), patients associated with the hospital, elderly people associated with the nursing home, etc. Of course, the above-mentioned high-risk persons and high-risk areas can be reasonably defined and set according to the actual situation in the physical world, and the embodiments of the present application do not limit this. If the above-mentioned high-risk persons are detected or predicted, it is assumed that they may cause a certain degree of safety hazards to the driving of the vehicle, so at this time the intelligent driving model can predict and output the expected trajectory and corresponding movement probability of the high-risk persons, and make a comprehensive decision based on the interaction between the expected trajectory and the driving trajectory of the vehicle (in the future).

[0066] For example, if a rolling ball is detected in front of the vehicle, and its rolling trajectory intersects with the vehicle's trajectory at some point in the future, it can be inferred that a child may intrude into the current lane, and the vehicle can be controlled to slow down to avoid collision. For another example, if the map shows that there is a school or kindergarten on the right side of the road ahead, the map information, the vehicle's driving status, visual images and other multimodal data can be combined to determine that the school gate is facing the current lane where the vehicle is located. At this time, it can be detected in real time and focused on whether there are pedestrians crossing the road at the school gate, and lanes can be changed to a lane relatively farther away from the school gate in advance.

[0067] In one embodiment, when the risk prediction model identifies high-risk driving factors in the current environment, the system can also output risk warning information for the high-risk driving factors to the passengers in the vehicle, so that the passengers (especially the driver) can be informed of the factors in time, improve their perception of the current environment, and facilitate the passengers to deal with risks in advance. Among them, the above-mentioned risk warning information can be output to the user in any suitable form such as displaying a picture, playing audio (such as playing a prompt voice through a microphone in the car) and / or controlling vibration (such as steering wheel vibration). Taking the display picture as an example, a picture containing risk warning information can be displayed through a car screen or a HUD (Head-Up Display) device, such as a complete bird's-eye view (Bird's Eye View, i.e. BEV) picture of the current space of the vehicle can be displayed on the central control screen or dashboard screen of the vehicle, or some key risk warning information can be displayed through the HUD. Among them, the above-mentioned picture can include the current state of the vehicle, surrounding lanes / vehicles / pedestrians / markings and many other contents for users to view.

[0068] Take the bird's-eye view as an example. Figure 5 As shown, the screen displays various contents in a three-dimensional form, such as the self-vehicle 501 is displayed in the middle of the screen, and the vehicles next to it, road elements (lane lines, zebra crossings, traffic lights, etc.), and people nearby (such as people riding electric bicycles on the left rear, children riding scooters / yo-scooters on the outside of the left front road, and babies walking on the zebra crossing in front, etc.) are displayed according to the actual spatial position relationship. At this time, the traffic light is green. According to the driving direction of the self-vehicle and the driving direction and movements of the child riding the yo-scooter, the intelligent driving system can predict that the child is likely to slide into the intersection and run a red light, so it is judged as a high-risk driving factor at this time. In this regard, the child can be highlighted by a red indicator box, and the prompt text such as "There is a child sliding on a yo-scooter, please pay attention to safety" can be emphasized in the text box, and no further elaboration will be given.

[0069] In one embodiment, if the aforementioned high-risk person is located at the intersection in front of the vehicle, the environmental data may include at least one of the signal light status at the intersection (such as the status of the traffic light, the remaining time of the red light, etc.), the camera status (whether there is a camera, the camera position, whether the camera is turned on, etc.) and / or the traffic controller status (such as the location of the traffic police, the traffic police gesture, etc.). The above data can be collected by the vehicle-mounted information collection device, or directly obtained from the map or navigation information, and the embodiment of the present application does not limit this.

[0070] Taking the common takeaway riders at intersections as an example, traditional intelligent driving models can usually only predict the trajectory of the takeaway rider based on the current motion state of the takeaway rider. If the takeaway rider is parked and not moving at the current moment, it is difficult to judge his driving intention, so the vehicle may collide with the rider after entering the intersection. However, the risk prediction model of this solution can further predict the trajectory of the takeaway rider and the probability of him crossing the road or running a red light when the takeaway rider is detected in the waiting area of ​​the intersection. The following is an explanation based on specific scenarios: Scenario 1: When the traffic light corresponding to the lane where the ego vehicle is located is identified to turn green, if the extension line of the delivery rider's front direction intersects with the driving trajectory line of the ego vehicle (i.e., the vehicle) in the middle part of the road, and other vehicles in the adjacent lane in the same direction as the ego vehicle have not started (are still in a parked state), the risk prediction model may infer that the probability of the delivery rider running a red light is 90%, so the ego vehicle will be controlled to remain stationary and wait for the delivery rider to pass before starting.

[0071] Scenario 2: When the traffic light corresponding to the lane where the vehicle is located turns green, if the extension line of the delivery rider's front direction intersects with the vehicle's driving trajectory in the middle part of the road, and other vehicles in at least one adjacent lane in the same direction as the vehicle have started, the risk prediction model may infer that the probability of the delivery rider running a red light has dropped to 20%, so the vehicle can be controlled to start. At the same time, the system can lock the delivery rider and continue to predict his next move (such as high-frequency collection of environmental data for the delivery rider), that is, focus on the delivery rider's movement intention and movement, so as to brake in time to avoid a collision.

[0072] Scenario 3: When a traffic policeman is identified at the intersection (such as next to a food delivery rider) (the food delivery rider usually does not run a red light at this time), regardless of whether the direction of the food delivery rider's vehicle head intersects with the driving trajectory of the vehicle, the risk prediction model will infer that it is almost impossible for the food delivery rider to run a red light (that is, the probability of running a red light is extremely low), so the vehicle can be controlled to drive normally according to the traffic lights or traffic police gestures.

[0073] Scenario 4: When a traffic surveillance camera is detected above the delivery rider, if the direction of the delivery rider's vehicle head intersects with the vehicle's driving trajectory in the middle of the road, the risk prediction model infers that the probability of the delivery rider running a red light is about 50%. At this time, the vehicle can be controlled to start slowly and enter the intersection, and lock the delivery rider during driving and continue to predict his next move. I will not go into details.

[0074] In another embodiment, if the high-risk person includes a child, the environmental data may include at least one of the first individual behavior of the child (i.e., the child's own behavior, such as walking, running, etc.), the interactive behavior between the child and the adult walking with him / her (such as walking side by side, holding hands, the pedestrian carrying / holding the child, waving to call the child, etc.), and the second individual behavior of the adult walking with him / her (such as looking down at the phone, walking fast, etc.). The above data can be collected by the vehicle-mounted information collection device.

[0075] Taking the child safety behavior analysis scenario as an example, when the vehicle is about to pass a crosswalk or travel on a road close to a sidewalk, the risk prediction model can further determine whether there are adults nearby after identifying the child, and further infer and predict based on the behavior of the child and / or adult to determine whether the child will suddenly break into the current lane where the vehicle is located. Among them, adults and children in this solution can be comprehensively judged by many dimensions such as height, body shape, posture, etc., and the specific judgment method is not limited in the embodiments of this application. The following is an explanation in conjunction with specific scenarios: Scenario 1: If an adult is identified as holding a child's hand, looking down at the child, and not appearing to be in a hurry (relaxed and not anxious) (in expression and action, etc.), the risk prediction model can infer that the probability of the child running into the current lane is extremely low (e.g., zero), and the vehicle can be controlled to drive normally.

[0076] Scenario 2: If an adult is identified as holding a child's hand, but the adult is looking across the intersection and appears to be in a hurry (panicked and anxious), the risk prediction model can infer that the adult (in a hurry) may run a red light with the child and rush into the current lane. At this time, the probability of the adult and the child crossing the road may be 50%, so the vehicle can be controlled to slow down and stop and wait; if the adult does not run a red light, the vehicle can be controlled to start normally and drive through the crosswalk.

[0077] Scenario 3: If it is recognized that there is no adult next to the child, the child can be locked as the focus target, and the child's action intention can be analyzed through his / her behavior (such as body movements). If it is predicted that the child has the intention to run, the probability of running a red light can be as high as 80%. At this time, the vehicle can be controlled to stop and wait; if the child does not make a crossing move, the vehicle can be controlled to pass slowly; if the child has already made a crossing move, continue to wait until the child passes in front of the vehicle before starting to drive through the crosswalk.

[0078] Scenario 4: If an adult is identified as being present but not holding the child's hand, and the child shows signs of running to the other side of the intersection and the adult does not stop the child in time, the risk prediction model can infer that the probability of the child running a red light is 80%. At this time, the vehicle can be controlled to stop and wait; the vehicle can start to drive after the child and the adult have passed in front of the vehicle. If the adult is identified as successfully blocking the child from running a red light (such as reaching out to stop the child immediately after discovering the child), the probability of the child running a red light can be corrected to zero, and the vehicle can be controlled to start driving normally.

[0079] In one embodiment, when the environmental data input into the risk prediction model may include sound signals (horns of other vehicles, shouts of people in the vehicle, etc.) and / or light signals (turn signal signals, double flash light signals, high beam flashing signals, fog light signals, etc.) emitted by other vehicles around the vehicle, the high-risk driving content output by the model may include reminder information or blocking information of the surrounding vehicles. Specifically, the vehicle can use an external microphone to detect the surrounding sound wave signals in real time. When the horn of a nearby vehicle is detected, the duration and frequency of the horn can be analyzed; the vehicle can also use an external visual camera to detect the light signals emitted by other vehicles in real time, and predict / infer the reminders (such as reminding that the road ahead is narrowing, reminding that the vehicle ahead is faulty or has obstacles, prompting the vehicle to accelerate, etc.) and blocking information (such as preventing the vehicle from changing lanes or turning, etc.) issued by the surrounding vehicles to the vehicle based on the above signals. At this time, the comprehensive driving control instruction output by the risk prediction model can be at least one of the following instructions. At this time, the vehicle control module can control the vehicle driving accordingly according to the instruction: such as controlling the vehicle to stop changing lanes and maintain the current lane according to the stop lane change instruction, controlling the vehicle to accelerate according to the acceleration instruction, controlling the vehicle to decelerate according to the deceleration instruction, controlling the vehicle to drive close to the current lane and away from the other vehicles according to the pull over instruction, controlling the vehicle to pull over (left or right according to local regulations) according to the pull over instruction, etc.

[0080] Still Figure 4Taking the scenario shown as an example, assuming that the ego vehicle V1 is changing lanes to the right lane R2 along the trajectory L3 (i.e., the target trajectory at this time) (e.g., from turning on the right turn signal to turning off the right turn signal after the lane change is completed), if the driver of the right front vehicle V4 (a truck whose driver has a wide field of vision) observes that the road ahead is merging to the left or there is a faulty vehicle parked, he may turn on the hazard lights (to alert the rear vehicle that there is an abnormality ahead), and / or, if the driver of the right rear vehicle V5 wants to stop the ego vehicle V1 from changing lanes to the right because it is close to the vehicle V1, he may honk or flash the high beam quickly. At this time, the intelligent driving system running in the ego vehicle V1 can predict that continuing to change lanes may be dangerous based on the above light signals and / or horn sounds, so it can control the vehicle to stop changing lanes and keep driving in lane R2 (i.e., continue to follow vehicle V2) or change lanes to lane R1 along the trajectory L1 to stay away from danger. Assuming that the speed of ego vehicle V2 is lower than the upper speed limit specified in lane R2, and there are no other vehicles and possible risks in front of it, if the rear vehicle V1 is detected to honk or flash the high beam quickly, the intelligent driving system running in ego vehicle V2 can predict the intention of the rear vehicle V1 to urge ego vehicle V2 to accelerate, and can control ego vehicle V2 to accelerate according to the acceleration command. Assuming that the current speed of ego vehicle V2 is higher than the right front vehicle V4, and it is recognized that vehicle V4 is loaded with heavy objects / hazardous chemicals, when ego vehicle V2 exceeds vehicle V4 in lane R2, the intelligent driving system can control ego vehicle V2 to drive close to the left side of lane R2 to stay as far away from vehicle V4 as possible, to avoid causing adverse interference to the driving of vehicle V4 and causing abnormalities; and / or, it can also accelerate to quickly overtake vehicle V4 to minimize the parallel time with the vehicle and reduce the possibility of danger. Assume that when ego vehicle V2 is driving behind vehicle R2, vehicle V4 in front of it on the right turns on its hazard lights and slowly stops in lane R3, or vehicle V4 is parked in lane R3 with its door open at a small angle. At this time, the intelligent driving system can predict that when passing vehicle V4, vehicle V4 may suddenly open its door and cause a collision (i.e., encounter a "door opening kill"). Therefore, the intelligent driving system can control ego vehicle V2 to drive close to the left side of lane R2 to stay as far away from vehicle V4 as possible, or even control ego vehicle V2 to change lanes to the left lane R1 to avoid possible collision risks in advance.

[0081] In one embodiment, when the environmental data includes the behavior status data of the occupants in the vehicle, the intelligent driving system can control the vehicle driving according to one of the following integrated driving control instructions. For example, if the high-risk driving content includes the speeding information of the vehicle, the vehicle can be controlled to decelerate according to the deceleration instruction, so that the deceleration is controlled according to the occupant behavior when the vehicle is speeding to avoid danger. If the high-risk driving content includes the low-speed information of the vehicle, the vehicle can be controlled to accelerate according to the acceleration instruction, so that the acceleration is controlled according to the occupant behavior when the vehicle speed is too low to improve driving efficiency. If the high-risk driving content includes the trajectory deviation information of the vehicle, the vehicle can be controlled to change lanes according to the lane change instruction or the vehicle can be controlled to turn according to the turning instruction, so that the vehicle can be controlled to turn slightly according to the occupant behavior when the vehicle is about to or has crossed the line to avoid violations or accidents; or when the vehicle deviates from the navigation path, the vehicle can be controlled to return to the path or continue to drive along a new path, etc.

[0082] For example, the intelligent driving system can predict the driver's intention based on the driver's expression, voice, action and / or physical state information (such as heartbeat, body temperature, blood sugar / blood oxygen content, etc.), and control the vehicle accordingly. For example, the in-vehicle visual camera can track the driver's eye movement in real time, record the number of gazes on a certain side of the vehicle (such as the right or left side) and the duration of each gaze in a unit time; if the number and duration of gazes on a certain side meet the corresponding threshold (such as the number of times not less than 5 times and the duration of each gaze not less than 200ms, etc.), it is inferred that the lane on this side may be the area of ​​interest to the driver. In this regard, if it is determined that there is enough space for lane change on this side, the vehicle can be controlled to change lanes immediately (because the driver usually observes the side rearview mirror several times before changing lanes to a certain side to confirm whether it is safe); or, if it is determined that there are high-risk driving factors on this side, the vehicle can be controlled to slow down or even stop. In addition, it can also be combined with navigation information to predict whether a traffic accident has occurred on this side, and when an accident occurs in front, the vehicle can be controlled to stay as far away from the accident location as possible (such as when an accident occurs in front of the current lane, switch from the current lane to other lanes in advance to avoid congestion), etc., which will not be repeated.

[0083] Among them, the risk level corresponding to the high-risk driving content output by the risk prediction model can also be obtained, and the risk level can be input into the intelligent driving model, so that the control intensity contained in the comprehensive driving control instruction output by the model is positively correlated with the risk level (that is, the comprehensive driving control instruction whose control intensity is positively correlated with the risk level is output). For example, by analyzing the expressions and voices of the driver and the co-driver, the dangerous scenarios currently faced by the vehicle and their urgency can be comprehensively judged, and the corresponding comprehensive driving control instructions can be output to control the vehicle to deal with them. Using instructions of corresponding degrees according to the urgency of high-risk driving factors to control vehicle driving can help ensure smooth and smooth driving of the vehicle when the urgency is low, and improve the riding experience of the passengers; and when the urgency is high, rapid and precise vehicle control can be achieved to avoid risks as much as possible and improve driving safety. For details, please refer to the following scenarios: Scenario 1: If the front passenger says "There are a lot of cars ahead, be careful", and the driver's facial features are "slightly frowning and slightly closed lips", the system will predict that the driver is only slightly worried about the road conditions ahead, but does not need to brake urgently. At this time, the braking force in the deceleration command can be set to 10% of the maximum force (that is, the output includes a deceleration command with a braking force of 10% of the maximum force), and the vehicle can be controlled to perform slight braking accordingly to slow down.

[0084] Scenario 2: If the front passenger makes a sound of "brake quickly, there's going to be a collision", and the driver's facial features are "eyes wide open, mouth slightly open, and nervous expression", the system will predict that the driver is very alert at this time, and the vehicle is still within the controllable range. At this time, the braking force in the deceleration command can be set to 70% of the maximum force (that is, the output includes a deceleration command with a braking force of 70% of the maximum force), and the vehicle can be controlled to brake more strongly accordingly to slow down as quickly as possible.

[0085] Scenario 3: If the driver screams “Ah, it’s about to crash, brake quickly” and his facial expression is extremely terrified, the system will predict that the driver is extremely nervous and the dangerous situation is very urgent (that is, the vehicle is about to collide). At this time, the braking force in the deceleration command can be set to the maximum force (that is, the output includes a deceleration command with a braking force of 100% of the maximum force), and the vehicle can be controlled to brake with full force accordingly to achieve the greatest possible stop.

[0086] In one embodiment, multimedia resources may be playing while the vehicle is driving, such as music being played on the car audio system, slides being played on the co-pilot screen, and videos being played on the back screen. At this time, if the risk prediction model identifies that there are high-risk driving factors in the current environment, if the degree of danger is relatively mild (such as the degree of danger is lower than the threshold), the resource can continue to be played; if the degree of danger is relatively serious (such as the degree of danger is not lower than the threshold), the occupants may not be interested in watching / listening to the multimedia video if it continues to play, and it may even interfere with the driver's driving of the vehicle. Therefore, the playback volume of the multimedia resource unit can be controlled to be lowered, or even the multimedia resource can be directly paused. Exemplarily, if the vehicle drives from a wide main road into a narrow alley with many pedestrians / obstacles, the volume of the music can be controlled to be lowered or the music can be directly paused / turned off when it is predicted that there are many and serious risk factors around the vehicle, without the driver having to manually operate so that he can concentrate on driving and avoid distraction and accidents.

[0087] In one embodiment, the intelligent driving system can also use the risk prediction model to determine the sensitive factors in the current environment based on the environmental data, and control the vehicle to perform preset actions for the sensitive factors to isolate or eliminate the sensitive factors. For example, if it is identified that a sprinkler truck is approaching the vehicle, the windows and / or sunroof can be controlled to be automatically closed to prevent the water mist after the sprinkler truck passes into the vehicle (isolating the water mist). For another example, if it is identified that the dust / haze / smog in the air outside the vehicle is relatively serious or it is predicted that the front extreme environment will enter a road section with relatively serious dust / haze / smog, the vehicle's windows, sunroof and / or external circulation can be controlled to be automatically closed to prevent poor quality air from entering the cabin and affecting the passengers' riding experience (isolating dust / haze / smog, etc.). For another example, if it is identified that the oncoming vehicle turns on the high beam when driving at night and the driver of the vehicle squints or can make a voice that reflects his own emotions, the high beam can be automatically controlled to flash to prompt the oncoming vehicle to turn off the high beam (i.e., eliminate the oncoming high beam) to improve the safety of the meeting.

[0088] In one embodiment, the intelligent driving system can also identify high-risk events during the driving process of the vehicle, such as lane change events with a close distance to adjacent vehicles, emergency braking (i.e., emergency braking) events, braking events that lead to exiting the intelligent driving state, collisions (such as rear-end collisions, being rear-ended, scratches, collisions, etc.), sliding events (may be caused by factors such as rain / snow / ice on the road), etc., which will not be repeated. The above-mentioned high-risk events may occur due to poor control of the vehicle by the driver or the intelligent driving system. In this regard, a risk playback record in multimedia form can be generated for the high-risk events, so that the driver can improve his driving ability by viewing the record (where high-risk driving factors can be marked for easy understanding), or the record can be returned to the server as sample data for subsequent iterative training of the intelligent driving system, thereby improving the intelligent driving level of this system through real driving data.

[0089] In one embodiment, the intelligent driving system can also identify high-value areas passed by the vehicle during driving, such as preset locations such as scenic spots and amusement parks, automatically identify high-quality scenery or scenery that meets the personalized needs of passengers (such as users currently logged in to the car computer), etc. The video taken by the vehicle when passing near the above-mentioned area may be high-value data for the user, so a multimedia travel playback record can be generated for the high-value area for the user to view. Exemplarily, high-value areas can be automatically identified during driving and vlogs of beautiful scenery along the way can be automatically generated (or generated by one-click operation by the user during driving or after parking) to enhance the user's travel experience. In addition, when the vehicle is associated with a preset account (such as a social account, a cloud storage account, etc.), the user can also operate on the car computer to upload (publish or store) vlogs with one click, etc., without exporting video data, further simplifying user operations.

[0090] The risk playback record and travel playback record can be stored in the data storage module in the intelligent driving system (see Figure 2 In addition, the risk playback record and travel playback record may be in any data form, such as photos, videos, audio and / or text, etc., and the present application embodiment does not limit this.

[0091] It should be noted that the various vehicle control methods (such as acceleration, deceleration, lane changing, etc.) in the aforementioned embodiments should comply with the relevant laws, regulations and standards of the relevant countries and regions to ensure that the solutions described in this application are implemented in compliance with the relevant laws, regulations and standards.

[0092] Corresponding to the embodiment of the vehicle driving control method, this specification also provides an embodiment of a vehicle driving control device.

[0093] See also Figure 6 , Figure 6 This is a hardware structure diagram of an electronic device where a vehicle's driving control device is located, shown in an exemplary embodiment. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0094] See also Figure 7 , Figure 7 FIG. 1 is a block diagram of a vehicle driving control device according to an exemplary embodiment. The vehicle driving control device can be applied to Figure 6 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device may include: The data acquisition unit 701 is used to acquire environmental data of the current environment in which the vehicle is located, and use the risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; The instruction acquisition unit 702 is used to input the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtain the comprehensive driving control instruction output by the model after reasoning; The vehicle control unit 703 is used to control the driving of the vehicle according to the comprehensive driving control instruction.

[0095] Optionally, the environment data includes basic environment data and extended environment data. The data acquisition unit 701 is specifically used to predict the high-risk driving factors in the current environment based on the extended environment data using the risk prediction model and output corresponding high-risk driving content; The instruction acquisition unit 702 is specifically used to input the basic environment data and the high-risk driving content into the intelligent driving model respectively.

[0096] Optionally, the data acquisition unit 701 is specifically used for at least one of the following: In the case where the environmental data includes image data outside the vehicle, the image data outside the vehicle is input into the risk prediction model, and high-risk driving content of high-risk driving factors outside the vehicle output after inference by the model is obtained; In the case where the environmental data includes sound and light signal data outside the vehicle, the sound and light signal data outside the vehicle is input into the risk prediction model, and the high-risk driving content of the high-risk driving factors outside the vehicle output after the model is inferred is obtained; In the case where the environmental data includes in-vehicle occupant behavior data, the behavior state data is input into the risk prediction model, and the high-risk driving content of the high-risk driving factors in the vehicle output after being processed by the model is obtained.

[0097] Optionally, the intelligent driving model includes a perception processing module, a path planning module and an instruction generation module which are cascaded in sequence, and the instruction acquisition unit 702 is specifically used for: In the case where the high-risk driving content includes predicted semantic information, the predicted semantic information is input into the path planning module, so that the path planning module determines a target path matching the high-risk driving factor from various candidate paths predicted based on the environmental data, and the instruction generation module outputs a comprehensive driving control instruction for the target path; and / or, In the case where the high-risk driving content includes a predicted control signal, the predicted control signal is input into the instruction generation module so that the instruction generation module adjusts the driving control instructions for the target path according to the predicted control signal, and outputs the adjusted comprehensive driving control instructions, wherein the target path and its driving control instructions are predicted based on the environmental data.

[0098] Optional, In the case where the environmental data includes the current driving status of other vehicles around the vehicle, the high-risk driving content includes the expected driving trajectory information of the other vehicles, and the vehicle control unit 703 is specifically used to: control the vehicle to change lanes into an adjacent lane according to the lane change instruction; In a case where the environmental data includes vehicle congestion information in a first adjacent lane of a current lane, the high-risk driving content includes expected lane change information of other vehicles in the first adjacent lane changing lanes into the current lane, and the vehicle control unit 703 is specifically used to: control the vehicle to decelerate according to a deceleration instruction, or control the vehicle to change lanes into a second adjacent lane of the current lane according to a lane change instruction.

[0099] Optionally, when the high-risk driving factor includes a high-risk person located at least one side ahead of the vehicle's driving direction, and / or an associated person located in the high-risk area on at least one side, the high-risk driving content includes the expected trajectory and movement probability of the high-risk person, and the vehicle control unit 703 is specifically used for one of the following: The vehicle is controlled to stop and wait according to the waiting instruction, the vehicle is controlled to drive at a low speed according to the slow driving instruction, and the vehicle is controlled to decelerate according to the deceleration instruction.

[0100] Optional, If the high-risk person is located at the intersection in front of the vehicle, the environmental data includes at least one of the status of a traffic light, a camera, and / or a traffic controller at the intersection; If the high-risk persons include children, the environmental data includes at least one of a first personal behavior of the child, an interaction behavior between the child and an adult accompanying the child, and a second personal behavior of the adult accompanying the child.

[0101] Optionally, when the environmental data includes sound signals and / or light signals emitted by other vehicles around the vehicle, and the high-risk driving content includes reminder information or blocking information of the surrounding vehicles, the vehicle control unit 703 is specifically used for one of the following: The vehicle is controlled to stop changing lanes and maintain driving in the current lane according to the stop lane change instruction, the vehicle is controlled to accelerate according to the acceleration instruction, the vehicle is controlled to decelerate according to the deceleration instruction, the vehicle is controlled to drive close to the current lane and away from the other vehicles according to the pull over instruction, and the vehicle is controlled to pull over and park according to the pull over instruction.

[0102] Optionally, when the environmental data includes behavior status data of the vehicle occupants, the vehicle control unit 703 is specifically used for one of the following: If the high-risk driving content includes speeding information of the vehicle, controlling the vehicle to decelerate according to the deceleration instruction; If the high-risk driving content includes low-speed information of the vehicle, controlling the vehicle to accelerate according to the acceleration instruction; If the high-risk driving content includes the trajectory deviation information of the vehicle, the vehicle is controlled to change lanes according to a lane change instruction or the vehicle is controlled to turn according to a turn instruction.

[0103] Optionally, a degree control unit 704 is further included, which is used to: The risk level corresponding to the high-risk driving content output by the risk prediction model is obtained, and the risk level is input into the intelligent driving model so that the control intensity contained in the comprehensive driving control instruction is positively correlated with the risk level.

[0104] Optionally, a high frame rate sampling unit 705 is further included, which is used to: When the risk prediction model identifies the high-risk driving factor, the vehicle-mounted information collection device is controlled to collect environmental data related to the high-risk driving factor at a higher frame rate.

[0105] Optionally, a multimedia playback control unit 706 is further included, configured to: In the case that the vehicle is playing a multimedia resource, if it is determined that the danger level of the high-risk driving factor is not lower than a threshold, the playing volume of the multimedia resource is reduced or the playing of the multimedia resource is paused.

[0106] Optionally, a risk warning unit 707 is further included, which is used to: Output risk warning information for the high-risk driving factors to the occupants in the vehicle.

[0107] Optionally, the system further includes a playback generation unit 708, which is used to: In the event that a high-risk event occurs during the driving of the vehicle, generating a multimedia risk playback record for the high-risk event; and / or, When the vehicle passes through a high-value area during driving, a travel playback record in multimedia form is generated for the high-value area.

[0108] Optionally, a sensitive factor processing unit 709 is further included, which is used to: The risk prediction model is used to determine sensitive factors in the current environment based on the environmental data, and the vehicle is controlled to perform preset actions for the sensitive factors to isolate or eliminate the sensitive factors.

[0109] Optionally, the risk prediction model is a visual language large model (VLM); and / or the intelligent driving model is a driving model based on rules and / or neural networks.

[0110] The implementation process of the functions and effects of each unit in the device is specifically described in the implementation process of the corresponding steps in the method, which will not be repeated here.

[0111] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0112] The systems, devices, modules or units described in the embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0113] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] The foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "one", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0119] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0120] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

[0121] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A vehicle driving control method, characterized in that: include: Acquire environmental data of the current environment in which the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; Inputting the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtaining the comprehensive driving control instructions output by the model after reasoning; The vehicle is controlled to travel according to the integrated travel control instruction.

2. The method according to claim 1, characterized in that The environmental data includes basic environmental data and extended environmental data, and the risk prediction model is used to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; Inputting the environmental data and the high-risk driving content into the intelligent driving model respectively includes: Using the risk prediction model to predict high-risk driving factors in the current environment based on the extended environment data and outputting corresponding high-risk driving content; The basic environment data and the high-risk driving content are respectively input into the intelligent driving model.

3. The method according to claim 1, characterized in that The using the risk prediction model to predict the high-risk driving factors in the current environment based on the environmental data and outputting corresponding high-risk driving content includes at least one of the following: In the case where the environmental data includes image data outside the vehicle, the image data outside the vehicle is input into the risk prediction model, and high-risk driving content of high-risk driving factors outside the vehicle output after inference by the model is obtained; In the case where the environmental data includes sound and light signal data outside the vehicle, the sound and light signal data outside the vehicle is input into the risk prediction model, and the high-risk driving content of the high-risk driving factors outside the vehicle output after the model is inferred is obtained; In the case where the environmental data includes in-vehicle occupant behavior data, the behavior state data is input into the risk prediction model, and the high-risk driving content of the high-risk driving factors in the vehicle output after being processed by the model is obtained.

4. The method according to claim 1, characterized in that The intelligent driving model includes a perception processing module, a path planning module and an instruction generation module which are cascaded in sequence, and the inputting of the high-risk driving content into the intelligent driving model includes: In the case where the high-risk driving content includes predicted semantic information, the predicted semantic information is input into the path planning module, so that the path planning module determines a target path matching the high-risk driving factor from various candidate paths predicted based on the environmental data, and the instruction generation module outputs a comprehensive driving control instruction for the target path; and / or, In the case where the high-risk driving content includes a predicted control signal, the predicted control signal is input into the instruction generation module so that the instruction generation module adjusts the driving control instructions for the target path according to the predicted control signal, and outputs the adjusted comprehensive driving control instructions, wherein the target path and its driving control instructions are predicted based on the environmental data.

5. The method according to claim 1, characterized in that In the case where the environmental data includes current driving states of other vehicles around the vehicle, the high-risk driving content includes expected driving trajectory information of the other vehicles, and the controlling the vehicle to drive according to the comprehensive driving control instruction includes: controlling the vehicle to change lanes into an adjacent lane according to a lane change instruction; In a case where the environmental data includes vehicle congestion information in a first adjacent lane of a current lane, the high-risk driving content includes expected lane change information of other vehicles in the first adjacent lane changing lanes into the current lane, and controlling the vehicle driving according to the comprehensive driving control instruction includes: controlling the vehicle to decelerate according to a deceleration instruction, or controlling the vehicle to change lanes into a second adjacent lane of the current lane according to a lane change instruction.

6. The method according to claim 1, characterized in that In the case where the high-risk driving factor includes a high-risk person located at least one side ahead of the driving direction of the vehicle, and / or an associated person located in the high-risk area on the at least one side, the high-risk driving content includes the expected trajectory and movement probability of the high-risk person, and controlling the driving of the vehicle according to the comprehensive driving control instruction includes one of the following: The vehicle is controlled to stop and wait according to the waiting instruction, the vehicle is controlled to drive at a low speed according to the slow driving instruction, and the vehicle is controlled to decelerate according to the deceleration instruction.

7. The method according to claim 6, characterized in that If the high-risk person is located at the intersection in front of the vehicle, the environmental data includes at least one of the status of a traffic light, a camera, and / or a traffic controller at the intersection; If the high-risk persons include children, the environmental data includes at least one of a first personal behavior of the child, an interaction behavior between the child and an adult accompanying the child, and a second personal behavior of the adult accompanying the child.

8. The method according to claim 1, characterized in that In the case where the environmental data includes sound signals and / or light signals emitted by other vehicles around the vehicle, and the high-risk driving content includes reminder information or blocking information of the surrounding vehicles, controlling the driving of the vehicle according to the comprehensive driving control instruction includes one of the following: The vehicle is controlled to stop changing lanes and maintain driving in the current lane according to the stop lane change instruction, the vehicle is controlled to accelerate according to the acceleration instruction, the vehicle is controlled to decelerate according to the deceleration instruction, the vehicle is controlled to drive close to the current lane and away from the other vehicles according to the pull over instruction, and the vehicle is controlled to pull over and park according to the pull over instruction.

9. The method according to claim 1, characterized in that: In the case where the environmental data includes the behavior state data of the vehicle occupants, controlling the vehicle driving according to the integrated driving control instruction includes one of the following: If the high-risk driving content includes speeding information of the vehicle, controlling the vehicle to decelerate according to the deceleration instruction; If the high-risk driving content includes low-speed information of the vehicle, controlling the vehicle to accelerate according to the acceleration instruction; If the high-risk driving content includes the trajectory deviation information of the vehicle, the vehicle is controlled to change lanes according to a lane change instruction or the vehicle is controlled to turn according to a turn instruction.

10. The method according to claim 9, characterized in that Also includes: The risk level corresponding to the high-risk driving content output by the risk prediction model is obtained, and the risk level is input into the intelligent driving model so that the control intensity contained in the comprehensive driving control instruction is positively correlated with the risk level.

11. The method according to claim 1, characterized in that: Also includes: When the risk prediction model identifies the high-risk driving factor, the vehicle-mounted information collection device is controlled to collect environmental data related to the high-risk driving factor at a higher frame rate.

12. The method according to claim 1, characterized in that Also includes: In the case that the vehicle is playing a multimedia resource, if it is determined that the danger level of the high-risk driving factor is not lower than a threshold, the playing volume of the multimedia resource is reduced or the playing of the multimedia resource is paused.

13. The method according to claim 1, characterized in that Also includes: Output risk warning information for the high-risk driving factors to the occupants in the vehicle.

14. The method according to claim 1, characterized in that Also includes: In the event that a high-risk event occurs during the driving of the vehicle, generating a risk playback record in multimedia form for the high-risk event; and / or, When the vehicle passes through a high-value area during driving, a travel playback record in multimedia form is generated for the high-value area.

15. The method according to claim 1, characterized in that Also includes: The risk prediction model is used to determine sensitive factors in the current environment based on the environmental data, and the vehicle is controlled to perform preset actions for the sensitive factors to isolate or eliminate the sensitive factors.

16. The method according to claim 1, characterized in that The risk prediction model is a visual language large model (VLM); and / or, The intelligent driving model is a driving model based on rules and / or neural networks.

17. A vehicle travel control device, characterized in that: include: A data acquisition unit, used to acquire environmental data of the current environment in which the vehicle is located, and use a risk prediction model to predict high-risk driving factors in the current environment based on the environmental data and output corresponding high-risk driving content; An instruction acquisition unit, used to input the environmental data and the high-risk driving content into the intelligent driving model respectively, and obtain the comprehensive driving control instruction output by the model after reasoning; A vehicle control unit is used to control the driving of the vehicle according to the comprehensive driving control instruction.

18. An intelligent driving system, characterized in that: The system includes a risk prediction model, an intelligent driving model and a vehicle control module. The risk prediction model is used to predict high-risk driving factors in the current environment based on environmental data of the current environment in which the vehicle is located and output corresponding high-risk driving content; The intelligent driving model is used to perform reasoning based on the environmental data and the high-risk driving content, and output a comprehensive driving control instruction obtained by reasoning; The vehicle control module is used to control the vehicle driving according to the comprehensive driving control instructions.

19. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 15 by running the executable instructions.

20. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 16.

21. A computer program product, comprising a computer program and / or instructions, wherein the computer program and / or instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 16.

Images