Automatic driving vehicle environment situation prediction method and device, electronic equipment and storage medium

Abstract

The invention discloses an automatic driving vehicle environment situation prediction method and device, electronic equipment and a storage medium. The method comprises the steps that cloud traffic information and sensing information of a vehicle are acquired; collecting voice instruction information in the vehicle; based on a pre-trained prediction model, the voice instruction information in the vehicle, the cloud traffic information and the perception information of the vehicle serve as input of the model, and the pre-trained prediction model is used for predicting the traffic congestion degree, the traffic safety risk and the road quality; and through the pre-trained prediction model, outputting and obtaining an automatic driving vehicle environment situation prediction condition. On one hand, the accuracy of automatic driving vehicle environment situation prediction is improved, and on the other hand, the current automatic driving feeling can be adjusted according to the environment situation prediction condition.

Description

Technical Field

[0001] The present application relates to the technical fields of autonomous driving and vehicle environmental situation, and particularly relates to a method and device for predicting the environmental situation of an autonomous driving vehicle, an electronic device, and a storage medium. Background Art

[0002] The vehicle environmental situation refers to the traffic information collected by the current vehicle about the area where the vehicle is located through sensors, cloud centers, and other traffic and meteorological platforms, and the environmental situation around the vehicle is deduced based on this information. These traffic information includes, but is not limited to, vehicle driving information, traffic accident, traffic control, weather conditions and other information.

[0003] The accurate definition of the environmental situation around the vehicle is conducive to correctly guiding the vehicle perception system for active perception, and at the same time is conducive to making the vehicle planning and decision-making system more stable and reliable. In addition, the definition and extraction of the environmental situation (driving environmental situation) around the vehicle can also be used as prior knowledge to help the autonomous driving vehicle respond more accurately to the current scene planning.

[0004] In related technologies, autonomous driving vehicles are usually limited to local self-vehicle environment perception, resulting in the inability to predict the environmental situation over a long distance in advance and make advance plans. In addition, when autonomous driving vehicles perform path planning, they do not consider the impact of information of the driver or passengers in the vehicle on the driving environmental situation. Summary of the Invention

[0005] Embodiments of the present application provide a method and device for predicting the environmental situation of an autonomous driving vehicle, an electronic device, and a storage medium, so as to improve the accuracy of predicting the environmental situation of an autonomous driving vehicle and adjust the current autonomous driving experience in real time.

[0006] Embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for predicting the environmental situation of an autonomous driving vehicle, where the method includes:

[0008] Obtain cloud traffic information and the perception information of the vehicle itself;

[0009] Collect voice command information inside the vehicle itself;

[0010] Based on a pre-trained prediction model, use the voice command information inside the vehicle itself, the cloud traffic information, and the perception information of the vehicle itself as inputs to the model, and the pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality; and

[0011] Output the prediction situation of the environmental situation of the autonomous driving vehicle through the pre-trained prediction model.

[0012] In some embodiments, the prediction of the environmental situation of the autonomous vehicle output by the pre-trained prediction model includes:

[0013] Taking the voice command information in the vehicle recognized by the large language model, the encoded cloud traffic information, and the perception information of the vehicle as multi-dimensional feature information and inputting them into the situation inference network, and training the situation inference network to obtain the prediction model;

[0014] Output information of the quantitative description of the environmental situation of the autonomous vehicle including the predicted traffic congestion degree, predicted traffic safety risk, and predicted road quality on different lanes through the prediction model. The traffic congestion degree is used as an index to evaluate whether the vehicle is in a congested scenario; the traffic safety risk is used as an index to evaluate whether the vehicle is in an accident scenario; the road quality is used as an index to evaluate the quality of the road scenario where the vehicle is traveling.

[0015] In some embodiments, based on the pre-trained prediction model, taking the voice command information in the vehicle, the cloud traffic information, and the perception information of the vehicle as the input of the model, the pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality, including:

[0016] Extracting the voice information of the occupants in the vehicle through natural language processing to obtain the first feature information and inputting it into the pre-trained prediction model;

[0017] Collecting cloud traffic information including at least one of the following: meteorological information, traffic congestion degree, whether there is a traffic accident, and whether there is traffic control road conditions, and encoding the cloud traffic information to obtain the second feature information and inputting it into the pre-trained prediction model.

[0018] In some embodiments, based on the pre-trained prediction model, taking the voice command information in the vehicle, the cloud traffic information, and the perception information of the vehicle as the input of the model, the pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality, and further includes:

[0019] Taking the lane information and traffic sign information of the vehicle recognized according to the preset recognition algorithm as the third feature information and inputting it into the pre-trained prediction model;

[0020] Taking the information of obstacles around the vehicle recognized according to the preset perception algorithm as the fourth feature information and inputting it into the pre-trained prediction model.

[0021] In some embodiments, the method further includes:

[0022] In response to the score values corresponding to the traffic congestion level, traffic safety risk, and road quality in the predicted situation of the autonomous driving vehicle environment, adjust the algorithm deployment of the sensors and the physical deployment of the sensors in the perception module in the autonomous driving vehicle;

[0023] And / or, in response to the score values corresponding to the traffic congestion level, traffic safety risk, and road quality in the predicted situation of the autonomous driving vehicle environment, adjust the self-vehicle planned path in the planning module in the autonomous driving vehicle or save it as prior knowledge to the planning module.

[0024] In some embodiments, the method further includes:

[0025] Pre-establish an environmental situation prediction database, store cloud traffic information, the perception information of the self-vehicle on the vehicle side, and voice command information inside the self-vehicle according to time, and construct a training database and a test database;

[0026] Annotate the data in the environmental situation prediction database, and mark the distribution of environmental situation scores in each scene data according to the road condition information at that time;

[0027] When training the prediction model, input the training sample data including at least cloud traffic information, the perception information of the self-vehicle, and voice command information inside the self-vehicle into the deep learning network for training;

[0028] By adjusting the preset loss function, train the deep learning network until it converges to obtain the prediction model.

[0029] In some embodiments, the method further includes:

[0030] According to the predicted situation of the autonomous driving vehicle environment, determine whether the autonomous driving vehicle is in a traffic congestion scene and / or an accident scene;

[0031] If the autonomous driving vehicle has not entered a congested section, plan a new path in advance for driving;

[0032] If the autonomous driving vehicle has entered a congested section, perform lane selection planning according to the current environmental situation result;

[0033] If an accident occurs in the lane in front of the autonomous driving vehicle, select a target lane in advance and adjust the computing power according to the perception key area to perceive the oncoming vehicle behind the adjacent target lane.

[0034] In a second aspect, an embodiment of the present application further provides an autonomous driving vehicle environmental situation prediction device, where the device includes:

[0035] An acquisition module, configured to acquire cloud traffic information and the perception information of the self-vehicle;

[0036] A collection module for collecting voice command information inside the host vehicle.

[0037] An input module for using, based on a pre-trained prediction model, the voice command information inside the host vehicle, the cloud traffic information, and the perception information of the host vehicle as inputs to the model. The pre-trained prediction model is used to predict the traffic congestion level, traffic safety risks, and road quality; and

[0038] An output module for outputting, through the pre-trained prediction model, the predicted situation of the environment of the autonomous vehicle.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute the above method.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the above method.

[0041] The above at least one technical solution adopted in the embodiment of the present application can achieve the following beneficial effects: Obtain cloud traffic information and the perception information of the host vehicle, and collect voice command information inside the host vehicle. Then, based on a pre-trained prediction model, use the voice command information inside the host vehicle, the cloud traffic information, and the perception information of the host vehicle as inputs to the model. The prediction model can output a quantitative result and is used to predict the traffic congestion level, traffic safety risks, and road quality. Finally, through the pre-trained prediction model, output the predicted situation of the environment of the autonomous vehicle. Since the voice command information inside the host vehicle is added, the relevant decisions of the autonomous vehicle can better reflect the concern for the occupants inside the vehicle. Since the adopted prediction model is based on deep learning to classify driving scenarios and quantitatively represent the environmental situation, it is more conducive to making reasonable plans and perception strategies for decision-making. Description of the Drawings

[0042] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0043] Figure 1 It is a schematic diagram of environmental situation reasoning for the method for predicting the environment of an autonomous vehicle in an embodiment of the present application;

[0044] Figure 2Schematic flowchart of the method for predicting the environmental situation of an autonomous vehicle in an embodiment of the present application;

[0045] Figure 3 Schematic diagram of cloud information processing for the method for predicting the environmental situation of an autonomous vehicle in an embodiment of the present application;

[0046] Figure 4 Schematic diagram of situation reasoning for the method for predicting the environmental situation of an autonomous vehicle in an embodiment of the present application;

[0047] Figure 5 Schematic diagram of the structure of the device for predicting the environmental situation of an autonomous vehicle in an embodiment of the present application;

[0048] Figure 6 Schematic diagram of the structure of an electronic device in an embodiment of the present application. Detailed implementation manners

[0049] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0050] When an autonomous vehicle performs an autonomous driving task, the following problems may occur:

[0051] (1) Autonomous vehicles are usually limited to local perception of the vehicle's own environment, which results in the inability to predict the environmental situation over ultra-long distances in advance. This affects the traffic efficiency of autonomous vehicles and also affects a certain level of safety.

[0052] (2) The perception system of autonomous vehicles focuses on information about other vehicles, pedestrians, obstacles, etc. around the vehicle, including but not limited to lane line information. Usually, human drivers can perceive many high-level environmental situation semantics during driving. For example, based on past experience, they know that they need to slow down when there is a pothole or bump ahead, and the road section ahead will definitely be congested during the morning rush hour. However, such information cannot be extracted by the perception system of autonomous vehicles currently. Therefore, if a high-level environmental situation representation can be established, it will be beneficial for the planning system to make safer decision-making plans. Since the current autonomous vehicles only collect information through sensors and infer the characteristics of the surrounding obstacle environment through the algorithms installed, the information of drivers or passengers is not introduced. However, in the actual driving process, it can be found that drivers or passengers will make accurate descriptions according to the current driving environment, and this high-level language description also reflects the driving style (such as risk-stable type, etc.) that each driver or passenger hopes for. Introducing the high-level semantic description of the driving environment by people can assist in the discrimination of the current driving environment situation.

[0053] In summary, when autonomous vehicles perform autonomous driving tasks, they face different task requirements, different carrier device requirements, and different driving scenario requirements. At the same time, autonomous driving is required to be stable and reliable. Therefore, for the perception system to cope with changing environments and tasks and ensure stability and reliability, the autonomous driving system needs to decouple complex scenario tasks, and the perception algorithms solidified in the hardware devices should have higher stability.

[0054] The autonomous vehicle environmental situation prediction method in the embodiments of this application can process the input autonomous vehicle data and output the prediction of the autonomous vehicle environmental situation. Before predicting the autonomous vehicle environmental situation, common data engineering and model construction need to be carried out on the autonomous vehicle data. Before the environmental situation model is deduced, an environmental situation inference model needs to be trained according to the deep learning method, specifically including:

[0055] (1) Database establishment: Store information such as road conditions in the cloud, perception information of the vehicle end, and information of the vehicle itself according to time, and construct a training database and a test database.

[0056] (2) Data annotation: Manually split the stored time series information into multiple scenarios with the same or different time lengths, and for each scenario data, manually annotate the environmental situation score distribution according to the road conditions at that time.

[0057] (3) Training: Use the labeled original data for the training of the overall network.

[0058] The following will detail the technical solutions provided by the embodiments of this application with reference to the accompanying drawings.

[0059] An embodiment of the present application provides a method for predicting the environmental situation of an autonomous vehicle, as Figure 2 shown, a schematic flowchart of the method for predicting the environmental situation of an autonomous vehicle in the embodiment of the present application is provided. The method at least includes the following steps S210 to step S240:

[0060] Step S210, obtain cloud traffic information and the perception information of the vehicle itself.

[0061] "Cloud traffic information" refers to collecting traffic flow information of the cloud platform and weather information of the Internet through vehicle networking technology to a cloud computing platform, and then the cloud computing platform deduces the current traffic situation. And to ensure compatibility, the traffic situation can be encoded into a string of feature vectors through ONE-HOT and sent to the autonomous driving system of the autonomous vehicle in a timely manner.

[0062] It can be understood that the cloud traffic environment features include but are not limited to meteorological information, traffic condition information, sudden accident information, traffic control information, etc.

[0063] It can be understood that the traffic situation includes but is not limited to traffic congestion occurring on the current position section, the current section being unobstructed, or traffic accidents being likely to occur on the current section, etc.

[0064] Step S220, collect voice command information inside the vehicle itself.

[0065] The voice command information inside the vehicle itself includes but is not limited to language description information about the current road conditions, current weather, and current environment issued by vehicle occupants (drivers, passengers).

[0066] By collecting the high-level command information of vehicle occupants and performing speech recognition, the commands related to the environment that are more concerned are extracted. For example, traffic jam ahead, an accident occurred in the right lane, slow driving in the own lane, speeding now, etc.

[0067] It can be understood that the high-level command information of vehicle occupants can be encoded as tokens to form vector data and input into the network for training of one-dimensional features.

[0068] Step S230, based on a pre-trained prediction model, use the voice command information inside the vehicle itself, the cloud traffic information, and the perception information of the vehicle itself as the input of the model. The pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality.

[0069] As Figure 1As shown, the pre-trained prediction model can be obtained by inputting sample data into the situation inference network for training. The visual perception network and the lidar perception network perform perception information fusion on the collected perception information to obtain target obstacle information. The cloud traffic information network also transmits road conditions, weather, and emergencies to the situation inference network. At the same time, the lane information network and the traffic sign network respectively obtain information such as lane line information, traffic lights, speed limit signs, etc., and also transmit them to the situation inference network. The semantic information obtained after the voice command information in the vehicle is recognized by the NLP recognition model is also transmitted to the situation inference network.

[0070] It can be understood that Figure 1 The lane information network and the traffic sign network in [[ ]] adopt relevant algorithms based on machine learning or deep learning, and can adopt, including but not limited to, existing lane recognition algorithms or traffic sign recognition algorithms for processing.

[0071] It can be understood that Figure 1 The visual perception network and the lidar perception network in [[ ]] adopt relevant algorithms based on machine learning or deep learning. For the perception algorithm, it mainly realizes the 2D / 3D perception results of road surface obstacles based on sensors such as millimeter waves, cameras, and lidar, and outputs the characteristics of obstacles and post-processing information such as specific length, width, height, speed, and acceleration.

[0072] It can be understood that the situation inference network accesses the feature layer information inferred by the above networks as input, and then passes through the feature extraction network (for example, using the Resnet network). After extraction, classification scoring is performed according to the relevant situation distribution to obtain the final environmental situation distribution.

[0073] Step S240, through the pre-trained prediction model, output the environmental situation prediction of the autonomous vehicle.

[0074] Using the pre-trained prediction model to output the environmental situation prediction of the autonomous vehicle means using deep learning to realize the classification of driving scenarios and quantitatively representing the environmental situation, which is more conducive to making reasonable decisions for planning.

[0075] As [[ ]] Figure 1 shown, when applying the environmental situation distribution, the output distribution is published to the perception module and the planning module for relevant optimization decisions. The ways of optimization decisions include but are not limited to, in a congested scenario, according to the current environmental situation result, making a lane line selection plan for efficient passage. And in an accident scenario, preferentially select a lane with high efficiency for passage, rather than making a lane change and obstacle avoidance plan after approaching the accident location.

[0076] Through the above method, cloud traffic information, the perception information of the host vehicle, and the voice command information inside the host vehicle are obtained. Then, based on a pre-trained prediction model, the voice command information inside the host vehicle, the cloud traffic information, and the perception information of the host vehicle are used as the inputs of the model. The pre-trained prediction model is used to predict the traffic congestion level, traffic safety risks, and road quality. Finally, through the pre-trained prediction model, the prediction of the environmental situation of the autonomous vehicle is output. Through the prediction model, a quantitative definition of the current scenario can be achieved, so that the real-time adjustment of the current autonomous driving experience can be realized.

[0077] Through the above method, the voice command information inside the host vehicle collected is used as feature information and input into the prediction model. By means of the description of the occupants inside the host vehicle and reasoning about the current driving environment, the planning decisions and perception decisions made can better reflect the concern for the occupants inside the vehicle, and it is also beneficial to improve the riding experience of the people inside the vehicle.

[0078] Through the above method, deep learning is used based on a pre-trained prediction model to classify driving scenarios and quantitatively represent the environmental situation, which is more conducive to making reasonable decisions in planning. At the same time, it also avoids the problem that the support vector machine classification is inaccurate and imprecise, resulting in unstable planning.

[0079] Different from the related technologies, autonomous vehicles usually only limit to the local perception of the host vehicle environment, unable to predict the environmental situation over a long distance in advance and without using the voice command information of the occupants inside the vehicle. Through the above method, on the one hand, the perception results of the host vehicle information are considered, that is, the environmental data around the vehicle are obtained through sensors such as cameras and radars, as well as the relevant descriptions of the current environment by the occupants inside the vehicle, and the current traffic environment situation is automatically analyzed and identified. Including but not limited to, identifying whether there is a vehicle following in front, whether there is another vehicle suddenly changing lanes, and the traffic efficiency of each lane ahead, etc. On the other hand, the perception results of the cloud traffic and meteorological information are considered. Through vehicle networking technology, the traffic flow information of the cloud platform and the weather information of the Internet are collected to the cloud computing platform, and the current traffic situation is deduced by the large model of the cloud computing platform. Including but not limited to, there will be traffic congestion on the current position section, the current section will be unobstructed, or traffic accidents are likely to occur on the current section. By inputting the traffic feature information of the cloud, the vehicle perception information, the high-level semantic description of the occupants, and the state information of the host vehicle, the description of the current environmental situation and the range of the area of interest for perception are output.

[0080] In one embodiment of the present application, the prediction of the environmental situation of the autonomous vehicle output by the pre-trained prediction model includes: using the voice command information in the vehicle of the self-driving vehicle recognized by the large language model, the cloud traffic information after encoding processing, and the perception information of the self-driving vehicle as multi-dimensional feature information and inputting it into the situation inference network, and training the situation inference network to obtain the prediction model; using the prediction model to respectively output the output information of the quantitative description of the environmental situation of the autonomous vehicle, including the predicted traffic congestion degree, predicted traffic safety risk, and predicted road quality on different lanes. The traffic congestion degree is used as an index to evaluate whether the self-driving vehicle is in a congested scenario; the traffic safety risk is used as an index to evaluate whether the self-driving vehicle is in an accident scenario; the road quality is used as an index to evaluate the quality of the road scenario where the self-driving vehicle is traveling.

[0081] The voice command information in the vehicle of the self-driving vehicle is recognized by the large language model and its feature information is extracted. The feature information will be encoded as tokens to form vector data, which is input into the environmental situation model and used as one-dimensional feature. The cloud traffic information needs to be encoded and processed, including but not limited to meteorological information, traffic congestion degree, and whether there are traffic accidents or traffic control road conditions, etc. Then, this kind of information is encoded by one-hot or other similar methods to form numerical features, and is transmitted to each autonomous vehicle in a timely manner through vehicle-road-cloud network communication. In this way, for the situation perception of the driving environment, in addition to paying attention to the information of the self-driving vehicle, cloud data is also collected, which allows the self-driving vehicle to make advanced plans, improve traffic efficiency, and avoid traffic accidents to a certain extent.

[0082] Further, as Figure 4 shown, the prediction model is used to respectively output the output information of the quantitative description of the environmental situation of the autonomous vehicle, including the predicted traffic congestion degree, predicted traffic safety risk, and predicted road quality on different lanes. Using, including but not limited to, a deep learning classification network, taking the feature layer information of the network inference as input, and then passing through the feature extraction network of the fully connected layer to extract the traffic congestion judgment, traffic risk judgment, and road quality judgment. After the extraction is completed, classification scoring is carried out according to the situation distribution shown in Table 1.

[0083] Table 1

[0084]

[0085] As can be seen from Table 1, the prediction model is used to respectively output the output information of the quantitative description of the environmental situation of the autonomous vehicle, including the predicted traffic congestion degree, traffic safety risk, and road quality on different lanes, which can be evaluated for each lane.

[0086] It can be understood that the degree of traffic congestion is used as an index to evaluate whether the host vehicle is in a congested scenario. When dividing the congested scenario, it is possible to determine whether the section where the host vehicle is located is in a traffic congestion scenario by calculating the vehicle density on the road in real time and combining with the road condition information released by the traffic department.

[0087] It can be understood that the traffic safety risk is used as an index to evaluate whether the host vehicle is in an accident scenario. When dividing the accident scenario, one possible scenario is that when there are many vehicles during the morning rush hour, traffic accidents often occur in a certain lane, then it is considered that vehicle accumulation congestion will occur in the current lane.

[0088] It can be understood that the road quality is used as an index to evaluate the quality of the road scenario where the host vehicle is traveling. The road quality includes but is not limited to whether there are large pits on the road, whether there is water accumulation on the road surface, and whether there is snow on the middle road. The road quality can be used as an index to indicate whether the road is conducive to the smooth driving of autonomous vehicles.

[0089] The above method uses deep learning to classify driving scenarios and quantitatively represents the environmental situation, which is more conducive to making reasonable decisions in planning. At the same time, it also avoids the problem of inaccurate and imprecise classification by the support vector machine, resulting in unstable planning.

[0090] It can be understood that the output information of the quantitative description of the autonomous vehicle's environmental situation, which outputs the predicted traffic congestion degree, traffic safety risk, and road quality on different lanes, iterates the prediction model parameters according to the updated information in the voice command information in the host vehicle, the cloud traffic information, and the perception information of the host vehicle.

[0091] In an embodiment of the present application, based on a pre-trained prediction model, the voice command information in the host vehicle, the cloud traffic information, and the perception information of the host vehicle are used as inputs to the model. The pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality, including: extracting first feature information from the voice information of the occupants in the host vehicle through natural language processing and inputting it into the pre-trained prediction model; collecting cloud traffic information including at least one of the following: meteorological information, traffic congestion degree, whether there is a traffic accident, and whether there is traffic control road condition, and encoding the cloud traffic information to obtain second feature information and inputting it into the pre-trained prediction model.

[0092] The first feature information is the semantic information extracted from the voice information of the occupants in the host vehicle through natural language processing.

[0093] By using a large language model, since the large language model has powerful semantic understanding capabilities, it can infer the current driving environment based on the descriptions of the vehicle occupants, and thus the planning decisions and perception decisions made are more focused on the vehicle occupants, which is also beneficial to improving the riding experience of the passengers in the vehicle.

[0094] The high-level command information of the occupants inside the host vehicle can be subjected to speech recognition through an in-vehicle / cloud large model, and the commands related to the environment that are of more concern can be extracted. The semantic information obtained includes, but is not limited to, high-level semantic information such as there is a traffic jam ahead, an accident has occurred in the right lane, the vehicle is moving slowly in this lane, and the vehicle is speeding now, etc.

[0095] The second feature information is the information obtained by the autonomous vehicle from the cloud.

[0096] Such as Figure 3 As shown, in the collection stage of the cloud information, the cloud computing platform will collect meteorological information, traffic congestion levels, and whether there are traffic accidents or traffic control road conditions on the public Internet. In the encoding stage of the cloud information, such information is encoded into numerical features by one-hot or other similar methods, and is timely transmitted to each autonomous vehicle terminal system through the vehicle-road-cloud network system. It can be understood that such information is mainly encoded into features by methods such as MLP or EMB before the situation reasoning. For the situation perception of the driving environment, in addition to paying attention to the information of the host vehicle, cloud data is also collected, which can enable the host vehicle to make advanced plans, improve traffic efficiency, and also avoid traffic accidents to a certain extent.

[0097] In an embodiment of the present application, based on a pre-trained prediction model, the voice command information inside the host vehicle, the cloud traffic information, and the perception information of the host vehicle are used as the inputs of the model. The pre-trained prediction model is used to predict the traffic congestion level, traffic safety risks, and road quality, and further includes: using the lane information and traffic sign information of the host vehicle identified according to a preset recognition algorithm as the third feature information and inputting it into the pre-trained prediction model; using the obstacle information of the host vehicle identified according to a preset perception algorithm as the fourth feature information and inputting it into the pre-trained prediction model.

[0098] Identify the traffic signs and lane lines of the host vehicle according to the vision / laser perception model, and transmit the feature information to the situation reasoning network. For the perception of other surrounding interaction target information, perceive the target obstacles of the host vehicle based on in-vehicle sensors, and transmit the feature information to the situation reasoning network.

[0099] The third feature information, as the perception of the lane information and traffic sign information of the autonomous vehicle, can adopt common perception algorithms in related technologies, such as processing based on existing lane recognition algorithms and traffic sign recognition algorithms.

[0100] The fourth characteristic information is used as the target information perception of the autonomous vehicle, and common perception algorithms in related technologies are adopted. For example, based on sensors such as millimeter waves, cameras, and lidar, the 3D perception results of road obstacles are realized, and the characteristics of the obstacles and the post-processing information such as the specific length, width, height, speed, heading angle, and acceleration are output.

[0101] In an embodiment of the present application, the method further includes: in response to the score values corresponding to the traffic congestion degree, traffic safety risk, and road quality in the environmental situation prediction of the autonomous vehicle, adjusting the algorithm deployment of the sensors and the physical deployment of the sensors in the perception module in the autonomous vehicle; and / or, in response to the score values corresponding to the traffic congestion degree, traffic safety risk, and road quality in the environmental situation prediction of the autonomous vehicle, adjusting the self-vehicle planned path in the planning module or saving it as prior knowledge to the planning module.

[0102] When responding to any one or a combination of the score values corresponding to the traffic congestion degree, traffic safety risk, and road quality in the environmental situation prediction of the autonomous vehicle, identify whether there is a vehicle following, whether there is a sudden lane change by other vehicles, the traffic efficiency of each lane ahead, etc., and output the current environmental situation description and the range of the interested perception area.

[0103] When responding to any one or a combination of the score values corresponding to the traffic congestion degree, traffic safety risk, and road quality in the environmental situation prediction of the autonomous vehicle, decisions are made separately in the perception module and the planning module, so as to adjust the perception range and ability according to the current situation description. For example, for key perception areas, drivable areas, or areas prone to blind spots, a model with greater computing power will be replaced and multiple sensors in this direction will be enabled for key perception. Another example is the response to the situation. For example, if the current perceived situation is an accident in the left lane and the right lane is relatively unobstructed, the planning system can arrange the lane change plan according to the traffic efficiency.

[0104] In an embodiment of the present application, the method further includes: pre-establishing an environmental situation prediction database, storing cloud traffic information, the perception information of the self-vehicle on the vehicle side, and the voice command information in the self-vehicle in the vehicle according to time, and constructing a training database and a test database; annotating the data in the environmental situation prediction database, and marking the environmental situation score distribution according to the road condition information in each scenario data; when training the prediction model, inputting the training sample data including at least cloud traffic information, the perception information of the self-vehicle, and the voice command information in the self-vehicle into the deep learning network for training; and training the deep learning network to convergence to obtain the prediction model by adjusting the preset loss function.

[0105] Store information such as road conditions in the cloud, vehicle perception information, and information of the host vehicle according to time, and construct a training database and a test database. Manually split the stored time series information into multiple scenarios each with a duration of T seconds. For each scenario data, manually label the environmental situation score distribution according to the road conditions at that time, and use the labeled original data for the training of the overall network.

[0106] Further, when training the prediction model, input training sample data including at least cloud traffic information, the host vehicle's perception information, and voice command information inside the host vehicle into the deep learning network for training; by adjusting the preset loss function, train the deep learning network until it converges to obtain the prediction model. Similarly, the input features during the training process are the same as those during the validation / test model.

[0107] In an embodiment of the present application, the method further includes: judging whether the autonomous vehicle is in a traffic congestion scenario and / or an accident scenario according to the prediction of the autonomous vehicle's environmental situation; if the autonomous vehicle has not entered a congested section, plan a new route in advance for driving; if the autonomous vehicle has entered a congested section, perform lane selection planning according to the current environmental situation result; if an accident occurs in the lane in front of the autonomous vehicle, select a target lane in advance and adjust the computing power according to the perceived key area to sense the oncoming vehicles in the rear of the adjacent target lane.

[0108] When applying the environmental situation distribution, publish the output distribution to the perception module and the planning module for different decisions. The following is an explanation for typical scenarios:

[0109] (1) Congestion scenario: By real-time calculation and analysis to obtain the vehicle density on the road and combining with the road condition information released by the traffic department, it can be judged that the section where the host vehicle is located is in a traffic congestion scenario.

[0110] For the congestion scenario, if the host vehicle (autonomous vehicle) has not entered the congested section, a new route can be planned in advance for driving. The selection of the new route mainly depends on the global navigation system GNSS and the cloud traffic condition assessment.

[0111] For the congestion scenario, if the host vehicle (autonomous vehicle) has already driven in the congested section, according to the priority of commuting efficiency, lane selection planning can be performed according to the current environmental situation result for high-efficiency passing. At the same time, the perception system can concentrate the computing power to preferentially solve the vehicle states in the key attention areas to avoid traffic accidents.

[0112] (2) Accident scenario: During the morning rush hour, there are many vehicles, and traffic accidents often occur in a certain lane. At this time, vehicle congestion will accumulate in this lane. When the cloud navigation computing center calculates the environmental situation here and finds that traffic accidents occur frequently, the safety risk and congestion coefficient will increase. The self-driving vehicle will receive this information at a long distance. At the same time, combined with the situation awareness of the self-driving vehicle's environment, the planning module of the self-driving vehicle can change lanes at a long distance and preferentially select a lane with high-efficiency traffic instead of changing lanes and bypassing obstacles after approaching the accident location. Preferably, during the process of planning a lane change, the perception module of the self-driving vehicle will adjust the computing power requirements according to the key perception areas of the environmental situation, and focus on perceiving the oncoming vehicles from the rear of the adjacent target lane to achieve accurate and safe lane change operations.

[0113] The embodiment of the present application also provides an environmental situation prediction device 500 for a self-driving vehicle, as Figure 5 shown, which provides a structural schematic diagram of the environmental situation prediction device for a self-driving vehicle in the embodiment of the present application. The environmental situation prediction device 500 for a self-driving vehicle at least includes: an acquisition module 510, a collection module 520, an input module 530, and an output module 540, where:

[0114] In an embodiment of the present application, the acquisition module 510 is specifically configured to: acquire cloud traffic information and the perception information of the self-vehicle.

[0115] "Cloud traffic information" refers to collecting traffic flow information of the cloud platform and weather information of the Internet through vehicle networking technology to the cloud computing platform, and then the cloud computing platform deduces the current traffic situation. And to ensure compatibility, the traffic situation will be encoded into a string of feature vectors through ONE-HOT and transmitted to the automatic driving system of the self-driving vehicle in a timely manner.

[0116] It can be understood that the cloud traffic environment features include but are not limited to meteorological information, traffic condition information, and sudden accident information.

[0117] It can be understood that the traffic situation includes but is not limited to traffic congestion occurring in the current position section, the current section being unobstructed, or traffic accidents being likely to occur in the current section.

[0118] In an embodiment of the present application, the collection module 520 is specifically configured to: collect voice command information inside the self-vehicle.

[0119] The voice command information inside the self-vehicle includes but is not limited to language description information about the current road conditions, current weather, and current environment issued by the people inside the vehicle (driver, passenger).

[0120] By collecting the high-level instruction information of the vehicle occupants and performing speech recognition, the instructions related to the environment that are of particular concern are extracted. For example, traffic jam ahead, accident in the right lane, slow driving in the own lane, speeding now, etc.

[0121] It can be understood that the high-level instruction information of the vehicle occupants will be token-encoded to form vector data and input into the network for training of one-dimensional features.

[0122] In an embodiment of the present application, the input module 530 is specifically configured to: based on a pre-trained prediction model, use the voice instruction information in the own vehicle, the cloud traffic information, and the perception information of the own vehicle as the input of the model, and the pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality.

[0123] As Figure 1 shown, the pre-trained prediction model can be obtained by training the sample data input into the situation inference network. The visual perception network and the lidar perception network perform perception information fusion on the collected perception information to obtain target obstacle information. The cloud traffic information network obtains road conditions, weather, and emergencies and also transmits them to the situation inference network. At the same time, the lane information network and the traffic sign network respectively obtain lane line information, traffic lights, speed limit signs, etc. information, and also transmit them to the situation inference network. The voice instruction information in the own vehicle is recognized by the NLP recognition model to obtain semantic information, and also transmitted to the situation inference network.

[0124] It can be understood that Figure 1 the lane information network and the traffic sign network in

[0125] It can be understood that Figure 1 the visual perception network and the lidar perception network in

[0126] use relevant algorithms based on machine learning or deep learning. For the perception algorithm, it mainly realizes the 2D / 3D perception results of road surface obstacles based on sensors such as millimeter waves, cameras, and lidar, and outputs the features of the obstacles and the post-processing information such as the specific length, width, height, speed, and acceleration.

[0127] In an embodiment of the present application, the output module 540 is specifically configured to: through the pre-trained prediction model, output the environmental situation prediction of the autonomous vehicle.

[0128] The pre-trained prediction model is used to output the prediction of the environmental situation of the autonomous vehicle, that is, to classify the driving scenarios by using deep learning, and to quantitatively represent the environmental situation, so as to be more conducive to making reasonable decisions in planning.

[0129] As Figure 1 shown, when applying the environmental situation distribution, the output distribution is published to the perception module and the planning module for relevant optimization decisions. The ways of optimization decisions include, but are not limited to, selecting the lane line according to the current environmental situation result in the congestion scenario for efficient passage. And in the accident scenario, giving priority to selecting the lane with high efficiency for passage, rather than making a lane change and obstacle avoidance plan after approaching the accident site.

[0130] It can be understood that the above-mentioned autonomous vehicle environmental situation prediction device can implement each step of the autonomous vehicle environmental situation prediction method provided in the foregoing embodiment. The relevant explanations about the autonomous vehicle environmental situation prediction method are applicable to the transaction reconciliation device, and will not be elaborated here.

[0131] Figure 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 6 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include internal memory, such as high-speed random access memory (Random-Access Memory, RAM), and may also include non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0132] The processor, network interface, and memory can be interconnected through the internal bus. The internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 6 only a bidirectional arrow is used in

[0133] A memory for storing programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0134] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an automatic driving vehicle environment situation prediction device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0135] Obtain cloud traffic information and the perception information of the vehicle itself;

[0136] Collect voice command information inside the vehicle itself;

[0137] Based on a pre-trained prediction model, use the voice command information inside the vehicle itself, the cloud traffic information, and the perception information of the vehicle itself as the input of the model. The pre-trained prediction model is used to predict the traffic congestion degree, traffic safety risk, and road quality; and

[0138] Output the automatic driving vehicle environment situation prediction through the pre-trained prediction model.

[0139] The above as in this application Figure 2The method executed by the automatic driving vehicle environment situation prediction device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0140] The electronic device can also execute Figure 2 the method executed by the automatic driving vehicle environment situation prediction device in Figure 2 the illustrated embodiment and implement the functions of the automatic driving vehicle environment situation prediction device in

[0141] Embodiments of the present application also propose a computer-readable storage medium that stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 2 the method executed by the automatic driving vehicle environment situation prediction device in the illustrated embodiment and specifically used to execute:

[0142] Obtain cloud traffic information and the perception information of the vehicle itself;

[0143] Collect voice command information inside the vehicle itself;

[0144] Based on a pre-trained prediction model, the voice command information inside the host vehicle, the cloud traffic information, and the perception information of the host vehicle are used as the inputs of the model. The pre-trained prediction model is used to predict the traffic congestion level, traffic safety risks, and road quality; and

[0145] Through the pre-trained prediction model, the predicted situation of the autonomous driving vehicle environment is output.

[0146] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0148] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0154] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0155] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for predicting the environmental situation of an autonomous driving vehicle, wherein: The method comprises: Obtain cloud traffic information and vehicle perception information; Collect voice command information from inside the car; Based on a pre-trained prediction model, the voice command information in the vehicle, the cloud traffic information, and the perception information of the vehicle are used as inputs of the model, and the pre-trained prediction model is used to predict the degree of traffic congestion, traffic safety risks, and road quality; and The pre-trained prediction model is used to output the prediction of the environmental situation of the autonomous driving vehicle.

2. The method of claim 1, wherein: The output of the pre-trained prediction model to obtain the prediction of the autonomous driving vehicle environment situation includes: Input the voice command information in the vehicle recognized by the large language model, the cloud traffic information after encoding, and the perception information of the vehicle as multi-dimensional feature information into the situation reasoning network, and train the situation reasoning network to obtain the prediction model; The prediction model outputs output information of a quantitative description of the environmental situation of the autonomous driving vehicle, including the predicted traffic congestion degree, predicted traffic safety risk and predicted road quality on different lanes. The traffic congestion degree is used as an indicator for evaluating whether the vehicle is in a congestion scene; the traffic safety risk is used as a risk indicator for evaluating whether the vehicle is in an accident scene; and the road quality is used as a quality indicator for evaluating the road scene on which the vehicle is traveling.

3. The method of claim 2, wherein: The pre-trained prediction model uses the voice command information in the vehicle, the cloud traffic information, and the perception information of the vehicle as inputs of the model. The pre-trained prediction model is used to predict the degree of traffic congestion, traffic safety risks, and road quality, including: Extracting the voice information of the occupant in the vehicle through natural language processing to obtain first feature information, and inputting the first feature information into a pre-trained prediction model; Collect cloud traffic information including at least one of the following: weather information, traffic congestion level, whether there is a traffic accident, whether there is a traffic control road condition, encode the cloud traffic information to obtain second feature information, and input it into a pre-trained prediction model.

4. The method of claim 2, wherein: The pre-trained prediction model uses the voice command information in the vehicle, the cloud traffic information, and the perception information of the vehicle as inputs of the model. The pre-trained prediction model is used to predict the degree of traffic congestion, traffic safety risks, and road quality, and also includes: The lane information and traffic sign information of the vehicle identified by the preset recognition algorithm are used as the third feature information and input into the pre-trained prediction model; The obstacle information around the vehicle identified by the preset perception algorithm is used as the fourth feature information and input into the pre-trained prediction model.

5. The method of claim 4, wherein: The method further comprises: In response to the traffic congestion level, traffic safety risk, and road quality scores in the predicted situation of the autonomous driving vehicle environment, adjusting the algorithm deployment of sensors in the perception module and the physical deployment of sensors in the autonomous driving vehicle; And / or, in response to the score values ​​corresponding to the degree of traffic congestion, traffic safety risk and road quality in the predicted environmental situation of the autonomous driving vehicle, the self-vehicle planned path in the planning module of the autonomous driving vehicle is adjusted or saved as prior knowledge in the planning module.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: Establish an environmental situation prediction database in advance, store cloud traffic information, vehicle-side perception information, and voice command information in the vehicle according to time, and build a training database and a test database; Mark the data in the environmental situation prediction database, and mark the environmental situation score distribution in each scene data according to the road conditions at that time; When training the prediction model, inputting training sample data including at least cloud traffic information, perception information of the vehicle, and voice command information in the vehicle into a deep learning network for training; By adjusting the preset loss function, the deep learning network is trained until convergence to obtain the prediction model.

7. The method according to any one of claims 1 to 5, wherein: The method further comprises: Determining whether the autonomous driving vehicle is in a traffic congestion scenario and / or an accident scenario based on the predicted environmental situation of the autonomous driving vehicle; If the autonomous vehicle does not enter a congested section of road, it will plan a new route in advance; If the autonomous vehicle has entered a congested section of road, it will select and plan the lane line based on the current environmental situation results; If an accident occurs in the lane ahead of the autonomous vehicle, the target lane is selected in advance and the computing power is adjusted based on the key perception areas to sense the vehicle coming from behind near the target lane.

8. An automatic driving vehicle environment situation prediction device, wherein: The device comprises: The acquisition module is used to obtain cloud traffic information and the perception information of the vehicle; A collection module, used to collect voice command information from inside the vehicle; An input module, for using the voice command information in the vehicle, the cloud traffic information, and the perception information of the vehicle as inputs of the model based on a pre-trained prediction model, wherein the pre-trained prediction model is used to predict the degree of traffic congestion, traffic safety risks, and road quality; and The output module is used to output the predicted situation of the autonomous driving vehicle environment through the pre-trained prediction model.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images