Automatic driving decision-making method and device, vehicle and electronic equipment

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an automatic driving decision-making method and device, a vehicle and electronic equipment, and the method comprises the steps: obtaining a time sequence look-around image of the vehicle in a driving process; determining a plurality of initial detection object reasoning descriptions of the vehicle in the current driving environment according to the time sequence look-around image; performing first quality scoring on each initial detection object reasoning description to obtain a first quality score of each initial detection object reasoning description; screening a target detection object reasoning description from the plurality of initial detection object reasoning descriptions according to the first quality score of each initial detection object reasoning description; and determining a target automatic driving decision of the vehicle in the current driving environment according to the target detection object reasoning description. The accuracy of the reasoning description of the target detection object is ensured by adopting the time sequence look-around picture and a quality scoring mechanism for the reasoning description of the initial detection object, so that the accuracy of the determination result of the target automatic driving decision is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an autonomous driving decision-making method, device, vehicle and electronic equipment. Background Art

[0002] With the rapid development of science and technology, the application of autonomous driving technology in the field of modern transportation is becoming increasingly important. Intelligent driving technology can significantly reduce traffic accidents caused by human driving errors and ensure road safety. It is an inevitable trend in the future development of automobiles.

[0003] In related technologies, driving decisions are usually made based on visual perception of objects in the driving scene and then according to preset driving rules; or a decision-making system based on high-precision maps and the Global Positioning System (GPS) uses GPS positioning and high-precision maps to achieve vehicle navigation and driving decisions.

[0004] However, in special scenarios such as tunnels, due to the complex and changeable lighting conditions in tunnels, sensors are easily affected by lighting differences when vehicles enter and exit tunnels, resulting in insufficient lighting, and GPS signals in tunnels are easily weakened or lost, leading to difficulties in positioning and navigation, resulting in low accuracy of the final autonomous driving decision-making results. Summary of the invention

[0005] The present application provides an autonomous driving decision-making method, device, vehicle and electronic device to solve the defects of related technologies such as low accuracy of the final autonomous driving decision generation results.

[0006] The first aspect of the present application provides an autonomous driving decision-making method, comprising:

[0007] Acquire a time-series surround view image of the vehicle during driving; wherein the time-series surround view image includes a plurality of consecutive frames of detection images of the vehicle under a plurality of different camera perspectives at the same time;

[0008] Determining, based on the time-series surround view image, a plurality of initial detection object inference descriptions of the vehicle in a current driving environment;

[0009] Performing a first quality scoring on each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions;

[0010] Filtering a target detection object reasoning description from the multiple initial detection object reasoning descriptions according to the first quality score of each of the initial detection object reasoning descriptions;

[0011] According to the target detection object reasoning description, a target autonomous driving decision of the vehicle in the current driving environment is determined.

[0012] In an optional implementation, the acquiring of the time-series surround view images of the vehicle during driving includes:

[0013] Acquire multiple frames of detection images of the vehicle under multiple different camera viewing angles at the same time;

[0014] According to the position information corresponding to each camera perspective, the detection images of the vehicle under multiple different camera perspectives at the same time in the continuous multiple frames are spliced ​​to obtain a time-series surround view image of the vehicle during driving.

[0015] In an optional implementation, determining, based on the time-series surround view images, multiple initial detection object inference descriptions of the vehicle in the current driving environment includes:

[0016] Determining spatial information and temporal information of the current driving environment according to the temporal surround view image;

[0017] According to the spatial information and temporal information of the current driving environment, keyword screening is performed in a preset vocabulary set to obtain a plurality of target keywords corresponding to the current driving environment; wherein the target keywords at least include the name of the detection object;

[0018] Based on the multiple target keywords, multiple initial detection object inference descriptions of the vehicle in the current driving environment are determined.

[0019] In an optional implementation, performing a first quality scoring on each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions includes:

[0020] Performing a first accuracy score, a first relevance score, a first completeness score, a first clarity score, and a first adaptability score on each of the initial detection object reasoning descriptions, respectively, to obtain a first accuracy score, a first relevance score, a first completeness score, a first clarity score, and a first adaptability score for each of the initial detection object reasoning descriptions;

[0021] Performing weighted fusion on the first accuracy score, the first relevance score, the first integrity score, the first clarity score and the first adaptability score of each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions;

[0022] Among them, the first quality score includes the first accuracy score, the first relevance score, the first completeness score, the first clarity score and the first adaptability score, the first accuracy score is used to characterize whether the location description and attribute description of the detection object in the initial detection object description match the current driving environment, the first relevance score is used to characterize whether the location description and attribute description of the detection object in the initial detection object description match the driving decision, the first completeness score is used to characterize the coverage degree of the initial detection object description of the necessary information for driving decision, the first clarity score is used to characterize whether the language expression of the initial detection object description is clear, and the first adaptability score is used to characterize whether the initial detection object description is applicable in the current driving environment.

[0023] In an optional implementation, determining a target automatic driving decision of the vehicle in a current driving environment based on the target detection object reasoning description includes:

[0024] Determining a plurality of initial intermediate guidance information for the vehicle in a current driving environment according to the target detection object reasoning description;

[0025] Performing a second quality scoring on each of the initial intermediate guidance information to obtain a second quality score of each of the initial intermediate guidance information;

[0026] Filtering target intermediate guidance information from the plurality of initial intermediate guidance information according to the second quality score of each of the initial intermediate guidance information;

[0027] determining a target autonomous driving decision for the vehicle in a current driving environment according to the target intermediate guidance information;

[0028] The target intermediate guidance information is used to characterize the impact of the target detection object reasoning description on the driving of the vehicle.

[0029] In an optional implementation, determining a target automatic driving decision of the vehicle in a current driving environment according to the target intermediate guidance information includes:

[0030] determining, based on the target intermediate guidance information, a plurality of initial autonomous driving decisions for the vehicle in a current driving environment;

[0031] Performing a third quality scoring on each of the initial autonomous driving decisions to obtain a third quality score for each of the initial autonomous driving decisions;

[0032] According to the third quality score of each of the initial autonomous driving decisions, a target autonomous driving decision is screened from the multiple initial autonomous driving decisions.

[0033] A second aspect of the present application provides an automatic driving decision-making device, comprising:

[0034] An acquisition module is used to acquire a time-series surround view image of the vehicle during driving; wherein the time-series surround view image includes a plurality of consecutive frames of detection images of the vehicle under a plurality of different camera perspectives at the same time;

[0035] Determine a model for determining, based on the time-series surround view image, a plurality of initial detection object reasoning descriptions of the vehicle in a current driving environment;

[0036] A scoring module, used for performing a first quality scoring on each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions;

[0037] A screening module, configured to screen a target detection object reasoning description from the plurality of initial detection object reasoning descriptions according to a first quality score of each of the initial detection object reasoning descriptions;

[0038] A decision model is used to determine a target autonomous driving decision for the vehicle in a current driving environment based on the target detection object inference description.

[0039] A third aspect of the present application provides a vehicle, comprising: a surround view image acquisition device and an automatic driving decision device;

[0040] The surround view image acquisition device is used to collect the time-series surround view images of the vehicle during driving, and send the collected time-series surround view images to the automatic driving decision-making device;

[0041] The autonomous driving decision device is used to receive the temporal surround view image, and adopt the autonomous driving decision method described in the first aspect and various possible designs of the first aspect above to determine the target autonomous driving decision of the vehicle in the current driving environment based on the temporal surround view image.

[0042] A fourth aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0043] The memory stores computer-executable instructions;

[0044] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0045] A fifth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0046] The sixth aspect of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method described in the first aspect and various possible designs of the first aspect.

[0047] The technical solution of this application has the following advantages:

[0048] The present application provides an automatic driving decision-making method, device, vehicle and electronic device, the method comprising: obtaining a sequential surround view image of the vehicle during driving; wherein the sequential surround view image includes a plurality of consecutive frames of detection images of the vehicle under a plurality of different camera perspectives at the same time; determining a plurality of initial detection object reasoning descriptions of the vehicle in the current driving environment according to the sequential surround view image; performing a first quality score on each initial detection object reasoning description to obtain a first quality score of each initial detection object reasoning description; screening a target detection object reasoning description from a plurality of initial detection object reasoning descriptions according to the first quality score of each initial detection object reasoning description; determining a target automatic driving decision of the vehicle in the current driving environment according to the target detection object reasoning description. The method provided by the above scheme ensures the accuracy of the target detection object reasoning description by adopting a sequential surround view image and a quality scoring mechanism for the initial detection object reasoning description, thereby improving the accuracy of the target automatic driving decision determination result. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0050] Figure 1 This is a schematic diagram of the structure of the autonomous driving decision-making system on which the embodiments of the present application are based;

[0051] Figure 2 A flowchart of an autonomous driving decision-making method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of an exemplary autonomous driving decision-making method provided in an embodiment of the present application;

[0053] Figure 4 A flowchart of another exemplary autonomous driving decision-making method provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structure of an automatic driving decision-making device provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0060] In recent years, autonomous driving technology has made significant progress, and many automakers are committed to developing efficient and safe autonomous driving systems. Breakthroughs in this field are due to the continuous advancement of sensor technology, computing power and machine learning algorithms, which enable autonomous vehicles to achieve different levels of autonomous driving capabilities in a variety of environments such as cities and highways. The application of advanced lidar, cameras and ultrasonic sensors enables vehicles to perceive the surrounding environment in real time and process data and make decisions through complex algorithms.

[0061] Despite its success in many scenarios, autonomous driving in tunnels still faces many challenges.

[0062] Specifically, these challenges mainly include:

[0063] 1. Lighting changes: The lighting conditions in the tunnel are complex and changeable. Due to the difference in light at both ends of the tunnel, the sensor may encounter sudden lack of light after the vehicle enters the tunnel. In addition, the reflection phenomenon of the walls, ground and other surfaces in the tunnel may cause the image read by the sensor to be distorted, making it difficult to identify the environment. Therefore, how to maintain high-precision perception capabilities in a low-light environment is an urgent problem to be solved.

[0064] 2. Space limitations: Tunnels are usually narrow, and vehicles must accurately judge the location of turns and obstacles while driving. Due to the geometric shape of the tunnel, the vehicle's control and steering must be very precise, and any slight error may cause an accident. In addition, the narrow space also limits the maneuverability of the vehicle, increasing the difficulty of safe avoidance in emergency situations.

[0065] 3. Dynamic obstacles: In tunnels, you may encounter construction vehicles, maintenance personnel, or other sudden dynamic obstacles. The state of these obstacles may change rapidly, such as the entry and exit of construction vehicles or the intrusion of pedestrians, requiring the autonomous driving system to have fast and flexible response capabilities. The system not only needs to detect these dynamic obstacles in real time, but also needs to quickly assess their impact on driving safety and make corresponding decisions.

[0066] 4. Signal interference: In some cases, GPS signals in tunnels may weaken or be completely lost, which increases the difficulty of positioning and navigation. Without reliable location information, it may be difficult for the vehicle to accurately judge the current environment, which in turn affects the accuracy of decision-making. In the case of signal interference, the autonomous driving system needs to have the ability to navigate autonomously to ensure that the vehicle can safely pass through the tunnel.

[0067] In summary, autonomous driving technology in tunnel scenarios faces multiple challenges such as lighting changes, space limitations, dynamic obstacles and signal interference. These factors make it particularly complicated to implement safe and reliable autonomous driving in tunnel environments, and put forward higher requirements for the research and development of related technologies. These factors make it particularly complicated to implement safe and reliable autonomous driving in tunnels.

[0068] In response to the above problems, the embodiments of the present application provide an automatic driving decision method, device, vehicle and electronic device, the method comprising: obtaining a sequential surround view image of the vehicle during driving; wherein the sequential surround view image includes a plurality of consecutive frames of detection images of the vehicle under a plurality of different camera perspectives at the same time; determining a plurality of initial detection object reasoning descriptions of the vehicle in the current driving environment according to the sequential surround view image; performing a first quality score on each initial detection object reasoning description to obtain a first quality score of each initial detection object reasoning description; screening a target detection object reasoning description from a plurality of initial detection object reasoning descriptions according to the first quality score of each initial detection object reasoning description; determining a target automatic driving decision of the vehicle in the current driving environment according to the target detection object reasoning description. The method provided by the above scheme ensures the accuracy of the target detection object reasoning description by adopting a sequential surround view image and a quality scoring mechanism for the initial detection object reasoning description, thereby improving the accuracy of the target automatic driving decision determination result.

[0069] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0070] First, the structure of the autonomous driving decision-making system on which this application is based is described:

[0071] The automatic driving decision-making method, device, vehicle and electronic device provided in the embodiments of the present application are applicable to vehicles with automatic driving functions. Figure 1 The figure shows a schematic diagram of the structure of the automatic driving decision system based on the embodiment of the present application, which mainly includes a vehicle end, a data acquisition device and an automatic driving decision device. Among them, the data acquisition device is used to collect the time-series surround view images of the vehicle during driving, and send the collected time-series surround view images to the automatic driving decision device, and the automatic driving decision device determines the target automatic driving decision of the vehicle according to the obtained time-series surround view images, and sends the target automatic driving decision to the vehicle end, so that the vehicle end can control the driving state of the vehicle according to the target automatic driving decision or voice report the target automatic driving decision; wherein, the vehicle end includes an electronic control unit (Electronic Control Unit, referred to as: ECU), etc.

[0072] The embodiment of the present application provides an automatic driving decision method for providing automatic driving decision for a vehicle with automatic driving function. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used to provide automatic driving decision for a vehicle with automatic driving function.

[0073] like Figure 2 FIG. 1 is a flow chart of an automatic driving decision-making method provided in an embodiment of the present application, and the method includes:

[0074] Step 201, acquiring a time-series surround view image of the vehicle during driving.

[0075] The time-series surround view image includes multiple consecutive frames of detection images of the vehicle under multiple different camera perspectives at the same time.

[0076] It should be noted that the time-series surround view picture used in the embodiment of the present application combines continuous multi-frame detection images from multiple different camera perspectives of the vehicle at the same time, providing richer and more comprehensive driving scene information. This enables the vehicle's autonomous driving decision-making system to capture the dynamic changes of objects in the scene, such as the acceleration and deceleration of other vehicles and the movement direction of pedestrians, rather than just static information at a certain moment. Compared with a single perspective or a single frame image, the time-series surround view picture greatly enhances the perception of complex driving environments such as tunnels, effectively avoiding the misjudgment or omission of objects due to perspective limitations or information loss, and lays the foundation for the subsequent accurate determination of the detection object reasoning description.

[0077] Specifically, the vehicle is equipped with image acquisition devices such as surround-view cameras, which create a time-series surround-view image by acquiring multiple consecutive frames of detection images from multiple different camera perspectives at the same time.

[0078] The surround view camera actually includes multiple cameras, each of which is deployed at a different position on the vehicle body to collect detection images from different camera perspectives. The detection images from multiple different camera perspectives obtained at each moment contain visual information from different directions around the vehicle at the same moment.

[0079] Step 202 : determining a plurality of initial detection object inference descriptions of the vehicle in the current driving environment based on the time-series surround view images.

[0080] Specifically, based on a preset reasoning model and according to the time-series surround view images, multiple initial detection object reasoning descriptions of the vehicle in the current driving environment can be determined.

[0081] Among them, the preset reasoning model can adopt the infer model, etc.

[0082] Step 203 : Perform a first quality scoring on each initial detection object inference description to obtain a first quality score of each initial detection object inference description.

[0083] Specifically, a first quality score may be performed on each initial detection object reasoning description based on a preset scoring model to obtain a first quality score of each initial detection object reasoning description.

[0084] Among them, the preset scoring model can adopt a critic model, etc.

[0085] Step 204 , screening the target detection object inference description from the multiple initial detection object inference descriptions according to the first quality score of each initial detection object inference description.

[0086] Specifically, the initial detection object inference description with the highest first quality score may be used as the target detection object inference description.

[0087] Step 205, determining the target autonomous driving decision of the vehicle in the current driving environment based on the target detection object inference description.

[0088] It should be noted that the target detection object inference description includes the location information and attribute information of the detection object in the current driving environment. The attribute information includes at least the size and name of the detection object. The detection objects include obstacles, pedestrians, and road signs. For example: 20 meters in front of the vehicle, located in the center of the lane, the attribute information is a detection object with a length of 3 meters and a width of 2 meters. The name of the detection object is construction vehicle (obstacle).

[0089] Specifically, based on the pre-set decision logic and rules, combined with the current state information of the vehicle, such as vehicle speed, driving direction, etc., the target detection object reasoning description can be comprehensively analyzed to determine the target automatic driving decision of the vehicle in the current driving environment. For example: if a stationary obstacle is detected in front and the distance is relatively close, whether emergency braking or avoidance operation is required is determined based on the vehicle speed.

[0090] On the basis of the above embodiments, in order to ensure the accuracy of the sequential surround view image, as an implementable method, in one embodiment, obtaining the sequential surround view image of the vehicle during driving includes:

[0091] Step 211, obtaining multiple frames of detection images of the vehicle under multiple camera viewing angles at the same time;

[0092] Step 2012, according to the position information corresponding to each camera perspective, multiple frames of detection images of the vehicle under multiple different camera perspectives at the same time are spliced ​​to obtain a time-series surround view image of the vehicle during driving.

[0093] It should be noted that the multiple different camera perspectives correspond to the front view FRONT, left front view FRONT_LEFT, right front view FRONT_RIGHT, back view BACK, left back view BACK_LEFT, right back view BACK_RIGHT, etc. Multiple perspectives and LIDAR information (laser radar information) can also be introduced, such as the bird's-eye view BEV (Bird-Eye-View) perspective, so as to obtain more comprehensive driving scene information, so as to make more reliable decisions.

[0094] Specifically, the appropriate image acquisition frequency can be set according to the requirements of the autonomous driving system for real-time environmental perception, such as collecting 30 frames of images per second. The setting of the image acquisition frequency can balance the amount of data processing and the timeliness of scene information acquisition, so as to ensure that sufficiently detailed scene change information can be obtained without causing excessive burden on subsequent processing due to excessive data volume. During vehicle driving, the system triggers the image acquisition devices of each camera perspective to collect images simultaneously at a set frequency. Each image acquisition device captures the vehicle's surrounding environment from its own perspective at the same time, ensuring that the acquired images are time-synchronized, so as to accurately reflect the all-round conditions around the vehicle at the same moment.

[0095] Furthermore, while obtaining the detection images of multiple consecutive frames of the vehicle under multiple different camera perspectives at the same time, the position information corresponding to each detection image is obtained. The position information has been pre-calibrated and stored in the system during the installation and debugging stage of the image acquisition device, and is used to accurately describe the position and shooting angle of each image acquisition device on the vehicle body. Finally, according to the position information corresponding to each camera perspective, the detection images of multiple consecutive frames of the vehicle under multiple different camera perspectives at the same time are spliced ​​to obtain the time-series surround view image of the vehicle during driving. Among them, in order to distinguish the camera perspectives corresponding to different detection images, different color borders can be marked for the detection images of different camera perspectives in the time-series surround view image. In practical applications, when processing each frame of the image, the previous two frames of the preamble picture are input together as the time-series information, which also achieves the purpose of increasing information items.

[0096] Specifically, in one embodiment, before image stitching, each frame of the image is preprocessed. First, image denoising is performed to remove noise caused by camera sensor (image acquisition device) noise or environmental interference to improve image quality. Then, image correction is performed. Due to the optical characteristics of the camera lens, the captured image may have a certain degree of distortion. The image is geometrically corrected through a pre-set correction algorithm to make it more realistically reflect the scene situation. At the same time, the image brightness, contrast and other parameters are adjusted to enhance the visual effect of the image to facilitate subsequent stitching.

[0097] On the basis of the above embodiments, in order to ensure the quality of generating the initial detection object reasoning description, as an implementable method, in one embodiment, multiple initial detection object reasoning descriptions of the vehicle in the current driving environment are determined according to the time-series surround view images, including:

[0098] Step 2021, determining the spatial information and temporal information of the current driving environment according to the temporal surround view image;

[0099] Step 2022, based on the spatial information and temporal information of the current driving environment, keyword screening is performed in a preset vocabulary set to obtain a plurality of target keywords corresponding to the current driving environment; wherein the target keywords at least include the name of the detection object;

[0100] Step 2023, determining multiple initial detection object inference descriptions of the vehicle in the current driving environment based on the multiple target keywords.

[0101] Among them, the spatial information includes at least the position information of at least one detection object, such as the three-dimensional coordinates of the obstacle in the current driving environment, etc. The timing information includes the position change information of the detection object in multiple consecutive frames, and the position change information includes the motion trajectory and motion speed, etc.

[0102] Specifically, when using a preset reasoning model for reasoning, in order to make the reasoning results diverse and provide multiple results for the evaluation model to score, different sampling strategies will be used. These sampling strategies can adjust the diversity and quality of the generated text to adapt to different needs and application scenarios. By selecting a suitable sampling strategy, a balance can be found between the diversity and quality of the generated text. Greedy sampling is suitable for tasks with high accuracy requirements, while top-k sampling can increase the diversity of the generated text to a certain extent. In an embodiment of the present application, top-k sampling is used. Specifically, k keywords are selected from a preset vocabulary set (tokens) as candidates, and then according to the likelihood scores of these keywords, the model is randomly selected from the most likely "k" options to obtain the target keyword. For example, when k=3, the model will randomly select one from the three most likely words. Top-k sampling is an optimization of greedy search. In greedy search, the model always chooses the option with the highest probability; while top-k sampling samples from the top k words (tokens), so that words with higher scores or probabilities have a chance to be selected. In many cases, the randomness brought by this sampling helps improve the quality of generated text.

[0103] Among them, any initial detection object reasoning description is based on multiple target keywords. For example, if the initial detection object reasoning description is "a red car that is slowing down is found 50 meters in front of the vehicle, and there is a construction warning sign nearby", the target keywords include "in front of the vehicle", "50 meters", "red car", "slow down", and "construction warning sign". These keywords jointly construct a complete description of the relevant objects in the current driving environment from aspects such as location, distance, detection object name, action, and environmental elements, providing a comprehensive and accurate basis for subsequent driving decisions.

[0104] On the basis of the above embodiment, in order to ensure the comprehensiveness and reliability of the first quality score, as an implementable manner, in one embodiment, a first quality score is performed on each initial detection object reasoning description to obtain a first quality score of each initial detection object reasoning description, including:

[0105] Step 2031, respectively performing a first accuracy score, a first relevance score, a first completeness score, a first clarity score, and a first adaptability score on each initial detection object reasoning description to obtain a first accuracy score, a first relevance score, a first completeness score, a first clarity score, and a first adaptability score of each initial detection object reasoning description;

[0106] Step 2032, weighted fusion is performed on the first accuracy score, the first relevance score, the first completeness score, the first clarity score and the first adaptability score of each initial detection object inference description to obtain a first quality score of each initial detection object inference description.

[0107] Among them, the first quality score includes a first accuracy score, a first relevance score, a first completeness score, a first clarity score and a first adaptability score. The first accuracy score is used to characterize whether the initial detection object description matches the current driving environment for the location description and attribute description of the detection object. The first relevance score is used to characterize whether the initial detection object description matches the driving decision for the location description and attribute description of the detection object. The first completeness score is used to characterize the degree of coverage of the necessary information for driving decision-making in the initial detection object description. The first clarity score is used to characterize whether the language expression of the initial detection object description is clear. The first adaptability score is used to characterize whether the initial detection object description is applicable in the current driving environment.

[0108] Specifically, each initial detection object reasoning description may be input into a preset scoring model, and the scoring model performs a first quality score on it from multiple dimensions including accuracy, relevance, completeness, clarity, and adaptability.

[0109] Specifically, for the first accuracy score, the scoring model analyzes the reasoning description of each detection object to determine whether it correctly identifies the position and attributes of the object (detection object) in the current driving scene, including whether the description of key attributes such as the position, category, and shape of the object is consistent with the actual scene. For the first relevance score, the scoring model examines the degree of relevance between the description content and the current driving scene and the detection object task. Focus on whether the description revolves around objects in the scene that are closely related to driving decisions, such as vehicles, pedestrians, and obstacles. For the first completeness score, the scoring model checks whether each description covers all necessary information, such as the key features of the object (such as the speed of the vehicle, the walking direction of the pedestrian), the state of motion (stationary, accelerating, decelerating, etc.), and the relative position relationship with the vehicle (distance, angle, etc.). For the first clarity score, the scoring model analyzes the language expression of the description to determine whether it is logically clear and semantically clear, and whether there are any unclear or ambiguous parts. For the first adaptability score, combined with the special conditions of the tunnel scene, such as lighting changes, narrow space, signal interference, etc., the scoring model determines the availability and effectiveness of the description in actual driving.

[0110] Among them, the weights of the first accuracy score, the first relevance score, the first integrity score, the first clarity score and the first adaptability score in the weighted fusion calculation can be determined according to user needs, and the embodiments of the present application are not limited.

[0111] Based on the above embodiment, as an implementable manner, in one embodiment, according to the target detection object reasoning description, determining the target automatic driving decision of the vehicle in the current driving environment includes:

[0112] Step 2051, determining a plurality of initial intermediate guidance information of the vehicle in the current driving environment according to the target detection object reasoning description;

[0113] Step 2052, performing a second quality rating on each initial intermediate guidance information to obtain a second quality score of each initial intermediate guidance information;

[0114] Step 2053, selecting target intermediate guidance information from the plurality of initial intermediate guidance information according to the second quality score of each initial intermediate guidance information;

[0115] Step 2054, determining the target autonomous driving decision of the vehicle in the current driving environment based on the target intermediate guidance information.

[0116] Among them, the target intermediate guidance information is used to characterize the impact of the target detection object reasoning description on the vehicle's driving.

[0117] It should be noted that the second quality score may also be comprehensively determined from multiple dimensions of accuracy, relevance, completeness, clarity and adaptability based on a scoring model.

[0118] Specifically, the initial intermediate guidance information with the highest second quality score may be used as the target intermediate guidance information.

[0119] Furthermore, in one embodiment, multiple initial autonomous driving decisions for the vehicle in the current driving environment can be determined based on the target intermediate guidance information; a third quality score can be performed on each initial autonomous driving decision to obtain a third quality score for each initial autonomous driving decision; and a target autonomous driving decision can be screened from the multiple initial autonomous driving decisions based on the third quality score of each initial autonomous driving decision.

[0120] Specifically, the third quality score can also be determined comprehensively from multiple dimensions of accuracy, relevance, completeness, clarity and adaptability based on the scoring model, and finally the initial autonomous driving decision with the highest third quality score is taken as the target autonomous driving decision.

[0121] For example, Figure 3 As shown, a flow chart of an exemplary automatic driving decision-making method provided by an embodiment of the present application is first input into the inference model as a real-time video frame (time-series surround view image) and a first inference model prompt word, the first inference model prompt word is "please detect objects in the current driving scene", and the output is the object description in the current driving scene (target detection object reasoning description). Then the real-time video frame and the second inference model prompt word are input into the inference model, the second inference model prompt word is "please judge which objects will affect the driving behavior and reasons of this vehicle based on the objects in the current driving scene", and the object description in the current driving scene outputted in the previous step is input at the same time, and the output is the objects and reasons that the objects in the current driving scene affect the driving behavior of this vehicle, that is, the output target intermediate guidance information. Finally, the real-time video frame and the third reasoning model prompt word are input into the reasoning model. The third reasoning model prompt word is "Please give driving behavior suggestions based on the objects and reasons that affect the driving behavior of this vehicle in the current driving scene. At the same time, the objects and reasons that affect the driving behavior of this vehicle in the current driving scene output in the previous step are input, and the final output is the target autonomous driving decision. Among them, the target detection object reasoning description and target intermediate guidance information obtained in the reasoning process serve as context information for the next step in the entire reasoning process.

[0122] For example, Figure 4As shown, a flow chart of another exemplary autonomous driving decision-making method provided in an embodiment of the present application mainly represents the connection and application process of the scoring model and the reasoning model, and takes the process of determining the target detection object reasoning description as an example. First, the temporal surround view image and the first reasoning model prompt word are input into the reasoning model, and the reasoning model generates a plurality of initial detection object reasoning descriptions through reasoning. The obtained plurality of initial detection object reasoning descriptions (Response 1 to N), the temporal surround view image and the first reasoning model prompt word are input into the scoring model, and a first quality score is performed on each initial detection object reasoning description based on the scoring model to screen the target detection object reasoning description, and the screened target detection object reasoning description is input into the reasoning model for subsequent reasoning.

[0123] It should be noted that in the actual application process of the embodiment of the present application, the first reasoning model prompt words, the second reasoning model prompt words and the third reasoning model prompt words are all automatically input to realize the automatic generation of vehicle autonomous driving decisions, that is, when the time-series surround view image is obtained, the first reasoning model prompt words are first input into the reasoning model, when the target detection object reasoning description is obtained, the second reasoning model prompt words are input into the reasoning model, and when the target intermediate guidance information is obtained, the third reasoning model prompt words are input into the reasoning model to obtain the target autonomous driving decision of the vehicle in the current driving environment.

[0124] It should be further explained that the main task of the reasoning model used in the embodiment of the present application is to complete three tasks in the field of autonomous driving, that is, three reasoning steps. Therefore, the reasoning model needs to be fine-tuned using the corresponding data so that it has the ability and knowledge to handle the tasks of the three reasoning steps in this architecture. The fine-tuning process of the reasoning model: Use open source autonomous driving data sets such as nuscenes, waymo, talk2car, and rank2tell to fine-tune the model so that the model has basic grounding and reasoning capabilities such as perception, planning, and decision-making in autonomous driving scenarios. The fine-tuning process is carried out using the ms-swift open source training framework.

[0125] Furthermore, in the embodiment of the present application, the main task of the scoring model is to score relevant answers in the field of autonomous driving. Therefore, the scoring model needs to be fine-tuned using corresponding data. Currently, the open source pre-training data in related fields include PRM800K, LLaVA-Critic-113K, etc., which can be used to fine-tune the scoring model so that it has the ability and knowledge to handle the scoring tasks in this architecture.

[0126] The data in the LLaVA-Critic-113K dataset can be divided into two categories:

[0127] 1. Pointwise-scoring: Score a single model response based on the evaluation prompt (0-100).

[0128] 2. Pairwise-ranking: For two (a pair of) model responses, give a partial order relationship between the two or declare a tie.

[0129] For these two types of training data, single-point ratings and rating reasons are generated by GPT4o, and the pairwise ranking preference results (choose A or B) are collected from existing data sets, and the preference reasons are generated by GPT4o.

[0130] The scoring model fine-tuned by LLaVA-Critic-113K will be able to score and evaluate multimodal samples.

[0131] Furthermore, in one embodiment, after obtaining the target autonomous driving decision of the vehicle in the current driving environment, the vehicle dynamics model and the virtual simulation environment can be used to simulate and verify the determined target autonomous driving decision. Specifically, the target autonomous driving decision (such as acceleration, deceleration, and steering instructions) can be input into the simulation environment to simulate the driving trajectory and state changes of the vehicle after executing the decision in the current driving environment. During the simulation process, the influence of various factors is considered, such as the inertia of the vehicle, road friction, the behavior of other traffic participants, etc. Through simulation, possible risks of the decision can be discovered in advance, such as whether the vehicle will collide with potential obstacles, whether the driving trajectory exceeds the safety range, etc.

[0132] The automatic driving decision method provided by the embodiment of the present application obtains the time-series surround view image of the vehicle during driving; wherein the time-series surround view image includes detection images of multiple consecutive frames of the vehicle under multiple different camera perspectives at the same time; according to the time-series surround view image, multiple initial detection object reasoning descriptions of the vehicle in the current driving environment are determined; each initial detection object reasoning description is scored for the first quality to obtain the first quality score of each initial detection object reasoning description; according to the first quality score of each initial detection object reasoning description, the target detection object reasoning description is screened from the multiple initial detection object reasoning descriptions; according to the target detection object reasoning description, the target automatic driving decision of the vehicle in the current driving environment is determined. The method provided by the above scheme ensures the accuracy of the target detection object reasoning description by adopting the time-series surround view image and the quality scoring mechanism for the initial detection object reasoning description, thereby improving the accuracy of the target automatic driving decision determination result. In addition, a dual expert interaction architecture is adopted, that is, two multimodal large models are adopted, model 1 is the infer model, and model 2 is the critic model. The end-to-end autonomous driving in the tunnel scene is divided into three steps, namely scene detection (outputting the reasoning description of the target detection object), object analysis (outputting the intermediate guidance information of the target), and driving decision suggestions (outputting the target autonomous driving decision). In each step, the reasoning model will generate N possible results, and the scoring model will score these results. Finally, the best-of-N (highest score) decision method is adopted, that is, the result with the highest score is selected as the context for the next step. This architecture based on a multimodal model combines detection, analysis, and decision-making end-to-end to improve the accuracy and consistency of the decision-making process: by introducing the scoring mechanism of the critic model, low-quality suggestions can be effectively filtered, thereby improving the reliability of decisions; and real-time detection and decision-making can flexibly respond to dynamic environmental changes; without relying on high-precision maps and GPS navigation, it can avoid the risks caused by poor signals in tunnels.

[0133] An embodiment of the present application provides an autonomous driving decision-making device for executing the autonomous driving decision-making method provided in the above embodiment.

[0134] like Figure 5 , which is a schematic diagram of the structure of the automatic driving decision-making device provided in an embodiment of the present application. The automatic driving decision-making device 50 includes: an acquisition module 501, a determination module 502, a scoring module 503, a screening module 504 and a decision module 505.

[0135] Among them, an acquisition module is used to acquire a time-series surround view image of the vehicle during driving; wherein the time-series surround view image includes multiple consecutive frames of detection images of the vehicle under multiple different camera perspectives at the same time; a determination model is used to determine multiple initial detection object inference descriptions of the vehicle in the current driving environment based on the time-series surround view image; a scoring module is used to perform a first quality score on each initial detection object inference description to obtain a first quality score for each initial detection object inference description; a screening module is used to screen a target detection object inference description from multiple initial detection object inference descriptions based on the first quality score of each initial detection object inference description; and a decision model is used to determine a target automatic driving decision of the vehicle in the current driving environment based on the target detection object inference description.

[0136] Regarding the autonomous driving decision-making device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0137] The autonomous driving decision-making device provided in the embodiment of the present application is used to execute the autonomous driving decision-making method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0138] An embodiment of the present application provides a vehicle for executing the automatic driving decision-making method provided in the above embodiment.

[0139] like Figure 6 FIG. 1 is a schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle includes: a surround image acquisition device and an automatic driving decision device.

[0140] Among them, the surround view image acquisition device is used to collect the time-series surround view images of the vehicle during the driving process, and send the collected time-series surround view images to the automatic driving decision-making device; the automatic driving decision-making device is used to receive the time-series surround view images, adopt the automatic driving decision-making method provided by the above embodiment, and determine the target automatic driving decision of the vehicle in the current driving environment according to the time-series surround view images.

[0141] Regarding the vehicle in this embodiment, the specific manner in which each part performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0142] The vehicle provided in the embodiment of the present application is used to execute the automatic driving decision-making method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0143] An embodiment of the present application provides an electronic device for executing the autonomous driving decision-making method provided in the above embodiment.

[0144] like Figure 7FIG. 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 70 includes: at least one processor 71 and a memory 72 .

[0145] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the autonomous driving decision-making method provided in the above embodiment.

[0146] The electronic device provided in the embodiment of the present application is used to execute the autonomous driving decision-making method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0147] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the autonomous driving decision-making method provided in any of the above embodiments is implemented.

[0148] The storage medium containing computer executable instructions provided in the embodiments of the present application can be used to store computer executable instructions of the autonomous driving decision-making method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0149] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the autonomous driving decision-making method provided in the aforementioned embodiment.

[0150] The embodiments of the present application provide a computer program product that can be used to execute computer instructions of the autonomous driving decision-making method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0154] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0155] A part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0156] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An automatic driving decision-making method, characterized in that: include: Acquire a time-series surround view image of the vehicle during driving; wherein the time-series surround view image includes a plurality of consecutive frames of detection images of the vehicle under a plurality of different camera perspectives at the same time; Determining, based on the time-series surround view image, a plurality of initial detection object inference descriptions of the vehicle in a current driving environment; Performing a first quality scoring on each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions; Filtering a target detection object reasoning description from the multiple initial detection object reasoning descriptions according to the first quality score of each of the initial detection object reasoning descriptions; According to the target detection object reasoning description, a target autonomous driving decision of the vehicle in the current driving environment is determined.

2. The method according to claim 1, characterized in that The step of acquiring the time-series surround view images of the vehicle during driving includes: Acquire multiple frames of detection images of the vehicle under multiple different camera viewing angles at the same time; According to the position information corresponding to each camera perspective, the detection images of the vehicle under multiple different camera perspectives at the same time in the continuous multiple frames are spliced ​​to obtain a time-series surround view image of the vehicle during driving.

3. The method according to claim 1, characterized in that The step of determining, based on the time-series surround view images, a plurality of initial detection object inference descriptions of the vehicle in the current driving environment includes: Determining spatial information and temporal information of the current driving environment according to the temporal surround view image; According to the spatial information and temporal information of the current driving environment, keyword screening is performed in a preset vocabulary set to obtain a plurality of target keywords corresponding to the current driving environment; wherein the target keywords at least include the name of the detection object; Based on the multiple target keywords, multiple initial detection object inference descriptions of the vehicle in the current driving environment are determined.

4. The method according to claim 1, characterized in that: The performing a first quality scoring on each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions includes: Performing a first accuracy score, a first relevance score, a first completeness score, a first clarity score, and a first adaptability score on each of the initial detection object reasoning descriptions, respectively, to obtain a first accuracy score, a first relevance score, a first completeness score, a first clarity score, and a first adaptability score for each of the initial detection object reasoning descriptions; Performing weighted fusion on the first accuracy score, the first relevance score, the first integrity score, the first clarity score and the first adaptability score of each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions; Among them, the first quality score includes the first accuracy score, the first relevance score, the first completeness score, the first clarity score and the first adaptability score, the first accuracy score is used to characterize whether the location description and attribute description of the detection object in the initial detection object description match the current driving environment, the first relevance score is used to characterize whether the location description and attribute description of the detection object in the initial detection object description match the driving decision, the first completeness score is used to characterize the coverage degree of the initial detection object description of the necessary information for driving decision, the first clarity score is used to characterize whether the language expression of the initial detection object description is clear, and the first adaptability score is used to characterize whether the initial detection object description is applicable in the current driving environment.

5. The method according to claim 1, characterized in that: Determining the target automatic driving decision of the vehicle in the current driving environment according to the target detection object reasoning description includes: Determining a plurality of initial intermediate guidance information for the vehicle in a current driving environment according to the target detection object reasoning description; Performing a second quality scoring on each of the initial intermediate guidance information to obtain a second quality score of each of the initial intermediate guidance information; Filtering target intermediate guidance information from the plurality of initial intermediate guidance information according to the second quality score of each of the initial intermediate guidance information; determining a target autonomous driving decision for the vehicle in a current driving environment according to the target intermediate guidance information; The target intermediate guidance information is used to characterize the impact of the target detection object reasoning description on the driving of the vehicle.

6. The method according to claim 5, characterized in that Determining a target automatic driving decision of the vehicle in the current driving environment according to the target intermediate guidance information includes: determining, based on the target intermediate guidance information, a plurality of initial autonomous driving decisions for the vehicle in a current driving environment; Performing a third quality scoring on each of the initial autonomous driving decisions to obtain a third quality score for each of the initial autonomous driving decisions; According to the third quality score of each of the initial autonomous driving decisions, a target autonomous driving decision is screened from the multiple initial autonomous driving decisions.

7. An automatic driving decision-making device, characterized in that: include: An acquisition module is used to acquire a time-series surround view image of the vehicle during driving; wherein the time-series surround view image includes a plurality of consecutive frames of detection images of the vehicle under a plurality of different camera perspectives at the same time; Determine a model for determining, based on the time-series surround view image, a plurality of initial detection object reasoning descriptions of the vehicle in a current driving environment; A scoring module, used for performing a first quality scoring on each of the initial detection object reasoning descriptions to obtain a first quality score of each of the initial detection object reasoning descriptions; A screening module, configured to screen a target detection object reasoning description from the plurality of initial detection object reasoning descriptions according to a first quality score of each of the initial detection object reasoning descriptions; A decision model is used to determine a target autonomous driving decision for the vehicle in a current driving environment based on the target detection object inference description.

8. A vehicle, characterized in that: include: Surround view image acquisition equipment and autonomous driving decision-making equipment; The surround view image acquisition device is used to collect the time-series surround view images of the vehicle during driving, and send the collected time-series surround view images to the automatic driving decision-making device; The autonomous driving decision device is used to receive the temporal surround view image, and adopt the autonomous driving decision method as described in any one of claims 1 to 6 to determine the target autonomous driving decision of the vehicle in the current driving environment based on the temporal surround view image.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 6 is implemented.

Images