Method and device for presenting auxiliary driving decision-making process

Abstract

The invention provides a method and a device for presenting an auxiliary driving decision process. The method comprises the steps that under the condition that an auxiliary driving function of a vehicle is started, decision-making related information corresponding to an auxiliary driving module of the vehicle is acquired, and the decision-making related information is used for influencing the decision-making process of the auxiliary driving module; and outputting the decision-making related information in a cabin of the vehicle to enable passengers in the cabin to perceive the decision-making process.

Description

Technical Field

[0001] The present invention relates to the field of assisted driving of vehicles, and particularly to a method and device for presenting an assisted driving decision-making process. Background Art

[0002] At present, assisted driving functions are widely applied to various vehicles. With the increasing development of assisted driving technology, the opacity of assisted driving logic is increasing, and the interpretability is gradually decreasing. It is often difficult for users to know the reasons and specific decision-making processes of a certain assisted driving decision, resulting in a decrease in the user's trust and willingness to use the assisted driving function, which urgently needs to be improved.

[0003] In the related art, a reminder is often sent to the driver when the boundary conditions of the system function (such as sensor failure, function failure, etc.) are triggered. However, this method only enables the user to know the cause or result of the failure, and still cannot know the specific decision-making process of the assisted driving. Summary of the Invention

[0004] In view of this, the present invention provides a method and device for presenting an assisted driving decision-making process, in order to solve the deficiencies in the related art by outputting decision-related information of the assisted driving module.

[0005] Specifically, the present invention is implemented by the following technical solutions: According to the first aspect of the present invention, a method for decision-related information is provided, including: When the assisted driving function of the vehicle is turned on, obtaining decision-related information corresponding to the assisted driving module of the vehicle, where the decision-related information is used to affect the decision-making process of the assisted driving module; Outputting the decision-related information in the cockpit of the vehicle, so that the occupants in the cockpit can perceive the decision-making process.

[0006] According to the second aspect of the present invention, a device for presenting an assisted driving decision-making process is provided, including: An information acquisition unit, configured to obtain decision-related information corresponding to the assisted driving module of the vehicle when the assisted driving function of the vehicle is turned on, where the decision-related information is used to affect the decision-making process of the assisted driving module; An information output unit, configured to output the decision-related information in the cockpit of the vehicle, so that the occupants in the cockpit can perceive the decision-making process.

[0007] According to the third aspect of the present invention, a vehicle is provided, including: A processor, a memory for storing processor-executable instructions, and an assisted driving module for providing an assisted driving function; Wherein, the processor runs the executable instructions to implement the method described in the foregoing first aspect.

[0008] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the foregoing first aspect are implemented.

[0009] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: As can be seen from the above embodiments, in the case where the vehicle has enabled the assisted driving function, the decision-related information affecting the decision-making process of the assisted driving module of the vehicle is acquired, and this information is output in the cockpit so that the occupants in the cockpit can perceive the decision-making process of the assisted driving module.

[0010] It can be understood that the decision-related information affects the decision-making process of the assisted driving module, that is, it affects at least one key step in the decision-making link of the module (such as perception, prediction, planning, etc.). Therefore, by outputting the above decision-related information to the occupants, not only the interpretability of the decision-making process performed by the assisted driving module is improved, but also the occupants can accurately and comprehensively learn the presented decision-making process based on this information, which helps to improve the decision-making transparency of the assisted driving module and reduce the understanding difficulty of the occupants, and further helps to improve the trust and willingness to use of the occupants for the assisted driving function. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0012] Figure 1 It is a schematic diagram of the architecture of an assisted driving system shown in an embodiment of the present invention; Figure 2 It is a flowchart of a method for presenting an assisted driving decision-making process shown in an embodiment of the present invention; Figure 3 It is a flowchart of another method for presenting an assisted driving decision-making process shown in an embodiment of the present invention; Figure 4 It is a schematic diagram of the display effect of decision-related information shown in an embodiment of the present invention; Figure 5 It is a schematic diagram of a camera corresponding to an external vehicle image shown in an embodiment of the present invention; Figure 6 It is a schematic diagram of an evaluation method for the current installation state shown in an embodiment of the present invention; Figure 7 It is a schematic diagram of a path scoring process shown in an embodiment of the present invention; Figure 8 It is a schematic structural diagram of a vehicle shown in an embodiment of the present invention; Figure 9 It is a block diagram of a presentation device for an assisted driving decision-making process shown in an embodiment of the present invention. Detailed implementation manners

[0013] Here, exemplary embodiments will be described in detail, and examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0014] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the", and "said" used in the present invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0015] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0016] At present, assisted driving functions are widely applied to various vehicles. With the increasing development of assisted driving technology, the opacity of assisted driving logic is increasing day by day, and the interpretability is gradually decreasing. Users often find it difficult to know the reasons and specific decision-making processes of a certain assisted driving decision, that is, they cannot know the complete decision-making link including key steps such as perception, prediction, and planning and control (i.e., planning and control). This has led to a decrease in users' trust and willingness to use assisted driving functions, which is not conducive to the popularization and application of assisted driving functions and urgently needs to be improved.

[0017] In the related art, a reminder is often sent to the driver only when the boundary conditions of the system function (such as sensor failure, function failure, etc.) are triggered. However, this method only enables the user to know the cause or result of the failure, and still cannot know the specific decision-making process of the assisted driving, that is, cannot know the above complete decision-making link. Therefore, the above technical problem has not been effectively solved yet.

[0018] For this reason, the present application proposes a brand-new presentation scheme for the assisted driving decision-making process, that is, by obtaining and actively presenting to the occupant the decision-related information of the assisted driving module during the assisted driving decision-making process, so that the user can accurately and comprehensively know the specific decision-making process of the assisted driving module when realizing the assisted driving function, thereby improving the interpretability and transparency of the assisted driving, and further enhancing the occupant's trust and willingness to use the assisted driving function. The following describes this scheme in detail with reference to the drawings and embodiments.

[0019] Figure 1 FIG. is a schematic diagram of the hardware architecture of an assisted driving system shown in an embodiment of the present application. As Figure 1 shown, from a hardware perspective, the system may only include the vehicle 11, or may also include the vehicle 11 and the server 13 at the same time. If the assisted driving system only includes the vehicle 11, at this time, the assisted driving module (such as the neural network or the vision language large model VLM described below) may be deployed in the vehicle 11, such as deployed in the domain controller of the vehicle 11. If the assisted driving system includes the vehicle 11 and the server 13 at the same time, at this time, the assisted driving module may be deployed in the server 13, which will not be elaborated.

[0020] In addition to the domain control, the vehicle 11 may also be equipped with various types of sensors for collecting external environment data. The number of any type of sensor may be one or more. The present application embodiment does not limit the number and installation position of the above sensors. Exemplarily, the sensors may include cameras (such as a front-view camera 111, a side-view camera 112, a rear-view camera 113, etc.), a lidar 114, a millimeter-wave radar, an ultrasonic radar, etc., and may also include an irradiance meter (for detecting the light intensity), a rain gauge (for detecting the current rainfall), etc., which will not be elaborated. Of course, the vehicle may also be equipped with at least one in-vehicle sensor (such as an in-vehicle camera, an in-vehicle microphone, an in-vehicle biosensor, an in-vehicle odor sensor, etc.) at a suitable position to realize corresponding vehicle-mounted functions, and the present application embodiment does not limit this.

[0021] In addition, the vehicle can establish a network connection with a remote server 13 through a wireless communication module to perform data interaction with the server 13. For example, if the assisted driving module is deployed in the server 13, the vehicle 11 can send the environmental data collected by the foregoing sensors to the server 13 and receive decision-related information and vehicle control instructions returned by it. Among them, the server 13 can be a physical server including an independent host, or can also be a virtual server or cloud server hosted by a host cluster, etc. In addition, the embodiments of the present application do not limit the number, type, and specific interaction method with the vehicle of the server 13. As for the network 10 for interaction between the vehicle 11 and the server 13, it can be specifically selected to use a corresponding type of wireless network to implement communication based on the communication methods supported by the corresponding devices, and the present application does not limit this.

[0022] In addition, the vehicle (such as the vehicle 11) described in the present application, in terms of its functional form, can be a pickup truck, a sedan, an SUV (Sport Utility Vehicle), a recreational vehicle, a truck, etc.; in terms of its power form, it can be a fuel vehicle or a new energy vehicle (such as a hybrid vehicle, a pure electric vehicle, a hydrogen energy vehicle, a methanol energy vehicle, etc.), and the present invention does not limit the specific form of the vehicle. In addition, the occupant 12 in the cockpit can be the driver sitting in the driver's seat, or can also be at least one passenger sitting in other positions, and the present application also does not limit the number of occupants and their seating positions. Of course, the assisted driving function described in the present application is used to assist the driver in driving the vehicle, which will not be elaborated here.

[0023] The assisted driving module described in the embodiments of the present application can be used to make assisted driving decisions based on relevant data (such as the environmental data collected by the foregoing sensors), and its complete decision-making link can include key steps such as perception, prediction, and planning and control. Exemplarily, in the perception stage, at least one driving-sensitive factor such as lane lines, traffic lights, and pedestrians can be identified; in the prediction stage, high-risk factors such as pedestrian intrusion risk, vehicle collision risk, and road slipperiness risk can be predicted based on the above driving-sensitive factors and their operating parameters; in the planning and control stage, path planning can be performed and the vehicle can be controlled to drive along the planned path (specifically, it can include controlling the vehicle to turn, accelerate, decelerate, etc.). This solution is used to output decision-related information corresponding to the assisted driving module to the occupants of the vehicle, which is used to affect their decision-making process (that is, one or more pieces of information that affect the above at least one key step), so that the occupants can learn about the specific decision-making process of the assisted driving module through this information.

[0024] Figure 2 It is a flowchart of a method for presenting an assisted driving decision-making process shown in the embodiments of the present invention. This method is applied to a vehicle and includes the following steps 202-204.

[0025] Step 202, when the vehicle has enabled the assisted driving function, obtain the decision-related information corresponding to the assisted driving module of the vehicle, where the decision-related information is used to affect the decision-making process of the assisted driving module.

[0026] In one embodiment, the decision-related information can be obtained in real time and continuously output during all time periods after the assisted driving function is enabled, so as to fully present the decision-making process of the assisted driving function. Alternatively, in order to avoid the decision-related information output by this solution from disturbing the occupants, an off-ramp for the decision-related information output function can also be provided to the occupants (i.e., the users of this solution), so that the users can turn off the decision-related information output function provided by this solution through screen operations, gesture operations, or voice control. Obviously, in the latter case, this solution only outputs the decision-related information to the occupants when the output function of the decision-related information is enabled, which will not be elaborated here.

[0027] In one embodiment, the assisted driving module described in this application can be built based on any form of neural network framework and trained in a supervised or unsupervised manner. Exemplarily, the assisted driving module can adopt a Vision-Language Model (VLM), which is a multi-modal artificial intelligence model that combines the capabilities of Computer Vision (CV) and Natural Language Processing (NLP). It can simultaneously understand and process image (or video) and text information, and establish associations between the two to achieve more complex tasks. The assisted driving module implemented using VLM can perform analysis and reasoning from multiple dimensions based on multi-modal data (such as the corresponding types of environmental data collected by various types of sensors), and finally achieve comprehensive decision-making accurately and efficiently.

[0028] In the related art, after the vehicle enables the assisted driving function, it usually outputs the surrounding environment information of the vehicle in real time. For example, a real-time three-dimensional image of the space where the vehicle is located is displayed on the central control screen, and other vehicles, pedestrians, traffic lights, etc., which are sensitive driving factors around the vehicle itself, are displayed in the form of 3D models in real time to improve the perception ability of the occupants (especially the driver) of the vehicle's surrounding environment and facilitate their safe driving.

[0029] In this regard, it should be specifically noted that: the decision-related information output by this solution is not the above-mentioned surrounding environment information, but information with a higher dimension, richer content, and having a certain impact on the decision-making process of the assisted driving module. Exemplarily, such as Figure 4The figure shows the display content of the vehicle screen. In the environment display area 401 marked by the dashed box, 3D models of various objects in the vehicle's surrounding environment and other content are displayed, which is not essentially different from the display method in the related art. More importantly, the information display area 402 marked by the dashed box displays decision-related information such as the real-time image captured by the camera, the status description information of the current safety state, and multiple alternative paths rendered in real time. These information are used to present the decision-making process of the assisted driving module to the occupants.

[0030] Step 204, output the decision-related information in the cockpit of the vehicle so that the occupants in the cockpit can perceive the decision-making process.

[0031] After obtaining the decision-related information corresponding to the assisted driving module, the assisted driving system can output this information to the occupants in the cockpit so that they can accurately know the decision-making process of the assisted driving module presented by this information. Among them, the above decision-related information can be output for any occupant in the cockpit. For example, it can be output only to the driver to assist him in achieving safe driving; or only to the rear-row occupants so that they can timely perceive the vehicle's movement trend and rhythm and avoid motion sickness, etc.; it can also be output to the co-pilot so that when the driver (i.e., the main driver) is not convenient to view this information, the co-pilot can view it and timely remind the driver, etc., which will not be elaborated here.

[0032] In one embodiment, the decision-related information can be output in the cockpit of the vehicle in various ways to meet different scenario requirements. For example, the display device installed in the cockpit can be called to display the decision-related information; at this time, the information can be in the form of text, image or video, etc. The display device can be a screen, such as the instrument screen in front of the driver, the central control screen at the central position of the vehicle's center console, the left-shifted screen facing the rear passengers and set behind the front row on the left, etc., or it can be a HUD (Head-Up Display) device, such as C-HUD (Combiner HUD), W-HUD (Windshield HUD) or AR-UHD (Augmented Reality HUD), etc. The following embodiments will mainly continue to illustrate based on this embodiment.

[0033] For another example, the audio device installed in the cockpit can also be called to play the voice corresponding to the decision-related information. For example, the in-cockpit audio is called to read the decision-related information in text form so that the occupants (especially the driver) can still know the decision-related information when it is not convenient to view the screen or HUD.

[0034] For another example, when a network connection is established between the vehicle and the occupant's mobile terminal, the mobile terminal is called to output the decision-related information. The mobile terminal may be any form of electronic device such as the occupant's mobile phone, tablet device, laptop computer, personal digital assistant (PDA), wearable device (such as smart glasses, smart watches, etc.), virtual reality (VR) device, augmented reality (AR) device, etc. Based on the wired or wireless connection established between the mobile terminal and the vehicle, the vehicle can send the above decision-related information to the device for output, such as displaying information in the form of text, pictures or videos, or playing corresponding voices, etc., which will not be elaborated here.

[0035] In one embodiment, the decision-related information corresponding to the assisted driving module may include at least one of the following: environmental data, lane data, and / or path data to be input into the assisted driving module; driving sensitive factors related to vehicle driving identified (i.e., predicted) by the assisted driving module; the driving path planned (i.e., planned) by the assisted driving module, the vehicle control instructions for the driving path output by the assisted driving module, etc. It can be understood that the richer the output decision-related information is, the more helpful it is for the occupant to accurately and comprehensively understand the decision-making process of the assisted driving module. However, too many types or quantities of information may interfere with the occupant. Therefore, which specific information to output and in what form to output each type of information can be comprehensively considered according to various factors such as the vehicle function positioning, the identity of the occupants in the vehicle, the number of occupants, and the volume of voices in the vehicle. The embodiments of the present application do not limit this.

[0036] In one embodiment, when obtaining the decision-related information, a high-risk object identified by the assisted driving module may be determined, and at least an external image of the vehicle collected by an external camera of the vehicle and containing the high-risk object may be obtained. Determining the high-risk object is actually to obtain the description information of the high-risk object, such as the position, size, shape, color, type, danger level, etc. of the object. When obtaining the external image of the vehicle, the external image containing only the object may be obtained (for example, when the vehicle B overtaking the own vehicle A in the left lane is a high-risk object, only the external image of the vehicle B captured by the left camera may be obtained) and displayed according to the position information of the high-risk object; of course, the external images collected by all cameras may also be obtained and displayed uniformly, which will not be elaborated here.

[0037] Accordingly, when outputting decision-related information, the external vehicle image can be displayed, and the high-risk objects can be marked at corresponding positions on the external vehicle image. It can be understood that the above-mentioned external vehicle image can be captured and obtained in real time. Therefore, the process of displaying the external vehicle image is actually the process of playing the external video captured by the camera in real time. In this way, the occupant can be allowed to view the above-mentioned external vehicle image to know the real-time external environment, enhancing the occupant's perception ability of the external environment. By identifying the high-risk objects, the occupant can more accurately and quickly perceive information such as the risk factors and their positions in the external environment, thus helping to give an early warning (such as turning the steering wheel or stepping on the brakes, etc.) to avoid accidents.

[0038] As Figure 3 shown, after the occupant activates the assisted driving function, the identification information of the assisted driving AI+ can be seen (for the display effect, please refer to Figure 4 the identification 410 shown). Thereafter, on the one hand, according to the position information of the high-risk object, at least one corresponding target camera can be queried from the assisted driving decision-making data related to the assisted driving module (i.e., the "determination of the target camera number"), and then the external vehicle image or video captured by the target camera can be obtained and displayed. On the other hand, according to the position information of the high-risk object, the location of the high-risk object can be mapped to the corresponding position in the external vehicle image or video and marked.

[0039] Among them, the above-mentioned high-risk objects can be marked in various ways. For example, since the high-risk object is already included in the external vehicle image or video, the display parameters of the high-risk object in the image or video can be adjusted to highlight the high-risk object, or a danger sign can also be displayed at the position where the high-risk object is located to highlight the high-risk object; among them, the degree of highlighting of the high-risk object can be positively correlated with its degree of danger, so as to present the danger and urgency of the object.

[0040] As Figure 5 shown, the following are the system behaviors corresponding to several typical assisted driving intentions and the corresponding relationships with the cameras. For example, when the system behavior is that a lateral instruction is about to be responded to, the corresponding cameras (or cameras) are the front wide-angle camera and the side rear cameras on both sides; when the system behavior is that a longitudinal instruction is about to be responded to, the corresponding camera (or camera) is the front wide-angle camera, which will not be elaborated here.

[0041] As Figure 5 shown in the following flow chart, after obtaining the multi-channel video streams captured by multiple cameras, the perception results of the high-risk objects can be obtained, and processing such as ID tracking, target recognition, and coordinate transformation can be performed based on the results. Finally, the accurate position of the high-risk object in the external vehicle image can be determined, and the corresponding high-risk object can be accurately marked at the corresponding position during the process of rendering the video stream.

[0042] Based on the above embodiment, let's assume that there is a vehicle C (a truck) on the right lane of vehicle A (SUV) that is overtaking vehicle A; and there are multiple oncoming vehicles in the left lane, of which vehicle D is the closest. It can be seen that the above vehicles C and D are high-risk objects relative to vehicle A. At this time, the information display area 402 on the right side of the central control screen of vehicle A can display the vehicle A's external video 405 (the videos displayed in the three small windows are respectively taken by the front camera, the left camera and the right camera). Among them, in the vehicle external videos taken by the front camera and the right camera, vehicle C is marked with a dangerous thermal mark 406c; and in the vehicle external video taken by the left camera, vehicle D is marked with a dangerous thermal mark 406d. Moreover, since vehicle C is closer to vehicle A and the degree of danger is relatively higher, the mark 406d is darker in color and larger in area than the mark 406c.

[0043] In addition, avoidance instruction information for the aforementioned high-risk objects may be further displayed on the vehicle exterior image to instruct passengers (such as the driver of the vehicle) to avoid the aforementioned high-risk objects, thereby improving risk response efficiency. Figure 4 As shown, the screen may further display text such as "The vehicle on the right is too close, please keep to the left in the current lane" or play corresponding voice for the aforementioned vehicle C, or display text such as "The oncoming vehicle is driving too fast and has crossed the double yellow line, please flash your lights to remind and keep to the right in the current lane" or play corresponding voice for the aforementioned vehicle D. Of course, in view of the fact that the aforementioned assisted driving function has been turned on, the assisted driving module may also directly output avoidance instructions for the above-mentioned high-risk objects, so as to directly control the vehicle to avoid, further shorten the response path and delay, and further improve the risk response efficiency. In addition, the avoidance instruction information may also be an indicator mark on the current path, such as Figure 4 The offset path displayed in the environment display area 401 or the turning arrow displayed in the information display area are not described in detail.

[0044] In one embodiment, when obtaining decision-related information, the external environment data to be input into the auxiliary driving module (the data is used to describe the external environment in which the vehicle is currently located) can be obtained, and the current safety state of the external environment can be evaluated based on the external environment data. Alternatively, the evaluation result of the current safety state output by the auxiliary driving module after reasoning based on the external environment data can be directly obtained. Accordingly, when displaying decision-related information, the state description information of the current safety state can be output so that the occupant can know whether the external environment is safe (or dangerous) and its safety (or danger) degree based on the information.

[0045] Among them, the external environment data can be used to describe the external environment where the vehicle is currently located from multiple dimensions. For example, the external environment data may include natural environment data (such as the time period to which the current moment belongs, the current weather, the current lighting, etc.), lane data (such as lane width, lane type, traffic direction, etc.), and / or path data (such as traffic flow congestion situation, obstacle distribution on the road, etc.). Based on this, when evaluating the current safety state of the external environment based on the external environment data, weighted calculations can be performed from the multiple dimensions based on the external environment data, and the safety level (such as excellent, good, average, poor, etc.) and / or safety score (specific safety score, such as 0 - 100 points, the larger the score, the safer the external environment) used to represent the current safety state can be determined according to the calculation results. For example, Figure 3 as shown.

[0046] For example, Figure 6 as shown in (a) of , weighted operations can be performed based on external environment data such as natural environment data, lane data, and path data, and the operation results can be mapped to any one of the three safety levels: excellent, good, and average. Among them, Figure 6 as shown in (b) of , the safety level of the current safety state can be determined based on lane data and path data (i.e., "system state: good" shown in the figure, etc.), the safety level of the current safety state can be determined based on natural environment data, etc., which will not be elaborated here. It can be seen that the safety level is positively correlated with the quality of the weather condition, the quality of the lane, the congestion degree of the path, etc. respectively.

[0047] In addition, when the status description information of the current safety state is a status description text, when obtaining the status description text, in the case where the assisted driving module is VLM, the status description text of the current safety state output by the assisted driving module can be received, that is, the text generation ability of VLM is used to highly summarize the current safety state and output the corresponding status description text. Or, the status description text of the current safety state can also be generated based on the evaluation result of the current safety state according to a preset description text template. At this time, the preset description text template is used as a fallback strategy to generate the status description text to ensure that the text generation is not empty.

[0048] Similar to the safety level of the aforementioned current safety state, when obtaining decision-related information, the external environment data to be input into the assisted driving module (this data is used to describe the external environment where the vehicle is currently located) can be obtained, and the current driving scenario of the vehicle can be identified based on the external environment data. Or, the recognition result of the current driving scenario output by the assisted driving module after reasoning based on the external environment data can also be directly obtained. Correspondingly, when displaying decision-related information, the scenario description information of the current driving scenario can be output. For example,Figure 3 As shown, scene description text can be generated according to preset scene corpus (i.e., preset description text templates).

[0049] Among them, when the scene description information of the current driving scene is the scene description text, when obtaining the scene description text, in the case where the assisted driving module is VLM, the scene description text of the current driving scene output by the assisted driving module can be received. Or, based on the recognition result of the current driving scene, the scene description text of the current driving scene can be generated according to the preset description text template, which will not be elaborated here. Exemplarily, the above scene description information can be used to describe static (hanging) obstacles, other vehicles in the traffic flow, VRU (Vulnerable Road User, such as pedestrians, cyclists, and motorcyclists, etc. These road users are not protected by a metal shell like a car and are prone to injury in the event of a traffic accident), etc., which will not be elaborated here.

[0050] As Figure 4 shown, the scene description information of the current driving scene displayed in the information display area 402 is "Backlight scene, drive carefully", and the status description text of the current safety status is "The current driving environment safety level is good: the road ahead is open, the speed on the right is low, and there are few target vehicles".

[0051] In one embodiment, when obtaining decision-related information, multiple alternative paths output by the assisted driving module and the path score of each alternative path (at this time, a comprehensive score is given to each alternative path planned by the assisted driving module itself) can be obtained; or, multiple alternative paths output by the assisted driving module can be obtained, and the path score of each alternative path can be calculated (at this time, a comprehensive score is given to each alternative path planned by the assisted driving system for the assisted driving module). Based on this, when outputting decision-related information, the multiple alternative paths and the path score of each alternative path can be displayed in order from high to low path score. Among them, in view of the fact that the vehicle has enabled the assisted driving function, its current speed may not be zero; or other driving-sensitive factors around it may change, so the alternative paths at different times are often different. In this regard, the assisted driving system can render each alternative path planned by the assisted driving module in real time and display the path scores of each alternative path in turn, so that the occupant can accurately know the relative advantages and disadvantages of each alternative path by viewing the score.

[0052] Among them, when calculating the path score of each alternative path, the path score of each alternative path can be comprehensively calculated from at least two dimensions among driving safety (which can be characterized by the magnitude of the collision probability), compliance with regulations (such as whether running a red light, whether crossing the solid line, etc.), comfort (which can be characterized by the magnitude of lateral, longitudinal, and / or vertical acceleration), driving efficiency (which can be characterized by the expected time taken to reach the destination), and anthropomorphic degree (that is, whether the driving actions conform to human driving habits). For example, for any alternative path, the safety score, regulation score, comfort score, efficiency score, and anthropomorphic score of this path can be calculated, and the path score of this path can be comprehensively calculated (such as weighted) based on these scores.

[0053] As Figure 7 shown, safety verification can be first performed on each alternative path to screen out safe alternative paths without safety risks or with safety risks lower than the threshold; then, comprehensive scoring of each safe alternative path can be performed from the above five dimensions to obtain the corresponding path scores, and then each safe alternative path can be sorted according to the magnitude of the path scores. Finally, each path is rendered and displayed in real time in sequence according to the sorting result, such as the optimal path with a score of 98, alternative safe path 1 with a score of 90, etc.

[0054] As Figure 4 shown, multiple alternative paths 408 and the path scores of each alternative path are sequentially displayed below the information display area 402. In addition, the scores of the alternative path with the highest score in each dimension are also displayed. For example, the safety score, comfort score, and efficiency score of the optimal path with a path score of 98 are displayed in the form of a radar chart, which will not be elaborated here.

[0055] In one embodiment, in view of the fact that vehicle sensors may have errors or even false detections, which may cause the assisted driving module to make decisions that do not conform to the current driving environment. For example, due to the blind area of the radar, the alternative path with the highest score may not be the actual optimal path, and even if driving along this path, a traffic accident may occur. In view of the fact that the observation ability / viewpoint, etc. of the occupants (especially the driver) may be superior to those of the sensors, there may be some high-risk objects that are observed by the occupants but not perceived by the assisted driving module. Therefore, when the assisted driving function is turned on, the occupants (such as the driver) can be allowed to intervene in the decision-making process, such as allowing the user to select a certain path from each alternative path according to their own judgment. For example, in response to the driver of the vehicle selecting any one of the multiple alternative paths, the vehicle can be controlled to drive along the any one of the alternative paths.

[0056] In one embodiment, when obtaining decision-related information, the vehicle control command output by the assisted driving module can be obtained, such as Figure 3The lateral instructions shown (such as lane change instructions, avoidance instructions, detour instructions, left turn / right turn instructions, etc.), longitudinal deceleration instructions (such as following vehicle instructions, following stop instructions, yellow flash deceleration instructions, red light stop instructions, CUTIN intrusion deceleration instructions, etc.) or longitudinal acceleration instructions (such as following stop start instructions, green light start instructions, etc.). In this regard, when outputting decision-related information, the instruction description information of the vehicle control instruction can be output. For example, when the assisted driving module issues a red light stop instruction, a voice such as "Stop at red light" is broadcast; when the assisted driving module issues a green light start instruction, the edge of the control screen flashes three green light spots and / or a voice such as "Start at green light" is played, which will not be elaborated here. In this way, the vehicle control instructions issued by the assisted driving module can be fully informed to the occupants, so that the occupants can know the decision results of the assisted driving module.

[0057] In addition, for the vehicle control instructions output by the assisted driving module, before executing the instruction or outputting the instruction description information of the instruction, the instruction can also be verified (such as keyword verification, etc.) to ensure that the instruction output by the assisted driving module conforms to the current driving environment, so as to ensure "doing and saying correctly", that is, the vehicle control instructions that the assisted driving module can output conform to the current driving environment and the assisted driving system can correctly output the corresponding instruction description information.

[0058] As can be seen from the above embodiments, in the case where the vehicle has the assisted driving function enabled, the decision-related information affecting the decision-making process of the assisted driving module of the vehicle is obtained, and this information is output in the cockpit so that the occupants in the cockpit can perceive the decision-making process of the assisted driving module.

[0059] It can be understood that the decision-related information affects the decision-making process of the assisted driving module, that is, it affects at least one key step (such as perception, prediction, path planning and control, etc.) in the decision-making link of the module. Therefore, by outputting the above decision-related information to the occupants, not only the interpretability of the decision-making process carried out by the assisted driving module is improved, but also the occupants can accurately and comprehensively know the presented decision-making process according to this information, which helps to improve the decision-making transparency of the assisted driving module and reduce the understanding difficulty of the occupants, and further helps to improve the trust and willingness to use of the occupants for the assisted driving function.

[0060] Figure 8 It is a schematic structural diagram of a vehicle shown in an embodiment of the present invention. Please refer to Figure 8, at the hardware level, the vehicle includes a processor 801, a network interface 802, a memory 803, a non-volatile memory 804, and an internal bus 805. Of course, it may also include other hardware required for other services. One or more embodiments of the present invention can be implemented in a software manner. For example, the processor 801 reads the corresponding computer program from the non-volatile memory 804 into the memory 803 and then runs it. Of course, in addition to the software implementation manner, one or more embodiments of the present invention do not exclude other implementation manners, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, but can also be hardware or logical devices.

[0061] Figure 9 Block diagram of a presentation device for an assisted driving decision-making process shown in an embodiment of the present invention. Please refer to Figure 8 , this device can be applied to a vehicle such as Figure 8 shown to implement the technical solution described in the present invention. This device includes: An information acquisition unit 901, configured to acquire decision-related information corresponding to the assisted driving module of the vehicle when the assisted driving function of the vehicle is turned on, where the decision-related information is used to affect the decision-making process of the assisted driving module; An information output unit 902, configured to output the decision-related information in the cockpit of the vehicle so that the occupants in the cockpit can perceive the decision-making process.

[0062] Optionally, the information output unit 902 is specifically used for at least one of the following: Invoking a display device installed in the cockpit to display the decision-related information; Invoking an audio device installed in the cockpit to play the voice corresponding to the decision-related information; When a network connection is established between the vehicle and the mobile terminal of the occupant, invoking the mobile terminal to output the decision-related information.

[0063] Optionally, The information acquisition unit 901 is specifically used for: determining a high-risk object recognized by the assisted driving module, and at least acquiring an external image of the vehicle captured by an external camera of the vehicle and containing the high-risk object; The information output unit 902 is specifically used for: displaying the external image and marking the high-risk object at a corresponding position on the external image.

[0064] Optionally, the information output unit 902 is specifically used for: Adjust the display parameters of the high-risk object to highlight the high-risk object, or display a danger sign at the position where the high-risk object is located to highlight the high-risk object; wherein, the highlighting degree of the high-risk object is positively correlated with its risk degree.

[0065] Optionally, it further includes an avoidance indication unit 903, configured to: Display avoidance indication information for the high-risk object on the external image of the vehicle.

[0066] Optionally, The information acquisition unit 901 is specifically configured to: Acquire external environment data to be input into the assisted driving module, where the data is used to describe the external environment where the vehicle is currently located, and evaluate the current safety state of the external environment and / or identify the current driving scenario of the vehicle based on the external environment data; and / or, Acquire the evaluation result for the current safety state and / or the recognition result for the current driving scenario output by the assisted driving module; The information output unit 902 is specifically configured to: output the status description information of the current safety state and / or the scenario description information of the current driving scenario.

[0067] Optionally, the external environment data is used to describe the external environment where the vehicle is currently located from multiple dimensions, and the information acquisition unit 901 is specifically configured to: Perform weighted calculation from the multiple dimensions based on the external environment data, and determine the safety level and / or safety score used to characterize the current safety state according to the calculation result.

[0068] Optionally, the status description information of the current safety state is status description text, and the information acquisition unit 901 is specifically configured to: The status description information of the current safety state is status description text. Obtaining the status description text includes: when the assisted driving module is a visual language model VLM, receiving the status description text of the current safety state output by the assisted driving module; or, generating the status description text of the current safety state according to a preset description text template based on the evaluation result of the current safety state; and / or, The scenario description information of the current driving scenario is scenario description text. Obtaining the scenario description text includes: when the assisted driving module is a visual language model VLM, receiving the scenario description text of the current driving scenario output by the assisted driving module; or, generating the scenario description text of the current driving scenario according to a preset description text template based on the recognition result of the current driving scenario.

[0069] Optionally, the information acquisition unit 901 is specifically configured to: acquire multiple alternative paths output by the assisted driving module and the path scores of each alternative path; or, acquire multiple alternative paths output by the assisted driving module and calculate the path scores of each alternative path; the information output unit 902 is specifically configured to: display the multiple alternative paths and the path scores of each alternative path in descending order of the path scores.

[0070] Optionally, the information acquisition unit 901 is specifically configured to: comprehensively calculate the path scores of each alternative path from at least two dimensions of driving safety, compliance with regulations, comfort, driving efficiency, and anthropomorphic degree.

[0071] Optionally, it further includes a vehicle control unit 904, configured to: in response to the driver of the vehicle selecting any one of the multiple alternative paths, control the vehicle to travel along the any one of the alternative paths.

[0072] Optionally, the information acquisition unit 901 is specifically configured to: acquire a vehicle control instruction output by the assisted driving module; the information output unit 902 is specifically configured to: output instruction description information of the vehicle control instruction.

[0073] Correspondingly, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the presentation method of the assisted driving decision-making process described in any one of the above embodiments.

[0074] Correspondingly, the present specification further provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the steps of the presentation method of the assisted driving decision-making process described in any one of the above embodiments.

[0075] The systems, devices, modules, or units illustrated in the above embodiments may be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0076] In a typical configuration, a computer includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0077] The memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0078] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

Claims

1. A method for presenting an assisted driving decision-making process, characterized in that Including: When the vehicle has enabled the assisted driving function, obtain decision-related information corresponding to the assisted driving module of the vehicle, where the decision-related information is used to affect the decision-making process of the assisted driving module; Output the decision-related information inside the cockpit of the vehicle so that the occupants inside the cockpit can perceive the decision-making process.

2. The method according to claim 1, wherein The outputting the decision-related information inside the cockpit of the vehicle includes at least one of the following: Invoke the display device installed inside the cockpit to display the decision-related information; Invoke the audio device installed inside the cockpit to play the voice corresponding to the decision-related information; When there is a network connection established between the vehicle and the occupant's mobile terminal, invoke the mobile terminal to output the decision-related information.

3. The method according to claim 1, wherein The obtaining the decision-related information corresponding to the assisted driving module of the vehicle includes: determining a high-risk object identified by the assisted driving module, and at least obtaining an external image of the vehicle captured by an external camera of the vehicle and containing the high-risk object; The outputting the decision-related information includes: displaying the external image and marking the high-risk object at a corresponding position on the external image.

4. The method according to claim 3, wherein The marking the high-risk object at a corresponding position on the external image includes: Adjusting the display parameters of the high-risk object to highlight the high-risk object, or displaying a danger sign at the position where the high-risk object is located to highlight the high-risk object; wherein, the highlighting degree of the high-risk object is positively correlated with its danger degree.

5. The method according to claim 3 or 4, characterized in that, It further includes: Displaying avoidance instruction information for the high-risk object on the external image.

6. The method according to claim 1, wherein The obtaining the decision-related information corresponding to the assisted driving module of the vehicle includes: Obtaining external environment data to be input into the assisted driving module, where the data is used to describe the external environment where the vehicle is currently located, and evaluating the current safety state of the external environment and / or identifying the current driving scenario of the vehicle based on the external environment data; and / or, Obtaining the evaluation result for the current safety state and / or the identification result for the current driving scenario output by the assisted driving module; The outputting the decision-related information includes: outputting the state description information of the current safety state and / or the scenario description information of the current driving scenario.

7. The method according to claim 6, characterized in that, The external environment data is used to describe the external environment where the vehicle is currently located from multiple dimensions. The evaluating the current safety state of the external environment based on the external environment data includes: Performing weighted calculation from the multiple dimensions based on the external environment data, and determining a safety level and / or a safety score used to characterize the current safety state according to the calculation result.

8. The method according to claim 6, characterized in that, The state description information of the current safety state is a state description text. Obtaining the state description text includes: The status description information of the current safety status is status description text. Obtaining the status description text includes: when the assisted driving module is a vision language model VLM, receiving the status description text of the current safety status output by the assisted driving module; or, based on the evaluation result of the current safety status, generating the status description text of the current safety status according to a preset description text template; and / or, The scene description information of the current driving scene is scene description text. Obtaining the scene description text includes: when the assisted driving module is a vision language model VLM, receiving the scene description text of the current driving scene output by the assisted driving module; or, based on the recognition result of the current driving scene, generating the scene description text of the current driving scene according to a preset description text template.

9. The method according to claim 1, wherein Obtaining the decision-related information corresponding to the assisted driving module of the vehicle includes: obtaining multiple alternative paths output by the assisted driving module and the path score of each alternative path; or, obtaining multiple alternative paths output by the assisted driving module and calculating the path score of each alternative path. Outputting the decision-related information includes: sequentially displaying the multiple alternative paths and the path score of each alternative path in descending order of the path score.

10. The method according to claim 9, characterized in that, Calculating the path score of each alternative path includes: Comprehensively calculating the path score of each alternative path from at least two dimensions of driving safety, compliance with regulations, comfort, driving efficiency, and anthropomorphic degree.

11. The method according to claim 9, characterized in that, It further includes: In response to the driver of the vehicle selecting any one of the multiple alternative paths, controlling the vehicle to travel according to the any one of the alternative paths.

12. The method according to claim 1, wherein Obtaining the decision-related information corresponding to the assisted driving module of the vehicle includes: obtaining a vehicle control instruction output by the assisted driving module. Outputting the decision-related information includes: outputting the instruction description information of the vehicle control instruction.

13. A presentation device for an assisted driving decision-making process, characterized in that, It includes: An information acquisition unit, configured to obtain the decision-related information corresponding to the assisted driving module of the vehicle when the assisted driving function of the vehicle is enabled, where the decision-related information is used to affect the decision-making process of the assisted driving module. An information output unit, configured to output the decision-related information in the cockpit of the vehicle so that the occupants in the cockpit can perceive the decision-making process.

14. A vehicle, comprising: A processor, a memory for storing processor-executable instructions, and an assisted driving module for providing an assisted driving function; Wherein, the processor realizes the method according to any one of claims 1-12 by running the executable instructions.

15. A computer program product, comprising a computer program and / or instructions, characterized in that, The computer program and / or instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-12.

Images