An ai voice robot for a commercial vehicle and a control system thereof

Abstract

The application provides an AI voice robot for a commercial vehicle and a control system thereof, and relates to the technical field of driving assistance, wherein the robot comprises: a driver driving state acquisition module, which is used for acquiring the driving state of a driver of the commercial vehicle; and a multi-modal driving assistance interaction module, which is used for taking the driving state of the driver as an input of a cloud pre-trained AI large model, and performing multi-modal driving assistance interaction with the driver based on the output of the AI large model. In the AI voice robot for the commercial vehicle, the driver driving state acquisition module and the multi-modal driving assistance interaction module are integrated, and the AI large model is used to perform multi-modal driving assistance interaction with the driver according to the acquired driving state of the driver, so that the driving safety of the commercial vehicle is improved, the occurrence of traffic accidents is reduced, and the interactive experience is improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of driving assistance, in particular to an AI voice robot for a commercial vehicle and a control system thereof. BACKGROUND

[0002] Commercial vehicle drivers face many challenges in their daily work, such as fatigue caused by long driving, safety hazards under complex road conditions, etc., which pose a threat to driving safety. Traditional driving assistance systems usually rely on a single data source, such as GPS navigation or vehicle speed sensors, which are difficult to fully capture the real-time state of the driver, such as fatigue level or distraction. In addition, the interaction mode of existing systems is relatively simple, limited to basic voice prompts or visual alerts, lacking personalized and intelligent experience, and unable to effectively meet the needs of drivers in complex driving environments.

[0003] Therefore, there is an urgent need for a more intelligent and comprehensive driving assistance technology to improve the driving safety of commercial vehicles and optimize the interactive experience of drivers. SUMMARY

[0004] One of the purposes of the present application is to provide an AI voice robot for a commercial vehicle to solve the problems in the background art.

[0005] The AI voice robot for a commercial vehicle provided by the embodiments of the present application comprises:

[0006] A driver driving state acquisition module is configured to acquire the driving state of the driver of the commercial vehicle, wherein the driving state at least includes: the driving behavior image of the driver, the real-time position of the vehicle, the input voice instruction of the driver, and the operating condition of the vehicle.

[0007] A multi-modal driving assistance interaction module is configured to take the driving state of the driver as the input of the AI large model pre-trained in the cloud, and perform multi-modal driving assistance interaction with the driver based on the output of the AI large model.

[0008] Optionally, the driver driving state acquisition module comprises:

[0009] A camera unit is configured to acquire image information of the driver inside the commercial vehicle.

[0010] A positioning unit is configured to acquire real-time positioning information of the commercial vehicle.

[0011] A voice input unit is configured to acquire input voice information of the driver inside the commercial vehicle.

[0012] An in-vehicle communication gateway unit is configured to acquire operating information of in-vehicle devices of the commercial vehicle.

[0013] Optionally, the multi-modal driving assistance interaction module comprises:

[0014] a cloud communication unit configured to establish a remote communication connection with a cloud through 4G / 5G communication;

[0015] a display unit configured to interact with the driver through a picture display;

[0016] a voice output unit configured to interact with the driver through voice output.

[0017] Optionally, the pre-training step of the AI large model comprises:

[0018] training a pre-selected basic large language model by using a large number of training samples including at least historical dialogue records of commercial vehicle drivers, driving risk records, path optimization records, business collaborative scheduling records, shipment planning records, emotion management records, and point of interest recommendation records, to obtain the AI large model.

[0019] Optionally, the multi-modal driving assistance interaction module interacts with the driver in a multi-modal driving assistance manner based on the output of the AI large model, comprising:

[0020] interacting with the driver based on the output of the AI large model, including at least safety driving reminders, path dynamic optimization, business collaborative scheduling, shipment planning, emotion management, and point of interest recommendation.

[0021] Optionally, the multi-modal driving assistance interaction module is further configured to, when performing road condition risk early warning driving assistance to the driver, optimize early warning information of the road condition risk so that the driver can clearly perceive the road condition risk, and use the optimized early warning information to perform road condition early warning to the driver.

[0022] The multi-modal driving assistance interaction module optimizes the early warning information of the road condition risk so that the driver can clearly perceive the road condition risk, comprising:

[0023] analyzing a first average visibility of a risk area of a road condition risk in each of the current driving field of view of the driver and a plurality of future driving field of views within a time not exceeding a pre-warning time limit predicted on a driving map;

[0024] when the first average visibility exceeds a first visibility threshold, generating a first view on the driving map for viewing a first relative position relationship between the current driving position of the driver and the risk area from a bird's eye view, and using information for the driver to view the first view as the optimized early warning information; otherwise, analyzing a second average visibility of each of the current driving field of view and the future driving field of view on the driving map;

[0025] sequentially traversing the markers corresponding to the plurality of second average visibilities that are higher than a second visibility threshold;

[0026] In each traversal, the second relative position relationship between the current driving position of the driver and the traversed marker, the third relative position relationship between the traversed marker and the risk area, and the distinctness information of the traversed marker jointly reflect the ability value of the driver to clearly perceive the road risk through the traversed marker on the driving map;

[0027] After the sequential traversal is completed, a second view of the second relative position relationship of the marker corresponding to the maximum ability value is generated on the driving map, a third view of the second relative position relationship and the third relative position relationship of the marker corresponding to the maximum ability value is generated on the driving map, and information for the driver to sequentially view the second view and the third view is provided as the optimized warning information.

[0028] Optionally, the markers include at least other vehicles, roadside facilities, buildings, and traffic warning devices.

[0029] Optionally, the ability value acquisition step includes:

[0030] Based on a preset quantification system, the second relative position relationship, the third relative position relationship, and the distinctness information are quantified to reflect the ability of the driver to clearly perceive the road risk through the traversed marker, and first, second, and third quantification values are sequentially obtained.

[0031] Based on weights set according to the influence of the first, second, and third quantification values on the ability value, the first, second, and third quantification values are weighted and calculated, and the weighted calculation result is used as the ability value.

[0032] Optionally, the road condition warning for the driver using the optimized warning information includes:

[0033] Suggestion information for planning safety measures to be taken by the driver to cope with the road risk;

[0034] A time period in which the driver is least distracted when viewing the warning information and the suggestion information in a time period before the warning time limit is predicted.

[0035] In the time period in which the driver is least distracted, the warning information and the suggestion information are output to the driver.

[0036] The control system of the AI voice robot for the commercial vehicle provided by the embodiment of the present application comprises:

[0037] A control center is configured to control the driver driving state acquisition module, the multi-modal driving assistance interaction module, and the road condition warning optimization module.

[0038] The present application has the following beneficial effects:

[0039] The application is used in the AI voice robot of the commercial vehicle, integrates the driver driving state acquisition module and the multi-modal driving assistance interaction module, utilizes the AI large model to carry out the multi-modal driving assistance interaction with the driver according to the acquired driver driving state, improves the driving safety of the commercial vehicle, reduces the occurrence of traffic accidents, and improves the interaction experience.

[0040] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. The objects and other advantages of the present application can be achieved and obtained by the structure particularly pointed out in the written description and the accompanying drawings.

[0041] The technical solutions of the present application will be further described in detail below by means of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0043] Figure 1 The module structure schematic diagram of the AI voice robot for the commercial vehicle in the embodiment of the present application;

[0044] Figure 2 The hardware structure schematic diagram of the AI voice robot for the commercial vehicle in the embodiment of the present application;

[0045] Figure 3 The working mode schematic diagram of the AI voice robot for the commercial vehicle in the embodiment of the present application. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and do not limit the present application.

[0047] The research and development idea of the present application is based on AI technology, and the core is to use a pre-trained AI large model to realize comprehensive monitoring and intelligent analysis of the driving state of the driver. The system acquires multi-source data through the driver driving state acquisition module, including the driver's driving behavior image, vehicle real-time position, input voice instruction, and vehicle operating condition, etc. The AI large model judges the driver's state and potential needs in real time based on these data, and generates personalized driving assistance suggestions. For example, when signs of fatigue driving of the driver are identified, the system can remind the driver to go to the nearby rest area for rest in the form of voice through the multi-modal driving assistance interaction module, while combining image or text prompts to enhance the interaction effect. In addition, the introduction of the road condition warning optimization module enables the system to analyze potential risks in advance and issue warnings to help the driver make safer decisions. The design goal of the present application is to improve the driving safety of commercial vehicles and reduce the incidence of traffic accidents by integrating multiple modules and combining AI technology, while providing a natural and efficient interactive experience for the driver.

[0048] Embodiment 1:

[0049] The embodiment of the present application provides an AI voice robot for a commercial vehicle, as shown in the accompanying drawings, which comprises: Figure 1

[0050] A driver driving state acquisition module 1 is used to acquire the driving state of the driver of the commercial vehicle; the driving state at least includes: the driving behavior image of the driver, the real-time position of the vehicle, the input voice instruction of the driver, and the operating condition of the vehicle;

[0051] A multi-modal driving assistance interaction module 2 is used to take the driving state of the driver as the input of the cloud pre-trained AI large model, and perform multi-modal driving assistance interaction with the driver based on the output of the AI large model.

[0052] In the embodiment of the present application, the acquired driving state of the driver of the commercial vehicle at least includes: the driving behavior image of the driver, the real-time position of the vehicle, the input voice instruction of the driver, and the operating condition of the vehicle, etc. There is an AI large model pre-trained in the cloud, which can determine the content of driving assistance needed for the driver at this time according to the driving state of the driver, and output. Then, based on the output of the AI large model, multi-modal driving assistance interaction is performed with the driver, for example: the AI large model identifies the behavior, expression, etc. representing fatigue driving of the driver through the driving behavior image, and then reminds the driver to park and rest at the nearby rest area in the form of voice.

[0053] ​This invention relates to an AI voice robot for commercial vehicles, which integrates a driver driving status acquisition module and a multimodal driving assistance interaction module. By utilizing a large AI model to collect the driver's driving status, it conducts multimodal driving assistance interaction with the driver, thereby improving the driving safety of commercial vehicles, reducing traffic accidents, and enhancing the interactive experience.

[0054] Example 2:

[0055] In this embodiment of the invention, the driver driving status acquisition module includes:

[0056] The camera unit is used to collect image information of the driver inside the commercial vehicle;

[0057] The positioning unit is used to collect real-time positioning information of commercial vehicles;

[0058] A voice input unit is used to collect the voice input information of the driver inside the commercial vehicle;

[0059] The in-vehicle communication gateway unit is used to collect operating information of on-board equipment in commercial vehicles.

[0060] like Figure 2 As shown in the embodiment of the present invention, the camera unit can be a DSM camera, which is used to collect image information of the driver inside the commercial vehicle. Based on the image information, the driver's behavior, expression, etc. can be analyzed to determine whether the driver is experiencing dangerous driving conditions such as fatigued driving or distracted driving.

[0061] The positioning unit can be a satellite positioning device, which is used to collect real-time positioning information of commercial vehicles, enabling continuous tracking of the vehicle's location.

[0062] The voice input unit can be a multi-MIC device, which is used to collect the input voice information of the driver in the commercial vehicle. The multi-MIC setting can avoid interference from the ambient noise in the vehicle. Based on the input voice information, the voice commands input by the driver can be recognized.

[0063] The in-vehicle communication gateway unit can be an in-vehicle communication gateway device, such as... Figure 3 As shown, this enables AI voice robots to interface with in-vehicle smart screens, in-vehicle audio systems, air conditioning systems, in-vehicle terminals, and TBOX via the vehicle's bus, collecting operational information from these in-vehicle devices in commercial vehicles.

[0064] The image information, real-time positioning information, input voice information, and operation information collected by each unit together constitute the driver's driving status.

[0065] Example 3:

[0066] In this embodiment of the invention, the multimodal driving assistance interaction module includes:

[0067] a cloud communication unit configured to establish a remote communication connection with a cloud through 4G / 5G communication;

[0068] a display unit configured to interact with the driver through picture display;

[0069] a voice output unit configured to interact with the driver through voice output.

[0070] In the embodiments of the present application, the cloud communication unit can be a 4G / 5G communication device, such as Figure 3 as shown, which enables the AI voice robot to establish a remote communication connection with the cloud platform, i.e., the cloud, through 4G / 5G communication, so that the driver's driving state can be sent to the cloud as the input of the pre-trained AI large model therein, and the output of the AI large model is received. In addition, the AI voice robot can also be connected with the smart phone through Bluetooth.

[0071] As shown in Figure 2 , the display unit can be a display screen, which can interact with the driver through picture display. Specifically, a virtual character can be set in the picture to further improve the interactive experience. The voice output unit can be a sound box, which can interact with the driver through voice output. For example, the voice can remind the driver to drive safely.

[0072] Embodiment 4:

[0073] In the embodiments of the present application, the pre-training step of the AI large model includes:

[0074] A large number of at least commercial vehicle driver historical dialogue records, driving risk records, path optimization records, business collaborative scheduling records, shipment planning records, emotion management records, and interest point recommendation records are taken as training samples to train the preselected basic large language model to obtain the AI large model.

[0075] The commercial vehicle driver historical dialogue record is the dialogue record between the driver and the system or the dispatching platform, which helps the model understand the driver's needs and common language; the driving risk record includes risk events such as emergency braking and collision warning, which helps the model to predict and remind potential dangers; the path optimization record is the data of optimized route planning, which helps the model to learn how to optimize the path; the business collaborative scheduling record is the record of collaborative scheduling of the driver's transportation business, which helps the model to learn how to collaboratively schedule the driver's transportation tasks; the shipment planning record is the record of the driver's shipment planning, which helps the model to understand the driver's shipment planning habits, preferences, etc.; the emotion management record is the driver's emotion fluctuation record, which helps the model to learn which recognized emotions need to be intervened; the interest point recommendation record includes the driver's preferred places or activities, which helps the model to learn which places need to be recommended to the driver.

[0076] The above information is used as a training sample to train a pre-selected basic large language model (such as DeepSeek, etc.) to obtain an AI large model. It is capable of outputting content for interaction with the driver, including at least safety driving reminders, path dynamic optimization, business collaborative scheduling, shipment planning, emotion management, and interest point recommendations.

[0077] Embodiment 5:

[0078] In the embodiments of the present application, the multi-modal driving assistance interaction module interacts with the driver based on the output of the AI large model for multi-modal driving assistance interaction, including:

[0079] Based on the output of the AI large model, the driver is interacted with at least including safety driving reminders, path dynamic optimization, business collaborative scheduling, shipment planning, emotion management, and interest point recommendations.

[0080] Based on embodiment 4, the output of the AI large model can be used to interact with the driver at least including safety driving reminders, path dynamic optimization, business collaborative scheduling, shipment planning, emotion management, and interest point recommendations. It greatly improves the ability of multi-modal driving assistance for commercial vehicle drivers, improves the assistance effect, and improves the interaction experience.

[0081] As shown in Figure 3 The AI large model enables the cloud platform to act as a safety monitoring system to monitor whether the driver has a safety driving risk; also as a shipment management system to assist in shipment planning; also as a fleet dispatching system to perform business collaborative scheduling; also as a voice large model to realize basic conversation with the driver; also as a map system to provide interest point recommendation assistance, etc.

[0082] Embodiment 6:

[0083] Due to the large size and heavy weight of commercial vehicles, the cost of avoiding risks is high, and the driver's most concern is the front road condition warning information. However, when providing road condition warning for driving assistance, the existing technology usually only provides simple text or voice broadcast to inform the risk type and area. When the driver receives this information, he cannot clearly perceive the road condition risk at first, which affects his emergency response ability and decision-making efficiency.

[0084] In reality, the driver of a commercial vehicle knows that the cost of avoiding risks when driving a vehicle is high, and is relatively sensitive to the warning of the road conditions ahead. After receiving the warning, he can only perceive the road condition risk through his eyesight to achieve a clear perception. If the driver cannot perceive the road condition risk through his eyesight all the time, he will feel uneasy about not clearly perceiving the road condition risk and will always be on alert, which may even cause distraction.

[0085] For example, after receiving a warning, a driver will try their best to see the corresponding risk area. If they can see it, the driver will clearly understand the road risk, then judge its position relative to themselves, and take proactive measures. If they cannot see it, they will feel uneasy and continue to focus on searching.

[0086] However, road risks may not be visible to drivers. But to avoid untimely warnings, and given the time limit of warnings, how to enable drivers to perceive road risks visually before the warning time limit in the above special circumstances has become an urgent problem to be solved.

[0087] Therefore, in this embodiment of the invention, the multimodal driving assistance interaction module is also used to optimize the road condition risk warning information so that the driver can clearly perceive the road condition risk when providing driving assistance with road condition risk warning to the driver, and to use the optimized warning information to provide road condition warning to the driver.

[0088] The multimodal driving assistance interaction module optimizes the road condition risk warning information to enable the driver to clearly perceive the road condition risks, including:

[0089] The first average visibility of the risk areas of the road condition risk in each of the driver's current driving view and the predicted future driving view within the time limit not exceeding the warning time limit is analyzed on the driving map.

[0090] When the first average visibility exceeds the first visibility threshold, a first view is generated on the driving map to show the first relative positional relationship between the driver's current driving position and the risk area from a bird's-eye view, and the information of the first view for the driver to view is used as the optimized warning information; otherwise, the second average visibility of each landmark visible in the current driving field of vision and the future driving field of vision is analyzed on the driving map.

[0091] Iterate through the landmarks corresponding to each of the multiple second average visibility values ​​that are higher than the second visibility threshold;

[0092] Each time the map is traversed, the driver's ability to clearly perceive road condition risks is reflected by analyzing the second relative positional relationship between the driver's current driving position and the traversed landmarks, the third relative positional relationship between the traversed landmarks and the risk areas, and the salience information of the traversed landmarks.

[0093] After the sequential traversal ends, a second view of the second relative position relationship of the maximum capacity value corresponding marker is generated on the driving map, a third view of the second relative position relationship and the third relative position relationship of the maximum capacity value corresponding marker is generated on the driving map, and information for the driver to sequentially view the second view and the third view is taken as the optimized early warning information.

[0094] In the embodiment of the present application, the road condition risk refers to obstacles, accidents, traffic jams, construction and other special situations that need to be responded in advance in front of the driving road of the commercial vehicle. The early warning information of the road condition risk reported by the passing vehicle, the traffic management department and the like is optimized when received so that the driver can clearly perceive the road condition risk. When the driver is warned of the road condition by using the optimized early warning information, the driver can clearly perceive the road condition risk.

[0095] The driving map is an environmental perception map of the commercial vehicle, which contains the driving position, driving direction and future driving trajectory of the commercial vehicle, and also contains the surrounding environment information perceived by the camera, radar and the like on the commercial vehicle, such as the position of other vehicles, the driving direction and historical driving trajectory of other vehicles, roadside facilities, buildings and traffic warning devices and the like.

[0096] The early warning time limit refers to the latest time limit for the driver to receive the early warning of the road condition risk, which can be set as the remaining driving time of the commercial vehicle to reach a certain distance (such as 150 meters) from the risk area (road area where the road condition risk occurs) of the road condition risk.

[0097] The current driving field of view refers to the outward field of view range of the driver's cabin of the commercial vehicle at the current driving position (the field of view range that the driver can see outside through the window, windshield and the like in the driver's cabin), which can be pre-set as the standard driver's cabin outward field of view range when the commercial vehicle drives in different directions, and then the corresponding driver's cabin outward field of view range is determined according to the current driving direction of the commercial vehicle on the driving map, and the range blocked by the field of view is removed to obtain the current field of view range.

[0098] The future driving field of view can be predicted according to the average speed and future driving trajectory of the commercial vehicle within a time not exceeding the early warning time limit in the future, and the average speed and future driving trajectory of other vehicles in the surrounding area within a time not exceeding the early warning time limit in the future can also be predicted. According to the pre-set standard driver's cabin outward field of view range when the commercial vehicle drives in different directions, the driver's cabin outward field of view range of different trajectory points on the future driving trajectory of the commercial vehicle is determined, and the range blocked by the field of view is removed. According to the future driving trajectory of the surrounding other vehicles, the occurrence in the remaining field of view range is determined, and the future driving field of view is determined comprehensively.

[0099] When analyzing the first average visibility, the visibility of the risk area in the current driving field of view and each future driving field of view is calculated (the proportion of the area where the risk area appears in the field of view), and then the average value is calculated to obtain the first average visibility.

[0100] When the first average visibility exceeds the first visibility threshold, it means that the driver can continuously view the risk area in the field of view at present, and a first view for viewing the first relative position relationship between the current driving position of the driver and the risk area from a bird's eye view is generated as the pre-warning information to pre-warn the driver. When the driver views the first view, he or she can quickly know the relative position of the risk area and find it for further viewing, so as to clearly perceive the road risk.

[0101] When analyzing the second average visibility, the visibility of the marker in the current driving field of view and each future driving field of view is calculated (the proportion of the area where the marker appears in the field of view), and then the average value is calculated to obtain the second average visibility.

[0102] When the first average visibility does not exceed the first visibility threshold, it means that the driver cannot continuously view the risk area in the field of view at present, and a plurality of markers corresponding to the second average visibility higher than the second visibility threshold are sequentially traversed (the second average visibility higher than the second visibility threshold means that the driver can continuously view the corresponding marker in the field of view at present, so the marker is suitable for indirectly prompting the risk area), and the ability value representing the degree of the driver's ability to clearly perceive the road risk through the traversed marker is analyzed each time. After the sequential traversal is completed, a second view for viewing the second relative position relationship of the marker corresponding to the maximum ability value from a bird's eye view, a third view for viewing the second relative position relationship and the third relative position relationship of the marker corresponding to the maximum ability value are generated, and the information for the driver to sequentially view the second view and the third view is used as the optimized pre-warning information. When the driver receives the pre-warning information, he or she can first view the second view to quickly find the corresponding marker, and then view the third view to further know the relationship between the risk area and the marker, so as to indirectly clearly perceive the road risk through the field of view.

[0103] The significant information refers to the significant features of the marker, such as the color and size of the vehicle body.

[0104] The embodiment of the present application optimizes the pre-warning information of the road risk received by the driver to enable the driver to clearly perceive the road risk, so that the driver can immediately clearly perceive the road risk when receiving the optimized information, and improve the ability to respond to the perceived road risk and the efficiency of decision-making.

[0105] Overall, the driver can perceive the road risk through the eye field of view before the warning time limit, avoiding the driver from being unable to perceive the road risk through the eye field of view all the time, causing the driver to have an uneasy feeling of unclear perception of the road risk, and avoiding the driver from being constantly on alert, even causing distraction. The applicability of the warning information optimization is greatly improved.

[0106] Specifically, by introducing the driving map, predicting the future driving field of view, accurately analyzing the first average visibility, determining whether the driver can continuously view the risk area in the field of view, and optimizing the warning information based on the two results, the comprehensiveness of the system is improved.

[0107] Secondly, the view is generated in the way of bird's eye relative position relationship, ensuring that the driver can quickly find the risk area or the best marker after viewing, and improving the warning effect.

[0108] In addition, the second relative position relationship between the current driving position of the driver and the traversed marker, the third relative position relationship between the traversed marker and the risk area, and the saliency information of the traversed marker collectively reflect the ability value of the driver to clearly perceive the road risk through the traversed marker, and the marker with the maximum ability value is selected for indirect prompting, greatly improving the effect of the warning.

[0109] Embodiment 7:

[0110] In the embodiment of the application, the markers at least include other driving vehicles, roadside facilities, buildings, and traffic warning devices.

[0111] In the embodiment of the application, the markers can be other driving vehicles, roadside facilities, buildings, and traffic warning devices, and the facilities on the road that can be used for indirect warning are comprehensively utilized to improve the road condition warning capability of the system.

[0112] Embodiment 8:

[0113] In the embodiment of the application, the ability value acquisition step includes:

[0114] Based on the preset quantification system, the abilities of the driver to clearly perceive the road risk through the traversed markers are respectively quantified according to the second relative position relationship, the third relative position relationship, and the saliency information, and first, second, and third quantification values are obtained in sequence.

[0115] Based on the weights set according to the influence degree of the first, second, and third quantification values on the ability value, the first, second, and third quantification values are weighted and calculated, and the weighted calculation result is taken as the ability value.

[0116] In the embodiment of the present application, a quantification system is pre-set, in which a first quantification value is pre-set according to the size of the ability of the driver to clearly perceive the road risk through the traversed markers reflected by different second relative position relationships; a second quantification value is pre-set according to the size of the ability of the driver to clearly perceive the road risk through the traversed markers reflected by different third relative position relationships; and a third quantification value is pre-set according to the size of the ability of the driver to clearly perceive the road risk through the traversed markers reflected by different saliency information. Specifically, for example: the second relative position relationship represents that the farther the distance between the current driving position and the traversed marker, the closer the traversed marker to the risk area, which can make the driver perceive the risk area in which invisible area based on the vision, and the higher the corresponding first quantification value. For another example: the third relative position relationship represents that the closer the distance between the traversed marker and the risk area, the closer the traversed marker to the risk area, which can make the driver perceive the risk area in which invisible area based on the vision, and the higher the corresponding second quantification value. For another example: the saliency information is that the color of the vehicle body is a special color such as red, yellow, etc., which can make the user capture through the eyes, and the higher the corresponding third quantification value.

[0117] The weights are set according to the influence degree of the influence of the first quantification value, the second quantification value and the third quantification value on the ability value, the first quantification value, the second quantification value and the third quantification value are weighted calculated based on the weights corresponding to the three quantification values, and the weighted calculation result is taken as the ability value. The accuracy, comprehensiveness and efficiency of the ability value of the driver to clearly perceive the road risk through the traversed markers reflected by the second relative position relationship, the third relative position relationship and the saliency information are greatly improved, the applicability of determining the best marker is improved, and the effect of road condition warning optimization is improved.

[0118] Embodiment 9:

[0119] In the embodiment of the present application, the road condition warning is performed on the driver by using the optimized warning information, which comprises:

[0120] Planning suggestion information of safety measures to be taken by the driver to cope with the road risk;

[0121] Predicting a time period in which the driver is least distracted when viewing the warning information and the suggestion information in the future within the warning time limit;

[0122] Outputting the warning information and the suggestion information to the driver in the time period in which the driver is least distracted.

[0123] In the embodiment of the present application, first, the safety measures that the driver needs to take to cope with the road condition risk are planned, and then the suggestion information is generated based on the safety measures. For example, if the road condition risk is that there is an unknown obstacle in the current lane, the suggestion information is to avoid the obstacle. The time period in which the driver is least distracted when viewing the warning information and the suggestion information in the future within the warning time limit is predicted. When predicting, according to the complexity of the road condition that the driver needs to cope with in the future within the warning time limit, the corresponding time period in which the complexity of the road condition that needs to be coped with in a period of time is the smallest is taken as the time period in which the distraction is the smallest. In the time period in which the distraction is the smallest, the warning information and the suggestion information are output to the driver, so as to ensure that the driver is effectively warned before the warning time limit, and also to avoid the distraction of the driver and improve the driving safety of the driver.

[0124] Embodiment 10:

[0125] The control system of the AI voice robot for the commercial vehicle provided in the embodiment of the present application comprises:

[0126] The control center is used for controlling the driver driving state acquisition module, the multi-modal driving assistance interaction module and the road condition warning optimization module.

[0127] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and the equivalent technology thereof, the present application also intends to include these modifications and variations.

Claims

1. An AI voice robot for commercial vehicles, characterized in that, include: The driver driving status acquisition module is used to collect the driving status of commercial vehicle drivers; The driving status includes at least: images of the driver's driving behavior, the real-time location of the vehicle, the driver's input voice commands, and the vehicle's operating conditions; The multimodal driving assistance interaction module is used to take the driver's driving status as input to the cloud-pre-trained AI model, and based on the output of the AI ​​model, to conduct multimodal driving assistance interaction with the driver. The multimodal driving assistance interaction module is also used to optimize the road condition risk warning information so that the driver can clearly perceive the road condition risk when providing driving assistance with road condition risk warning to the driver, and to use the optimized warning information to provide road condition warning to the driver. The multimodal driving assistance interaction module optimizes the road condition risk warning information to enable the driver to clearly perceive the road condition risks, including: The first average visibility of the risk areas of the road condition risk in each of the driver's current driving view and the predicted future driving view within the time limit not exceeding the warning time limit is analyzed on the driving map. When the first average visibility exceeds the first visibility threshold, a first view is generated on the driving map to show the first relative positional relationship between the driver's current driving position and the risk area from a bird's-eye view, and the information of the first view for the driver to view is used as the optimized warning information; otherwise, the second average visibility of each landmark visible in the current driving field of vision and the future driving field of vision is analyzed on the driving map. Iterate through the landmarks corresponding to each of the multiple second average visibility values ​​that are higher than the second visibility threshold; Each time the map is traversed, the driver's ability to clearly perceive road condition risks is reflected by analyzing the second relative positional relationship between the driver's current driving position and the traversed landmarks, the third relative positional relationship between the traversed landmarks and the risk areas, and the salience information of the traversed landmarks. After the sequential traversal is completed, a second view of the second relative position relationship of the landmark corresponding to the maximum capability value is generated on the driving map, and a third view of the second and third relative position relationships of the landmark corresponding to the maximum capability value is generated. The information of the second and third views, which are viewed by the driver in sequence, is used as the optimized warning information.

2. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The driver driving status acquisition module includes: The camera unit is used to collect image information of the driver inside the commercial vehicle; The positioning unit is used to collect real-time positioning information of commercial vehicles; A voice input unit is used to collect the voice input information of the driver inside the commercial vehicle; The in-vehicle communication gateway unit is used to collect operating information of in-vehicle equipment in commercial vehicles.

3. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The multimodal driving assistance interaction module includes: The cloud communication unit is used to establish a remote communication connection with the cloud via 4G / 5G communication. The display unit is used to interact with the driver through a screen display; A voice output unit is used to interact with the driver via voice output.

4. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The pre-training steps for large AI models include: A large number of data, including at least historical dialogue records of commercial vehicle drivers, driving risk records, route optimization records, business collaboration and scheduling records, waybill planning records, emotion management records, and point-of-interest recommendation records, are used as training samples to train a pre-selected basic large language model to obtain a large AI model.

5. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The multimodal driving assistance interaction module, based on the output of a large AI model, engages in multimodal driving assistance interaction with the driver, including: Based on the output of the AI ​​big model, the system interacts with drivers in at least the following ways: safe driving reminders, dynamic route optimization, business collaborative scheduling, waybill planning, emotion management, and point of interest recommendations.

6. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The signage includes at least: other vehicles, roadside facilities, buildings, and traffic warning devices.

7. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The steps for obtaining the ability value include: Based on the preset quantification system, the ability of drivers to clearly perceive road condition risks through the traversed landmarks is quantified by the second relative position relationship, the third relative position relationship, and the salience information, respectively, and the first quantification value, the second quantification value, and the third quantification value are obtained in sequence. Based on the weights pre-set according to the degree of influence of the first, second, and third quantified values ​​on the capability value, the first, second, and third quantified values ​​are weighted and calculated, and the weighted calculation result is used as the capability value.

8. The AI ​​voice robot for commercial vehicles as described in claim 1, characterized in that, The method of using optimized early warning information to provide road condition warnings to drivers includes: Information recommending safety measures that drivers should take to address road condition risks; Predict the period in the future when drivers will be least distracted when viewing warning and advice information before the warning deadline; During periods of minimal distraction, provide drivers with warning and advisory information.

9. The control system for an AI voice robot for commercial vehicles as described in any one of claims 1-8, characterized in that, include: The control center is used to control the driver's driving status acquisition module and the multimodal driving assistance interaction module.

Images