Driver fatigue interaction method, RSU, OBU and storage medium

Abstract

The invention discloses a driver fatigue interaction method, an RSU, an OBU and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: monitoring the fatigue condition of a driver through various sensors connected with the OBU; the OBU performs comprehensive analysis according to the monitored fatigue condition and judges the fatigue degree of the driver; when it is judged that the fatigue degree meets the feedback condition, the OBU feeds back the current fatigue condition to the RSU through the V2X network; and when the fatigue degree fed back by the OBU is received, the RSU uses artificial intelligence to start an interaction service provided for the driver. According to the driver fatigue interaction method, the RSU, the OBU and the storage medium provided by the invention, the fatigue condition of the driver is monitored in real time and personalized chat service is provided, so that the traffic accident risk caused by fatigue can be effectively reduced, and the system does not interfere with the normal driving operation of the driver and has relatively high practicability and safety.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a driver fatigue interaction method, an RSU, an OBU and a storage medium. Background Art

[0002] With the development of society, cars have become one of the main ways for people to travel. However, long-term driving can easily make drivers tired, leading to driving errors and traffic accidents. Therefore, how to reduce driver fatigue and improve road safety has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present invention is to provide a driver fatigue interaction method, RSU, OBU and storage medium, which can use the voice interaction function to communicate with the driver to distract the driver's attention and reduce his fatigue.

[0004] The present invention provides the following scheme:

[0005] According to one aspect of the present invention, a driver fatigue interaction method is provided, the driver fatigue interaction method comprising:

[0006] Monitor the driver's fatigue status through various sensors connected to the OBU;

[0007] Based on the fatigue status detected, the OBU conducts a comprehensive analysis to determine the driver’s fatigue level;

[0008] When the judged fatigue level meets the feedback conditions, the OBU feeds back the current fatigue status to the RSU through the V2X network;

[0009] When receiving fatigue level feedback from the OBU, the RSU uses artificial intelligence to initiate interactive services for the driver.

[0010] Optionally, various sensors connected to the OBU can be used to monitor the driver's fatigue, including:

[0011] Acquire a template image of the driver in a typical fatigue state;

[0012] Obtain real-time facial images of the driver;

[0013] Extract feature vectors of the template image and the real-time facial image respectively;

[0014] Calculating image similarity between a feature vector of the template image and a feature vector of the real-time facial image;

[0015] According to the image similarity, the driver's fatigue status is obtained.

[0016] Optionally, various sensors connected to the OBU can be used to monitor the driver's fatigue, including:

[0017] Get the time data for adjusting the steering wheel angle;

[0018] The time data for adjusting the steering wheel angle includes the time interval data between the most recent time of adjusting the steering wheel angle and the previous time of adjusting the steering wheel angle;

[0019] Obtaining historical data of the time interval data;

[0020] The historical data includes data of the time interval of the preset historical time interval;

[0021] According to the historical data, obtaining average value data of the time interval data;

[0022] The fatigue status of the driver is obtained according to the average value data of the time interval data of the most recent steering wheel angle adjustment and the time interval data of the historical data.

[0023] Optionally, the OBU conducts a comprehensive analysis based on the monitored fatigue status to determine the driver's fatigue level, including:

[0024] The fatigue conditions obtained by different sensors are weighted averaged to obtain the driver's fatigue level.

[0025] Optionally, the OBU conducts a comprehensive analysis based on the monitored fatigue status to determine the driver's fatigue level, including:

[0026] The fatigue conditions obtained by different sensors are input into the pre-trained ANN to obtain the driver's fatigue level.

[0027] Optionally, the feedback conditions include: the driver's fatigue level exceeds a preset threshold;

[0028] The jump value of the driver's fatigue level exceeds the preset jump threshold; or

[0029] The weighted average of the derivative vector change of the driver's fatigue level exceeds a preset weighted threshold.

[0030] Optionally, also include:

[0031] When receiving the fatigue level feedback from the OBU, the RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles.

[0032] Optionally, the RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including:

[0033] Set the distance threshold between surrounding vehicles and the target vehicle;

[0034] When it is detected that the distance between the vehicle and the target vehicle is less than the distance threshold, a distance warning is sent to the vehicle whose distance is less than the distance threshold.

[0035] Optionally, the RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including:

[0036] Calculate the vehicle speed control threshold according to the distance threshold to the target vehicle;

[0037] Get the vehicle's speed and direction in real time;

[0038] When the vehicle speed is greater than the speed control threshold and the direction is a preset direction, a speed warning is sent to the vehicle.

[0039] Optionally, the RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including:

[0040] Obtain the real-time change of vehicle speed;

[0041] When the real-time change in vehicle speed is greater than a preset change threshold, a speed warning is sent to the vehicle.

[0042] Optionally, the interaction includes: adjustment of the ambient light display parameters based on the AI ​​model; and

[0043] Human-computer interaction based on language big model.

[0044] Optional AI model-based ambient light display parameter adjustment, including:

[0045] When the fatigue level feedback from the OBU is received, the AI ​​model for adjusting the display parameters of the ambient light set on the OBU side is started;

[0046] The AI ​​model is used to adjust the display parameters of the ambient light in the cabin according to preset steps.

[0047] Optional AI model-based ambient light display parameter adjustment, including:

[0048] When the fatigue level feedback from the OBU is received, the AI ​​model for adjusting the display parameters of the ambient light set on the OBU side is started;

[0049] Use the AI ​​model to adjust the display parameters of the cabin ambient light to the preset parameters.

[0050] Optional human-computer interaction based on a large language model, including:

[0051] When the fatigue level is fed back by the OBU, the language model for human-computer interaction set at the OBU end is started;

[0052] Based on the language big model, initiate human-computer interaction with the driver.

[0053] According to two aspects of the present invention, a driver fatigue interaction method is provided, which is applied to RSU, and is characterized in that the driver fatigue interaction method comprises:

[0054] When the fatigue level feedback from the OBU is received, artificial intelligence is used to initiate interactive services for the driver.

[0055] According to three aspects of the present invention, a driver fatigue interaction method is provided, which is applied to an OBU, and is characterized in that the driver fatigue interaction method comprises:

[0056] Monitor the driver's fatigue status through various sensors connected to the OBU;

[0057] Based on the monitored fatigue status, a comprehensive analysis is conducted to determine the driver's fatigue level;

[0058] When the judged fatigue level meets the feedback conditions, the current fatigue status is fed back to the RSU through the V2X network.

[0059] According to four aspects of the present invention, there is provided an RSU, characterized in that it includes:

[0060] one or more processors;

[0061] a storage device for storing one or more programs,

[0062] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0063] According to five aspects of the present invention, an OBU is provided, characterized in that it includes:

[0064] one or more processors;

[0065] a storage device for storing one or more programs,

[0066] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0067] According to a sixth aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program executable by an RSU. When the computer program runs on the RSU, the RSU executes the steps of the method described above.

[0068] According to a seventh aspect of the present invention, there is provided a computer-readable storage medium storing a computer program executable by an OBU, wherein when the computer program runs on the OBU, the OBU executes the steps of the method described above.

[0069] Through the above scheme, the following beneficial technical effects are obtained:

[0070] By monitoring the driver's fatigue status in real time and providing personalized interactive services, the risk of traffic accidents caused by fatigue can be effectively reduced;

[0071] At the same time, the system will not interfere with the driver's normal driving operations and has high practicality and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a system architecture diagram of V2X communication provided by one or more embodiments of the present invention.

[0073] Figure 2 It is a flow chart of a driver fatigue interaction method provided by one or more embodiments of the present invention.

[0074] Figure 3 is a flow chart of fatigue status monitoring provided by one or more embodiments of the present invention.

[0075] Figure 4 is a flow chart of fatigue status monitoring provided by one or more embodiments of the present invention.

[0076] Figure 5 It is a schematic diagram of the principle of feedback conditions provided by one or more embodiments of the present invention.

[0077] Figure 6 It is a schematic diagram of the principle of vehicle speed and distance control provided by one or more embodiments of the present invention.

[0078] Figure 7 is a flow chart of vehicle distance control provided by one or more embodiments of the present invention.

[0079] Figure 8 is a flow chart of vehicle speed control provided by one or more embodiments of the present invention.

[0080] Fig. 9 is a flow chart of vehicle speed control provided by one or more embodiments of the present invention.

[0081] Fig.10 It is a flow chart of adjusting the display parameters of the atmosphere light provided by one or more embodiments of the present invention.

[0082] Fig.11It is a flowchart of human-computer interaction provided by one or more embodiments of the present invention.

[0083] Fig.12 It is a system architecture diagram of an RSU or OBU provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0084] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] like Figure 1 As shown in the figure, the V2X communication system mainly includes communication between vehicles (Vehicle-to-Vehicle, V2V), communication between vehicles and infrastructure (Vehicle-to-Infrastructure, V2I), communication between vehicles and networks (Vehicle-to-Network, V2N), and communication between vehicles and people (Vehicle-to-Pedestrian, V2P).

[0086] To solve the problem of communication capabilities between vehicles and transportation infrastructure, relevant scholars, organizations and industries have conducted a lot of research and reached some consensus on improving the safety and efficiency of transportation systems. With the improvement of sensing technology, the use of wireless connection technology and the increase in transmission speed, V2X provides solutions for supporting advanced safety applications to optimize vehicle traffic performance and user infotainment experience. This technology can effectively help vehicles notice blind spots while driving and remind drivers of different types of obstacles. It has great application prospects in avoiding traffic accidents, alleviating traffic congestion, and improving environmental quality. In addition, V2X can achieve rapid communication between all drivers and provide drivers with alerts of potential threats and obstacles. The system's safety features can avoid vehicle collisions and provide timely warnings of emergencies, including pedestrian notifications at crosswalks, queue warnings, road safety alerts, parking space detection, work zone warnings, curve speed warnings, parking warnings, and even weather impact warnings. In-vehicle network technology relies on the interconnection of drivers. As the technology becomes more and more common, vehicles are expected to communicate with anyone and anything, including cyclists and pedestrians.

[0087] Figure 2 is a flow chart of a driver fatigue interaction method provided by one or more embodiments of the present invention. Figure 2 , the driver fatigue interaction method includes the following operating steps:

[0088] S21 monitors the driver's fatigue status through various sensors connected to the OBU.

[0089] S22, OBU conducts a comprehensive analysis based on the monitored fatigue status to determine the driver's fatigue level.

[0090] S23, when the judged fatigue level meets the feedback conditions, the OBU feeds back the current fatigue status to the RSU through the V2X network.

[0091] S24, upon receiving the fatigue level feedback from the OBU, the RSU uses artificial intelligence to start providing interactive services to the driver.

[0092] Depend on Figure 2 The illustrated embodiment is applied in a V2X network environment. Specifically, an OBU is configured on the vehicle side. The OBU is used to complete the monitoring of the driver's fatigue status, the judgment of the fatigue level, and the fatigue level feedback. On the road side, an RSU is configured. The RSU completes the action of providing interactive services to the driver after receiving the driver's fatigue status feedback from the OBU.

[0093] However, it should be noted that Figure 2 The method execution process shown, and the allocation of each operation step between different network element devices, is only one implementation of the technical solution to be protected by the present invention. It does not exclude the technical solution of redistributing each step of the method between various network element devices in the V2X communication system.

[0094] For example, TA can be used as a central node to further participate in the central decision-making process of fatigue judgment and interactive service process.

[0095] In a typical implementation, in addition to the OBU, the vehicle side is also equipped with a variety of sensors interconnected with the OBU, including cameras, temperature sensors, and vehicle speed sensors.

[0096] Each sensor can monitor the driver's data in real time and transmit the monitoring data back to the OBU in real time. For example, the camera can collect the driver's facial image in real time and transmit the collected facial image data back to the OBU.

[0097] After receiving these fatigue conditions, the OBU conducts a comprehensive analysis of the various fatigue conditions received and obtains a judgment result on the fatigue degree.

[0098] The above fatigue level judgment result can be obtained, typically, by weighted average after uniform normalization of various fatigue status data. That is, different weight coefficients are assigned to each fatigue status data, and then the weighted average of each fatigue status is calculated. The final value is the fatigue level judgment result.

[0099] It should be understood that the final judgment result of the fatigue degree is a variable that changes in real time. In other words, the fatigue status signals collected by different sensors in real time are weighted averaged in real time, and the result is inevitably a variable that changes in real time.

[0100] After obtaining the fatigue level judgment result, the next step is to judge whether the preset feedback condition is met according to the fatigue level. There are multiple options for the feedback condition. Once the feedback condition is met, the fatigue level judgment result should be fed back to the roadside RSU.

[0101] The occurrence of feedback to the RSU indicates that the current vehicle driver has confirmed fatigue and needs to activate some auxiliary interactive means to relieve the driver's fatigue.

[0102] These auxiliary means of interaction include: adjusting the ambient lighting in the cabin, or intelligent chat based on a large language model.

[0103] The feedback condition may be a feedback condition based on the change in fatigue level itself, a feedback condition based on the change in the first-order derivative of fatigue level, or a feedback condition based on the change in various-order derivatives of fatigue level.

[0104] After the above control process, not only can it be detected in real time whether the driver is mentally fatigued, but also, if fatigue has occurred, it can be relieved in time, thereby effectively improving the safety of vehicle driving control.

[0105] Moreover, the execution process of the above control will not affect the driver's normal driving operation, and the scheme is quite credible and operational.

[0106] Figure 3 This is a process for monitoring fatigue status provided by one or more embodiments of the present invention. Figure 3 , through various sensors connected to the OBU, monitoring the driver's fatigue status includes the following steps:

[0107] S31, obtaining a template image of the driver in a typical fatigue state.

[0108] S32, obtaining a real-time facial image of the driver.

[0109] S33, extracting feature vectors of the template image and the real-time facial image respectively.

[0110] S34, calculating the image similarity between the feature vector of the template image and the feature vector of the real-time facial image.

[0111] S35, obtaining the driver's fatigue status according to the image similarity.

[0112] In by Figure 3 In the illustrated embodiment, the driver's fatigue status is acquired by acquiring the driver's facial image in real time.

[0113] It is understandable that in order to obtain the driver's fatigue status through real-time facial images, some data is needed to support the acquisition process. Figure 3 In the illustrated embodiment, the data support is a fatigue template image.

[0114] It is understandable that some facial images that are determined to be in a fatigue state can be obtained. These images that have been determined to be in a fatigue state are used as templates. When it is necessary to obtain the driver's fatigue state, the driver's facial images obtained in real time are compared with these fatigue template images to obtain the driver's fatigue state.

[0115] Specifically, by comparing the real-time facial image with the fatigue template image, scoring the similarity between the real-time facial image and the fatigue template image, and using the above score as a representation of the driver's fatigue condition, the driver's fatigue condition is obtained.

[0116] More specifically, since the image data has more details, the amount of calculation for direct comparison is large. Therefore, in the process of comparing the real-time facial image with the fatigue template image, the feature vectors of the real-time facial image and the fatigue template image are firstly extracted.

[0117] The feature vector is composed of a set of feature elements. The feature elements in the feature vector are all key variables that characterize whether the face in the image is in a state of fatigue. For example, the feature element can be the aspect ratio of the rectangle outside the eye image. The feature element can also be the degree of the upward movement of the corners of the mouth.

[0118] After extracting feature vectors from the real-time facial image and the fatigue template image respectively, the similarity between the two extracted feature vectors is calculated.

[0119] Specifically, the cosine similarity between two feature vectors may be calculated. The Tanimoto coefficient between two feature vectors may also be calculated. The Pearson correlation coefficient between two feature vectors may also be calculated. The Jaccard similarity coefficient between two feature vectors may also be calculated.

[0120] After the similarity between the two feature vectors is calculated, the calculated similarity is used as a representation of the driver's fatigue status.

[0121] The above-mentioned comparison between the real-time facial image and the fatigue template image by comparing the feature vectors is only a specific implementation of the above-mentioned comparison operation. Any implementation that can accurately determine the fatigue degree of the face in the real-time facial image can be used as a specific implementation of fatigue status acquisition.

[0122] Figure 4 This is a process for monitoring the elderly companionship status provided by one or more embodiments of the present invention. Figure 4 , through various sensors connected to the OBU, the driver's fatigue status is monitored, including:

[0123] S41, obtaining time data for adjusting the steering wheel angle.

[0124] The time data for adjusting the steering wheel angle is obtained including time interval data between the most recent time of adjusting the steering wheel angle and the previous time of adjusting the steering wheel angle.

[0125] S42, obtaining historical data of the time interval data.

[0126] The historical data includes data of the time interval of a preset historical time interval.

[0127] S43: Obtain average value data of the time interval data according to the historical data.

[0128] S44, obtaining the driver's fatigue status according to the average value data of the time interval data of the most recent steering wheel angle adjustment and the time interval data of the historical data.

[0129] It should be understood that if the steering wheel is frequently adjusted, that is, the steering wheel angle changes frequently, it means that the driver is manipulating the vehicle in real time according to various factors such as road conditions. If the driver's above adjustment actions are frequent enough, it means that the driver is currently awake and not in a fatigued state.

[0130] In by Figure 4 In the illustrated embodiment, the frequency of change of the steering wheel angle is used to obtain the real-time fatigue status of the driver.

[0131] Specifically, the time point when the steering wheel angle was last adjusted before the current time is recorded.

[0132] In order to accurately record the time point when the steering wheel angle is adjusted, an array for recording the adjustment time point can be set in the storage space. Whenever the steering wheel angle is adjusted, the time point information of the steering wheel angle adjustment is recorded in the last storage unit of the array. At the same time, the storage positions of other storage units in the array are moved forward in sequence.

[0133] When it is necessary to obtain the time point information when the steering wheel angle was last adjusted, the recorded time point information is obtained from the last storage unit of the array, and the obtained time point information is the time point when the steering wheel angle was last adjusted.

[0134] After obtaining the time point when the steering wheel angle was last adjusted, the interval between the time point of the last adjustment and the current adjustment time point is calculated.

[0135] It should be understood that the above interval, that is, the interval between two steering wheel angle adjustment actions, has a historical average value.

[0136] The above interval is subtracted from the historical mean of the interval to obtain the difference between the two, and the difference is used to characterize the driver's fatigue status.

[0137] Figure 5 is a schematic diagram of the principle of feedback conditions provided by one or more embodiments of the present invention. Figure 5 In some embodiments, the feedback condition is characterized by various derivatives of the fatigue degree. If the feedback condition is regarded as a one-dimensional vector, then each element in the one-dimensional vector is a derivative of the fatigue degree. For example, the first element is the fatigue degree itself, the second element is the first derivative of the fatigue degree, and so on. Alternatively, the first element is the first derivative of the fatigue degree, the second element is the second derivative of the fatigue degree, and so on.

[0138] When judging whether the feedback condition is met, the difference between the current fatigue level derivatives and the feedback condition vector is taken. It can be understood that this difference is also a vector. If the weighted average of this vector is obtained and the average value exceeds the preset threshold, it can be judged that the preset feedback condition has been met and feedback can be given to the RSU.

[0139] As a special example of the above embodiment, the feedback condition may be a one-dimensional vector with only one element, and the only element may be a threshold of the fatigue degree itself or a threshold of the first-order derivative of the fatigue degree.

[0140] Figure 6 Schematic diagram of the principle of vehicle speed and distance control provided by one or more embodiments of the present invention. Figure 6 The process of vehicle-to-vehicle distance and speed control includes the following steps:

[0141] S61, the target vehicle feeds back the fatigue level to the RSU.

[0142] S62, the RSU queries other vehicles near the target vehicle.

[0143] S63, the RSU performs vehicle distance and vehicle speed control on other vehicles found.

[0144] It should be understood that the distance and speed control performed by the RSU on other vehicles is performed on the premise that the driver of the vehicle is already fatigued. Because in the technical solution of the present application, the fatigue of the driver is equivalent to the target vehicle feeding back the fatigue degree to the RSU. Therefore, only when the target vehicle feeds back the fatigue degree to the RSU, the RSU will start the distance and speed control for other vehicles.

[0145] The main body that performs vehicle distance and speed control is the RSU, which is the network element device that receives the fatigue level.

[0146] The target of the vehicle-to-vehicle distance and speed control is other vehicles that are close to the target vehicle in terms of distance. Here, the target vehicle is the vehicle that feeds back the fatigue level to the RSU. Close in distance means close in geographical distance.

[0147] In a typical case, after receiving the fatigue level feedback from the target vehicle, the RSU will start its own LBS service and query other vehicles that are currently geographically close to the target vehicle based on the started LBS service.

[0148] By searching for other vehicles that are geographically close, the target network element device for executing the distance and speed control is found. At this time, the distance and speed control operation for other vehicles is started.

[0149] The current position of each vehicle can be obtained by requesting the current position of the vehicle in real time, so as to further query the vehicles that are close to the target vehicle in distance.

[0150] The advantage of real-time location query is that the location information obtained is relatively accurate and does not require too much computing resources of the RSU. However, this method also has some disadvantages, that is, it occupies more network traffic and consumes more network bandwidth resources.

[0151] Another way to obtain location information is to estimate the current location of the vehicle using historical data such as the vehicle's historical location, historical speed, historical driving direction, etc. This method of obtaining location takes up less network bandwidth resources, but consumes more local computing resources of the RSU.

[0152] Regardless of the method used to obtain location information, the ultimate goal is to obtain accurate location information of vehicles on the road, so that the RSU can accurately identify which vehicles should be subject to distance and speed control.

[0153] Figure 7 is a flow chart of vehicle distance control provided by one or more embodiments of the present invention. Figure 7 , RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including the following operating steps:

[0154] S71, setting a distance threshold between surrounding vehicles and the target vehicle.

[0155] S72, when it is monitored that the distance between the vehicles is less than the distance threshold, a distance warning is sent to the vehicle whose distance is less than the distance threshold.

[0156] It should be understood that the RSU will only start the distance control for the surrounding vehicles if the target vehicle feeds back the fatigue status to the RSU.

[0157] If the target vehicle reports fatigue to the RSU, it means that the driver of the target vehicle is already fatigued. At this time, the drivers of vehicles on the road that are close to the target vehicle should be notified as soon as possible that the drivers of nearby vehicles are already fatigued, and they should be reminded to drive carefully to avoid traffic accidents.

[0158] The simplest and most direct way to control traffic is to control the distance between nearby vehicles and the target vehicle. Because when two vehicles are close in space, the driving state of the other vehicle is most likely to affect the vehicle itself, and accidents are most likely to occur.

[0159] The simplest way to control the distance between vehicles is distance threshold control. That is, a distance threshold of the space distance between two vehicles is set in advance, and then the space distance between the two vehicles is monitored in real time. Moreover, when the space distance monitored in real time between the two vehicles is less than this distance threshold, a distance warning is sent to one of the two vehicles that is not the target vehicle, that is, the vehicle close to the target vehicle. This vehicle is notified to maintain a safe distance from the vehicle that has become fatigued.

[0160] The distance warning can be sent through the communication connection between the RSU and the vehicle. However, if the network between the RSU and the vehicle that needs to receive the warning is different, the warning can be sent to the other vehicle first, and then sent to the vehicle that needs to receive the warning information through the communication connection between the vehicles.

[0161] Moreover, in order to ensure that the above-mentioned warning information has indeed been received, after receiving the above-mentioned warning information, the vehicle needs to feedback confirmation information to the RSU. After receiving the confirmation information fed back by the vehicle, the RSU can confirm that the above-mentioned warning information has been received. If the above-mentioned warning information is not received within the specified time, the RSU needs to repeatedly send this warning information to the vehicle until the confirmation information of the warning information is returned to the RSU.

[0162] The reason why feedback confirmation communication is used for distance warning information is to ensure that the warning information is reliably transmitted to the vehicle and to promptly remind surrounding vehicles of possible risks in driving on the current road.

[0163] Figure 8 is a flow chart of vehicle speed control provided by one or more embodiments of the present invention. Figure 8 , RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including the following operating steps:

[0164] S81, calculating a vehicle speed control threshold according to a distance threshold to a target vehicle.

[0165] S82, obtaining the speed and direction of the vehicle in real time.

[0166] S83: When the speed of the vehicle is greater than the speed control threshold and the direction is a preset direction, a speed warning is sent to the vehicle.

[0167] In addition to controlling the distance between vehicles and surrounding vehicles, it is also necessary to control the speed of surrounding vehicles.

[0168] During the speed control process, a speed threshold is required. When the real-time speed of the vehicle is detected to be higher than the speed threshold, a speed warning should be sent to the vehicle. If the real-time speed of the vehicle is currently detected to be lower than the speed threshold, it means that the surrounding vehicles are currently in a safe speed range and no further speed control is required.

[0169] In this embodiment, the vehicle speed threshold is obtained by calculation. The safe distance between two vehicles divided by the driver's usual reaction time is the vehicle speed control threshold used in the control process.

[0170] During the vehicle speed control process, in addition to the parameters of the vehicle speed control threshold, the vehicle driving direction control parameter is also required.

[0171] The method of obtaining the direction of travel of the vehicle is very simple, that is, obtaining the real-time position of the vehicle at two consecutive time points, and referring to the difference between the two positions obtained is the actual direction of travel of the vehicle.

[0172] The speed of the vehicle is also relatively simple to obtain. The actual position of the vehicle at two consecutive time points is obtained, and the actual displacement of the vehicle during this period is obtained by calculating the spatial distance between the two consecutive positions. Then, this actual displacement is divided by the time interval between the two consecutive time points to obtain the actual speed of the vehicle.

[0173] After obtaining the actual speed of the vehicle, the obtained speed is compared with the speed control threshold. If the actual speed is higher than the speed control threshold, the direction of the vehicle is further determined. If the direction of the vehicle is also towards the fatigued vehicle, a speed warning should be issued to the vehicle.

[0174] If the actual vehicle speed is lower than the speed control threshold, it means that the current vehicle speed is within the normal value range, and there is no need to determine the vehicle's direction of travel.

[0175] Similar to the above-mentioned vehicle distance control, the above-mentioned speed warning issued to the vehicle also requires feedback confirmation from the vehicle to ensure that the issued speed warning has indeed been received by the vehicle.

[0176] Fig. 9 is a flow chart of vehicle speed control provided by one or more embodiments of the present invention. Fig. 9 , RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including the following operating steps:

[0177] S91, obtaining a real-time change in the vehicle speed.

[0178] S92: When the real-time change in vehicle speed is greater than a preset change threshold, a vehicle speed warning is sent to the vehicle.

[0179] Different from the above-mentioned embodiment, in Fig. 9 In the illustrated embodiment, the determination of when to send the vehicle speed warning is no longer based on the vehicle speed variable itself, but is based on the change in the vehicle speed.

[0180] This judgment logic is based on the fact that when driving on the road, if the speed of the vehicle changes suddenly, the probability of a traffic accident is higher. For example, if the vehicle brakes suddenly or accelerates suddenly, there is a high probability that the vehicle will get into a traffic accident soon.

[0181] The change in vehicle speed is calculated based on the vehicle speed acquired in real time. In other words, the difference between the vehicle speed at the previous moment and the vehicle speed at the next moment is the change in vehicle speed.

[0182] There is also a safe range for the speed change, which can be obtained by analyzing the historical vehicle trajectories.

[0183] For example, the driving trajectories of vehicles on the road can be collected. The driving trajectories where accidents occur can be selected from all driving trajectories. The driving data before and after the accident point can be selected from the driving trajectories where accidents occur. Based on these driving data, the range of vehicle speed changes can be calculated. It can be understood that the range of vehicle speed changes before and after the accident point is a dangerous data range. Eliminating these dangerous ranges of changes from the normal range of vehicle speed changes will give the normal range of vehicle speed changes under normal circumstances.

[0184] With the normal range of vehicle speed variation, the threshold of vehicle speed variation can be set accordingly.

[0185] Alternatively, the normal range of the vehicle speed change can be used as a reference for judging the vehicle speed change. However, when the vehicle speed change is within the normal range, it is considered that the current driving is safe and no warning information needs to be sent to the vehicle. Once the vehicle speed change exceeds the normal range obtained through data screening, it is considered that the current driving is unsafe and a warning information needs to be sent to the vehicle.

[0186] As in the previous embodiment of the present application, Fig. 9 The vehicle speed warning information sent by the illustrated embodiment also requires vehicle feedback confirmation information.

[0187] Fig.10 It is a flow chart of adjusting the display parameters of the atmosphere light provided by one or more embodiments of the present invention.

[0188] See also Fig.10 , the AI ​​model-based ambient light display parameter adjustment includes:

[0189] S101, when the fatigue level feedback from the OBU is received, the AI ​​model for adjusting the ambient light display parameters set at the OBU end is started.

[0190] S102: Using the AI ​​model, the display parameters of the ambient light in the cabin are adjusted according to a preset step size.

[0191] When the driver is detected to be fatigued, some interactive means need to be initiated to relieve the driver's fatigue.

[0192] These auxiliary means of interaction include: adjustment of the display parameters of the ambient light in the cabin, and voice chat with the driver.

[0193] The most significant and effective adjustment of the display parameters of the ambient light is the adjustment of the display brightness and display color temperature. When the driver is tired, the display brightness of the ambient light should be brightened and the color temperature should be adjusted to a cooler tone, so that the driver is less likely to become sleepy.

[0194] Moreover, the adjustment of the ambient light display parameters is based on the AI ​​model. The AI ​​model will comprehensively consider various parameters inside and outside the cabin and make comprehensive adjustments to the display parameters of the ambient light.

[0195] The timing of starting the adjustment of the AI ​​model is very important, that is, when the RSU receives the fatigue level feedback from the OBU. Only when the fatigue level feedback is received, the RSU will start the AI ​​model running on the OBU.

[0196] In some embodiments, the AI ​​model adjusts the display parameters in accordance with a preset step length. In this embodiment, whether it is the adjustment of the display brightness or the adjustment of the display color temperature, each adjustment is made in a fixed step length. In other words, the adjustment amount of each adjustment is fixed.

[0197] In other embodiments, the adjustment of the display parameters by the AI ​​model is an adjustment with an adjustment target set. That is, in each adjustment process, the adjustment amount of the adjustment is uncertain, but the adjustment target amount is certain.

[0198] Regardless of the adjustment method, the ultimate goal is to prevent the driver in the cockpit from becoming sleepy again, or to enable the driver to quickly recover from the current fatigue state.

[0199] Fig.11 is a flowchart of human-computer interaction provided by one or more embodiments of the present invention. Fig.11 , human-computer interaction based on language big model, including:

[0200] S111, when the fatigue level feedback from the OBU is received, the language large model for human-computer interaction set at the OBU end is started.

[0201] S112, based on the language big model, starts the human-computer interaction with the driver.

[0202] Another way of interaction is human-computer interaction based on a large language model, which is what we usually call a chat service.

[0203] According to scientific research data, after starting the chat service, drivers can recover from fatigue faster and better concentrate on the driving operation process.

[0204] The language model runs locally on the OBU and is started according to the instructions of the RSU. After the language model is started according to the instructions of the RSU, the driver can interact with the OBU through the large model to relieve his fatigue.

[0205] The present application also provides a driver fatigue interaction method. The method is applied to RSU, and the driver fatigue interaction method includes: upon receiving the fatigue level feedback from OBU, using artificial intelligence to start the interactive service provided to the driver.

[0206] The present application also provides a driver fatigue interaction method. The method is applied to OBU, and the driver fatigue interaction method includes: monitoring the driver's fatigue status through various sensors connected to OBU; performing comprehensive analysis based on the monitored fatigue status to determine the driver's fatigue level; when the determined fatigue level meets the feedback condition, the current fatigue status is fed back to RSU through the V2X network.

[0207] Fig.12 It is a schematic diagram of the structure of the computer system of RSU and OBU shown in an exemplary embodiment of the present application. It should be noted that: Fig.12 The computer system 1200 of the RSU and OBU is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0208] like Fig.12 , the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part to the random access memory (RAM) 1203, such as executing the method in the above embodiment. In RAM 1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through a bus. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0209] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0210] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the device of the present application are executed.

[0211] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0212] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0213] In the corresponding drawings of the above embodiments, connecting lines may indicate the connection relationship between the various components to indicate more constituent signal paths and / or one or more ends of some lines may have arrows to indicate the main information flow direction. Connecting lines, as a mark, are not a limitation to the scheme itself, but the use of these lines in combination with one or more exemplary embodiments helps to connect circuits or logic units more easily. Any represented signal (determined by design requirements or preferences) may actually include one or more signals that can be transmitted in either direction and implemented with any appropriate type of signal scheme.

[0214] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0215] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented. The computer-readable storage medium may be included in the RSU or OBU described in the above embodiments, or may exist independently without being assembled into the RSU or OBU.

[0216] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0217] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0218] It should be noted that the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0219] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0221] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A driver fatigue interaction method, characterized in that: The driver fatigue interaction method comprises: Monitor the driver's fatigue status through various sensors connected to the OBU; Based on the fatigue status detected, the OBU conducts a comprehensive analysis to determine the driver’s fatigue level; When the judged fatigue level meets the feedback conditions, the OBU feeds back the current fatigue status to the RSU through the V2X network; When receiving fatigue level feedback from the OBU, the RSU uses artificial intelligence to initiate interactive services for the driver.

2. The method according to claim 1, characterized in that The driver's fatigue status is monitored through various sensors connected to the OBU, including: Acquire a template image of a driver in a typical fatigue state; Obtain real-time facial images of the driver; Extract feature vectors of the template image and the real-time facial image respectively; Calculating image similarity between a feature vector of the template image and a feature vector of the real-time facial image; According to the image similarity, the driver's fatigue status is obtained.

3. The method according to claim 1, characterized in that The driver's fatigue status is monitored through various sensors connected to the OBU, including: Get the time data for adjusting the steering wheel angle; The time data for adjusting the steering wheel angle includes the time interval data between the most recent time of adjusting the steering wheel angle and the previous time of adjusting the steering wheel angle; Obtaining historical data of the time interval data; The historical data includes data of the time interval of the preset historical time interval; According to the historical data, obtaining average value data of the time interval data; The fatigue status of the driver is obtained according to the average value data of the time interval data of the most recent steering wheel angle adjustment and the time interval data of the historical data.

4. The method according to claim 1, characterized in that: Based on the fatigue status detected, the OBU conducts a comprehensive analysis to determine the driver's fatigue level, including: The fatigue conditions obtained by different sensors are weighted averaged to obtain the driver's fatigue level.

5. The method according to claim 1, characterized in that Based on the fatigue status detected, the OBU conducts a comprehensive analysis to determine the driver's fatigue level, including: The fatigue conditions obtained by different sensors are input into the pre-trained ANN to obtain the driver's fatigue level.

6. The method according to claim 1, characterized in that Feedback conditions include: the driver’s fatigue level exceeds a preset threshold; The jump value of the driver's fatigue level exceeds the preset jump threshold; or The weighted average of the derivative vector change of the driver's fatigue level exceeds a preset weighted threshold.

7. The method according to claim 1, characterized in that Also includes: When receiving the fatigue level feedback from the OBU, the RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles.

8. The method according to claim 7, characterized in that The RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including: Set the distance threshold between surrounding vehicles and the target vehicle; When it is detected that the distance between the vehicle and the target vehicle is less than the distance threshold, a distance warning is sent to the vehicle whose distance is less than the distance threshold.

9. The method according to claim 7, characterized in that: The RSU controls the distance between nearby vehicles and the target vehicle, as well as the speed of nearby vehicles, including: Calculate the vehicle speed control threshold according to the distance threshold to the target vehicle; Get the vehicle's speed and direction in real time; When the vehicle's speed is greater than the speed control threshold and the direction is a preset direction, a speed warning is sent to the vehicle.

10. A driver fatigue interaction method, applied to RSU, characterized in that: The driver fatigue interaction method comprises: When the fatigue level feedback from the OBU is received, artificial intelligence is used to initiate interactive services for the driver.

11. A driver fatigue interaction method, applied to OBU, characterized in that: The driver fatigue interaction method comprises: Monitor the driver's fatigue status through various sensors connected to the OBU; Based on the monitored fatigue status, a comprehensive analysis is conducted to determine the driver's fatigue level; When the judged fatigue level meets the feedback conditions, the current fatigue status is fed back to the RSU through the V2X network.

12. An RSU, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to claim 10.

13. An OBU, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to claim 11.

14. A computer-readable storage medium, characterized in that: It stores a computer program executable by the RSU, and when the computer program runs on the RSU, the RSU executes the steps of the method according to claim 10.

15. A computer-readable storage medium, characterized in that: It stores a computer program executable by the OBU. When the computer program runs on the OBU, the OBU executes the steps of the method according to claim 11.

Images