Driving state abnormality alerting method and device, vehicle, and storage medium

Abstract

A driving state abnormality alerting method and device, a vehicle, and a storage medium. The method comprises: acquiring driving environment information, historical dialogue information and vehicle travel data; on the basis of the driving environment information, the historical dialogue information and the vehicle travel data, in combination with a pre-trained interactive intervention model, generating alert content corresponding to a current environment; and generating a voice alert according to the alert content. An output voice alert is associated with the vehicle travel data and the driving environment information in the current environment, such that the voice alert is more accurate and reliable; and the voice alert is generated on the basis of the historical dialogue information, such that the service quality of the voice alert can be effectively improved.

Description

Abnormal driving state reminder method, device, car and storage medium Technical Field

[0001] The present application relates to the field of intelligent driving, and in particular to a method, device, vehicle, and storage medium for abnormal driving status reminders. Background Art

[0002] With the improvement of people's living standards and the popularization of automobiles, cars have become an important means of transportation, bringing great convenience to people's lives and work. However, during the driving process, abnormal driving conditions may occur, such as fatigue or driver distraction, which can easily lead to traffic accidents, resulting in personal injury and property damage.

[0003] To promptly detect abnormal driver status, a visually-based driver assistance system can be implemented, using the driver's status information as input to a state monitoring model. When the driver's status information becomes abnormal, the state monitoring model provides a voice prompt or manual warning. However, identification based on the driver's status is easily affected by environmental information, resulting in low accuracy and hindering the provision of high-quality intelligent intervention. Technical issues

[0004] In view of this, the embodiments of the present application provide a method, device, automobile and storage medium for abnormal driving status reminder to solve the problem in the prior art that when identifying the driver's status, it is easily interfered by environmental information, resulting in low accuracy of status recognition and being unfavorable for providing high-quality intelligent intervention. Technical Solutions

[0005] A first aspect of an embodiment of the present application provides a method for reminding of abnormal driving status, the method comprising:

[0006] Obtain driving environment information, historical conversation information, and vehicle driving data;

[0007] Generate reminder content corresponding to the current environment based on the driving environment information, historical conversation information, and vehicle driving data, combined with a pre-trained interactive intervention model;

[0008] Generate a voice reminder according to the reminder content.

[0009] In combination with the first aspect, in a first possible implementation manner of the first aspect, the driving environment information includes in-vehicle environment information and out-vehicle environment information;

[0010] Based on the driving environment information, conversation information, and vehicle travel data, combined with a pre-trained interactive intervention model, a reminder content corresponding to the current environment is generated, including:

[0011] Inputting the in-vehicle environment information into a pre-trained state monitoring model to obtain the driver's state information;

[0012] The in-vehicle environment information, the out-vehicle environment information, the historical conversation information, the vehicle driving data and the driver's status information are input into the interactive intervention model to generate reminder content corresponding to the current environment.

[0013] In combination with the first aspect or the first possible implementation of the first aspect, in a second possible implementation of the first aspect, obtaining driving environment information includes:

[0014] Acquiring at least one of lane line information, pedestrian information, surrounding vehicle information, obstacle information, and traffic signal information of a scene in which the vehicle is located through an automatic data acquisition system;

[0015] At least one of the driver's facial orientation information, eye information, mouth information, and hand information is obtained through a driver monitoring system.

[0016] In conjunction with the first possible implementation of the first aspect, in a third possible implementation of the first aspect, before generating reminder content corresponding to the current environment based on the driving environment information, historical conversation information, and vehicle driving data in combination with a pre-trained interactive intervention model, the method further includes:

[0017] Get the sound information collected by the microphone in the current scene;

[0018] Based on the driving environment information, historical conversation information, and vehicle driving data, combined with the pre-trained interactive intervention model, a reminder content corresponding to the current environment is generated, including:

[0019] Inputting the in-vehicle environment information into a pre-trained state monitoring model to obtain the driver's state information;

[0020] The driving environment information, historical conversation information, vehicle travel data, driver's status information and the sound information are input into a pre-trained interactive intervention model to generate reminder content corresponding to the current environment.

[0021] In conjunction with the first possible implementation of the first aspect, in a fourth possible implementation of the first aspect, inputting the driving environment information, historical conversation information, vehicle travel data, driver status information, and voice information into a pre-trained interactive intervention model includes:

[0022] Extracting image features from an image using a feature extraction model, wherein the image includes an in-vehicle environment image and an out-vehicle environment image in the driving environment information;

[0023] Recognizing text information in the historical conversation information and the sound information through a speech recognition model, and extracting feature vectors from the text information based on a pre-trained vector extraction model;

[0024] The image features, the feature vectors, the driver's state information, and the vehicle driving data are connected according to predetermined markers, and the data connected by the markers are input into the interactive intervention model.

[0025] In conjunction with the first aspect, in a fifth possible implementation of the first aspect, before generating reminder content corresponding to the current environment based on the driving environment information, historical conversation information, and vehicle driving data in combination with a pre-trained interactive intervention model, the method further includes:

[0026] Acquiring sample data, wherein the sample data includes driving environment information, historical conversation information, vehicle travel data, and standard content;

[0027] Inputting the driving environment information, historical conversation information, and vehicle driving data in the sample data into the interactive intervention model to calculate the reminder content;

[0028] The difference between the reminder content and the standard content is determined, and the parameters of the interactive intervention model are adjusted according to the difference until the difference meets the predetermined requirement.

[0029] In conjunction with the fifth possible implementation of the first aspect, in a sixth possible implementation of the first aspect, determining the difference between the reminder content and the standard content includes:

[0030] The word shift distance between the reminder content and the standard content is calculated, and the difference between the reminder content and the standard content is determined according to the word shift distance.

[0031] A second aspect of an embodiment of the present application provides a driving state abnormality reminder device, the device comprising:

[0032] An information acquisition unit, used to acquire driving environment information, historical conversation information, and vehicle travel data;

[0033] A reminder content generation unit, configured to generate reminder content corresponding to the current environment based on the driving environment information, historical conversation information, and vehicle driving data, in combination with a pre-trained interactive intervention model;

[0034] The voice reminder unit is used to generate a voice reminder according to the reminder content.

[0035] A third aspect of an embodiment of the present application provides a car, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in any one of the first aspects are implemented.

[0036] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented. Beneficial effects

[0037] The beneficial effects of the embodiments of the present application compared with the prior art are: the present application obtains driving environment information, historical conversation information and vehicle driving data, and performs calculations based on the driving environment information, historical conversation information and vehicle driving data in combination with the interactive intervention model to obtain reminder content corresponding to the current environment, and generates voice reminders based on the reminder content, so that the voice reminders are associated with the vehicle driving data and driving environment information in the current environment, making the output voice reminders more accurate and reliable, and based on the generation of historical conversation information, the service quality of the voice reminders can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or exemplary technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] FIG1 is a schematic diagram of an implementation scenario of a method for reminding of abnormal driving status provided by an embodiment of the present application;

[0040] FIG2 is a schematic diagram of an implementation flow of a method for reminding of abnormal driving status provided by an embodiment of the present application;

[0041] FIG3 is a schematic diagram of an implementation flow of a method for generating reminder content provided in an embodiment of the present application;

[0042] FIG4 is a schematic diagram of an implementation flow of a training interactive intervention model provided in an embodiment of the present application;

[0043] FIG5 is a schematic diagram of a driving state abnormality reminder device provided by an embodiment of the present application;

[0044] FIG6 is a schematic diagram of a car provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0045] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0046] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0047] To prevent accidents caused by driver fatigue or distraction, driver status data is typically collected while the driver is driving. This data is used to analyze abnormal driving conditions, such as fatigue or distraction. If the driver is in an abnormal state, an intervention reminder can be sent to the driver. However, this intervention is not always effective. After a period of time, the driver may return to their previous state, making it ineffective in preventing fatigue or distraction.

[0048] Figure 1 is a schematic diagram of an implementation scenario for a method for alerting drivers of abnormal driving conditions, as provided in an embodiment of the present application. As shown in Figure 1, the implementation scenario includes an ADAS (Automatic Data Acquisition System), a DMS (Driver Monitoring System), a microphone, a speaker, an ECU (Electronic Control Unit), and an interactive intervention system.

[0049] ADAS is used to collect external environmental information within the driving environment. This information includes one or more of the following: lane information, pedestrian information, vehicle information, obstacle information, and traffic signal information. Vehicle information can also include information such as the distance between the vehicle and other vehicles, and the speed and acceleration of other vehicles.

[0050] The DMS is used to collect in-vehicle environment information from the driving environment information. The in-vehicle environment information can be obtained through in-vehicle environment images. The in-vehicle environment information can include at least one of the following: the driver's facial orientation information, the driver's eye information, the driver's mouth information, and the driver's hand information.

[0051] The microphone can be used to collect sound information in the current scene, including the voice information of the driver and / or passengers. This sound information may include but is not limited to the content of the speech, tone particles, etc.

[0052] The ECU can be used to obtain vehicle driving data, including but not limited to the vehicle's (own vehicle's) speed, acceleration, and distance to the vehicle ahead or obstacles.

[0053] The speaker is used to play the calculated reminder content so that the driver can understand the current abnormal status in a timely manner.

[0054] The interactive intervention system can be used to calculate driving environment information collected by ADAS and DMS, vehicle driving data transmitted by the ECU, and voice information collected by the microphone, combined with pre-set historical conversation information, to generate output reminder content. For example, the state monitoring model can be used to calculate in-vehicle environmental information and determine the driver's state information based on the in-vehicle environmental information, including whether the driver is driving fatigued or distracted.

[0055] After determining the driver's status information, the in-vehicle environment information, out-of-vehicle environment information, historical conversation information, sound information in the current scene, vehicle driving data and the driver's status information can be input into the interactive intervention model to generate reminder content corresponding to the current environment.

[0056] Alternatively, the in-vehicle environment information, the out-of-vehicle environment information, the historical conversation information, the sound information in the current scene, and the vehicle driving data may be directly input into the interactive intervention model to generate reminder content corresponding to the current environment.

[0057] The state monitoring model predicts the driver's state information. Since the task scenario is relatively simple, the accuracy of the predicted driver's state information can be effectively improved. Merging the state monitoring model with the interactive intervention model increases data complexity and the difficulty of the task. However, the driver's state information output by the state monitoring model serves as redundant input to the interactive intervention model, improving its learning capabilities. This allows the interactive intervention model to not only see the predicted driver's state information but also learn other behavioral characteristics from driving environment information (such as images of the interior and / or exterior of the vehicle), thereby improving the accuracy of the output reminder content.

[0058] FIG2 is a schematic diagram of an implementation flow of a method for reminding of abnormal driving status provided by an embodiment of the present application, which is described in detail as follows:

[0059] In S201 , driving environment information, historical conversation information, and vehicle travel data are acquired.

[0060] Driving environment information can include in-vehicle and out-vehicle environment information. In-vehicle environment information can be obtained through the DMS, while out-vehicle environment information can be obtained through the ADAS. For example, in-vehicle environment information can be obtained through in-vehicle environment images, while out-vehicle environment information can be obtained through out-vehicle environment images.

[0061] In-vehicle environmental information may include at least one of the following: information about the driver's facial orientation, eye position, mouth position, or hand position. External environmental information may include one or more of the following: lane markings, pedestrians, vehicles, obstacles, and traffic signals. Vehicle information surrounding the vehicle may include information such as the distance between the vehicle and other vehicles, and the speed and acceleration of other vehicles.

[0062] Historical conversation information can include the driver's past conversations. This information can be recorded through a microphone, including conversations between the driver and other passengers, or conversations made by the driver via a phone, walkie-talkie, or other communication tools. Alternatively, historical conversation information can be sent from a cloud platform.

[0063] In a possible implementation, the driver's abnormality reminder method may further include a microphone for collecting sound information of the driver's current environment. The sound information is input into the interactive intervention model to obtain reminder content that better matches the current environment.

[0064] In S202 , based on the driving environment information, historical dialogue information, and vehicle driving data, combined with a pre-trained interactive intervention model, reminder content corresponding to the current environment is generated.

[0065] The interactive intervention model may include a model of a neural network structure. In order to improve the learning ability of the interactive intervention model, when using the interactive intervention model to generate reminder content, it may include, as shown in FIG3 :

[0066] In S301, the in-vehicle environment information is input into a pre-trained state monitoring model to obtain the driver's state information.

[0067] The in-vehicle environment images collected by the DMS can be input into a pre-trained state monitoring model to obtain the driver's state information in the current environment.

[0068] The condition monitoring model can be trained using pre-set sample data. For example, the sample data can include sample images and sample state information. The sample images are input into the condition monitoring model to be trained, and the state information corresponding to the sample images is calculated and output. The calculated state information is then compared with the standard state information. Based on the difference between the two, the parameters of the condition monitoring model are adjusted until the difference meets the pre-set requirements.

[0069] In S302 , the in-vehicle environment information, the out-vehicle environment information, the historical conversation information, the vehicle driving data, and the driver's status information are input into the interactive intervention model to generate reminder content corresponding to the current environment.

[0070] The interactive intervention model's large input data volume makes the task computationally challenging. By feeding the interactive intervention model with driver status information predicted from DMS data, redundant input is added to the model, improving its learning capabilities. This allows the model to not only predict status information but also learn other behavioral characteristics from DMS-collected imagery, such as images of the vehicle's interior, thereby increasing the accuracy of the output reminders.

[0071] When the application scenario includes a microphone, when calculating the reminder content, the driving environment information, historical conversation information, vehicle driving data, driver's status information and sound information in the current environment collected by the microphone can also be input into the pre-trained interactive intervention model to generate reminder content corresponding to the current environment.

[0072] In an embodiment of the present application, when driving environment information, historical conversation information, vehicle driving data, driver's status information, and sound information in the current environment collected by a microphone are input into a pre-trained interactive intervention model, the in-vehicle environment image and the out-vehicle environment image in the driving environment information can be flattened into two-dimensional data, and the two-dimensional data can be spliced ​​with other two-dimensional data, and the spliced ​​data can be input into the interactive intervention model. Through the attention mechanism, attention is paid to abnormal points in the input, including abnormalities in the driver's status information, accompanied by speeding due to excessive vehicle speed, or following a vehicle too closely, etc., thereby generating reminder content corresponding to the current environment, including speeding reminders, following a vehicle too closely, etc., effectively reducing the problem of overly simple reminder content, so that the driver can more effectively notice the problems caused by abnormal status.

[0073] When inputting data into the interactive intervention model, different data can be represented as a single data format. For example, the environmental image information obtained by the ADAS and DMS can be resized. For example, it can be processed into a 224×224 resolution image. The resized image can be used as input to the pre-trained CLIP ViT-L / 14 model (a neural network model) to extract image features from the environmental image information, including those of the interior and exterior environments. Downsampling can be used to extract key content from the environmental image while simultaneously reducing the data volume.

[0074] Furthermore, the pre-trained BERT-Large-Uncased neural network model (a neural network model) can be used to convert audio information converted from microphone input (including recognized text) and historical conversation information (including text from past conversations) into feature vectors. Vehicle driving data and additional data from ADAS and DMS (such as lane markings, pedestrians, and facial orientation) are used as input to a learnable embedding layer, where the interactive intervention model learns its features. All of these features are flattened into two-dimensional data and concatenated using special tags in the tagger.

[0075] For example, you can use <s>and <s>To indicate the beginning and end of a sequence. Special markers <text>and< / text> Indicates the start and end of the text embedding of the driver's voice data. Special marker and indicate the start and end of the encoded image embedding. Special markers <history> and< / history> Indicates the start and end of the history conversation embed. Special marker <other> and< / other> Indicates the start and end of the rest of the data embedding.

[0076] For example," <s> image_from_ADAS image_from_DMS <other> other_embeddings< / other> <history> dialogue_content< / history> <text> current_speech_text< / text> < / s> "is the multimodal input to the interactive intervention model. Based on the multimodal input model (i.e., the interactive intervention model), it is trained on the next token prediction task, i.e., learning to generate the next token based on the previous context. The training objective can include maximizing the log-likelihood of the token in the example.

[0077] In the embodiment of the present application, before using the interactive intervention model to calculate the reminder content, the interactive intervention model may also be trained. As shown in FIG4 , the training process may include:

[0078] In S401 , sample data is acquired, where the sample data includes driving environment information, historical conversation information, vehicle travel data, and standard content.

[0079] The sample data in the embodiments of this application is data that pre-defines the correspondence between driving information and reminder content. Driving information includes driving environment information, historical conversation information, vehicle driving data, and current environmental sound information. Different standard content can be mapped to different driving information. Generating diverse standard content based on different driving information helps to increase the diversity of reminder content. Furthermore, the reminder content is related to the conversation content, making the reminder more relevant to the driver's conversation.

[0080] In S402, the driving environment information, historical conversation information, and vehicle driving data in the sample data are input into the interactive intervention model to calculate the reminder content.

[0081] The environmental information in the sample data, including driving environment information, historical conversation information, vehicle driving data, or the sound information of the current environment in the sample data, is input into the interactive intervention model, and the reminder content generated by the interactive intervention model is calculated based on the parameters in the interactive intervention model.

[0082] In S403 , the difference between the reminder content and the standard content is determined, and the parameters of the interactive intervention model are adjusted according to the difference until the difference meets a predetermined requirement.

[0083] The alert content calculated by the interactive intervention model can be compared with the standard content to determine the accuracy of the current interactive intervention model calculation based on the calculated difference. The parameters of the interactive intervention model are adjusted based on this difference to reduce the difference to meet the predetermined requirements, thereby completing the training of the interactive intervention model.

[0084] Among them, when determining the difference between the reminder content calculated by the interactive intervention model and the standard content, the size of the difference can be determined based on the size of the word shift distance by calculating the word shift distance between the standard content and the calculated reminder content. That is, the larger the word shift distance, the greater the difference between the standard content and the reminder content, and the smaller the word shift distance, the smaller the difference between the standard content and the reminder content. Among them, when determining the word shift distance, the word vector in the standard content and the word vector in the reminder content can be determined, and the word shift distance can be determined based on the word vectors of the two. By determining the difference between the standard content and the reminder content through the word shift distance, the semantic relevance between the manually annotated answers and the generated reminder content can be well considered, and it will not be limited to the specific expression of the sentence, thereby improving the diversity of the reminder content.

[0085] In S203, a voice reminder is generated according to the reminder content.

[0086] The reminder content can be converted into voice through the voice module, and the voice content can be played through the speaker to provide the driver with diversified voice reminders.

[0087] This application obtains driving environment information, historical conversation information and vehicle driving data, and performs calculations based on the driving environment information, historical conversation information and vehicle driving data, combined with an interactive intervention model, to obtain reminder content corresponding to the current environment, and generates voice reminders based on the reminder content, thereby associating the voice reminders with the vehicle driving data and driving environment information in the current environment, making the voice reminders more accurate and reliable, and based on the generation of historical conversation information, it can effectively improve the service quality of voice reminders.

[0088] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0089] FIG5 is a schematic diagram of a driving state abnormality reminder device provided by an embodiment of the present application. As shown in FIG5 , the device includes:

[0090] The information acquisition unit 501 is used to acquire driving environment information, historical conversation information and vehicle travel data.

[0091] The reminder content generating unit 502 is configured to generate reminder content corresponding to the current environment based on the driving environment information, historical conversation information, and vehicle driving data in combination with a pre-trained interactive intervention model.

[0092] The voice reminder unit 503 is configured to generate a voice reminder according to the reminder content.

[0093] The device for notifying of abnormality in driving state shown in FIG5 corresponds to the method for notifying of abnormality in driving state shown in FIG2 .

[0094] Figure 6 is a schematic diagram of a vehicle provided in one embodiment of the present application. As shown in Figure 6 , vehicle 6 in this embodiment includes a processor 60, a memory 61, and a computer program 62 stored in memory 61 and executable on processor 60, such as a program for notifying drivers of abnormal driving conditions. When processor 60 executes computer program 62, it implements the steps described in each of the aforementioned embodiments of the method for notifying drivers of abnormal driving conditions. Alternatively, when processor 60 executes computer program 62, it implements the functions of each module / unit described in each of the aforementioned device embodiments.

[0095] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the vehicle 6.

[0096] The vehicle may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that FIG6 is merely an example of vehicle 6 and does not limit vehicle 6 . Vehicle 6 may include more or fewer components than shown, or a combination of certain components, or different components. For example, the vehicle may also include input and output devices, network access devices, buses, and the like.

[0097] The processor 60 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0098] The memory 61 can be an internal storage unit of the vehicle 6, such as a hard drive or memory in the vehicle 6. Alternatively, the memory 61 can be an external storage device in the vehicle 6, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 61 can include both an internal storage unit and an external storage device in the vehicle 6. The memory 61 is used to store the computer program and other programs and data required by the vehicle. The memory 61 can also be used to temporarily store data that has been output or is about to be output.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0100] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0104] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments through hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0106] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.< / s> < / s>

Claims

1. A method for reminding an abnormal driving state, characterized in that: The method comprises: Obtain driving environment information, historical conversation information and vehicle driving data; Generate reminder content corresponding to the current environment based on the driving environment information, historical conversation information and vehicle driving data, combined with a pre-trained interactive intervention model; Generate a voice reminder according to the reminder content.

2. The method according to claim 1, characterized in that The driving environment information includes in-vehicle environment information and out-vehicle environment information; According to the driving environment information, the dialogue information and the vehicle driving data, combined with the pre-trained interactive intervention model, a reminder content corresponding to the current environment is generated, including: Inputting the in-vehicle environment information into a pre-trained state monitoring model to obtain the driver's state information; The in-vehicle environment information, the out-vehicle environment information, the historical conversation information, the vehicle driving data and the driver's status information are input into the interactive intervention model to generate reminder content corresponding to the current environment.

3. The method according to claim 1 or 2, characterized in that Obtain driving environment information, including: Acquire at least one of lane line information, pedestrian information, surrounding vehicle information, obstacle information and traffic signal information of the scene where the vehicle is located through an automatic data acquisition system; At least one of the driver's facial orientation information, eye information, mouth information, and hand information is obtained through the driver monitoring system.

4. The method according to claim 2, characterized in that Before generating reminder content corresponding to the current environment based on the driving environment information, historical dialogue information and vehicle driving data in combination with a pre-trained interactive intervention model, the method further includes: Get the sound information collected by the microphone in the current scene; According to the driving environment information, historical dialogue information and vehicle driving data, combined with the pre-trained interactive intervention model, a reminder content corresponding to the current environment is generated, including: Inputting the in-vehicle environment information into a pre-trained state monitoring model to obtain the driver's state information; The driving environment information, historical conversation information, vehicle travel data, driver's status information and the sound information are input into a pre-trained interactive intervention model to generate reminder content corresponding to the current environment.

5. The method according to claim 2, characterized in that Inputting the driving environment information, historical dialogue information, vehicle travel data, driver status information and the sound information into a pre-trained interactive intervention model, including: Extracting image features from an image by a feature extraction model, wherein the image includes an in-vehicle environment image and an out-vehicle environment image in the driving environment information; Recognize the text information in the historical conversation information and the sound information through a speech recognition model, and extract feature vectors in the text information according to a pre-trained vector extraction model; The image features, the feature vectors, the driver's state information and the vehicle driving data are connected according to a predetermined marker, and the data connected by the marker is input into the interactive intervention model.

6. The method according to claim 1, characterized in that Before generating reminder content corresponding to the current environment based on the driving environment information, historical dialogue information and vehicle driving data in combination with a pre-trained interactive intervention model, the method further includes: Acquiring sample data, wherein the sample data includes driving environment information, historical conversation information, vehicle travel data, and standard content; Inputting the driving environment information, historical conversation information, and vehicle driving data in the sample data into the interactive intervention model to calculate the reminder content; The difference between the reminder content and the standard content is determined, and the parameters of the interactive intervention model are adjusted according to the difference until the difference meets the predetermined requirement.

7. The method according to claim 6, characterized in that Determining the difference between the reminder content and the standard content includes: The word shift distance between the reminder content and the standard content is calculated, and the difference between the reminder content and the standard content is determined according to the word shift distance.

8. A driving status abnormality reminder device, characterized in that: The device comprises: An information acquisition unit, used to acquire driving environment information, historical conversation information and vehicle driving data; A reminder content generating unit, configured to generate reminder content corresponding to the current environment based on the driving environment information, historical dialogue information and vehicle driving data, combined with a pre-trained interactive intervention model; The voice reminder unit is used to generate a voice reminder according to the reminder content.

9. A car comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images