Driving state prediction method and device, medium, equipment and program product

Abstract

The invention discloses a driving state prediction method and device, a medium, equipment and a program product, and the method comprises the steps: obtaining a plurality of groups of state data, including first state data collected in a plurality of time windows during the driving of a driver and second state data collected at a historical moment before the driver starts driving; performing feature extraction on the multiple groups of state data to obtain initial features corresponding to the multiple groups of state data; performing time sequence processing on the initial features corresponding to the multiple groups of state data to obtain time sequence features corresponding to the multiple groups of state data; based on the time sequence characteristics corresponding to the multiple sets of state data, weights corresponding to the multiple sets of state data are determined; based on the weights corresponding to the multiple groups of state data, performing weighted fusion on the time sequence features corresponding to the multiple groups of state data to obtain fusion features; and predicting the driving state of the driver in a time window after the plurality of time windows based on the fused features.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a driving state prediction method and device, medium, equipment and program product. Background Art

[0002] At present, cars, airplanes and other means of transportation have become indispensable tools for people to travel. While these means of transportation bring convenience to people's travel, they also cause a series of traffic safety problems. Some of these traffic safety problems are caused by drivers driving in bad conditions (such as fatigue, distraction). Therefore, it is necessary to predict the driver's state to improve traffic safety. The current related technologies usually predict the driver's state based on the driver's state data during driving. However, there are many factors that affect the driver's state. The above method is difficult to fully obtain the factors that affect the driver's state. Therefore, the prediction accuracy is low. Summary of the invention

[0003] In a first aspect, an embodiment of the present application provides a method for predicting a driving state, the method comprising: acquiring multiple groups of state data, the multiple groups of state data comprising first state data within multiple time windows collected during a driver's driving and second state data collected at a historical moment before the driver starts driving; performing feature extraction on the multiple groups of state data to obtain initial features corresponding to the multiple groups of state data; performing time series processing on the initial features corresponding to the multiple groups of state data to obtain time series features corresponding to the multiple groups of state data; determining weights corresponding to the multiple groups of state data based on the time series features corresponding to the multiple groups of state data; performing weighted fusion on the time series features corresponding to the multiple groups of state data based on the weights corresponding to the multiple groups of state data to obtain fused features; and predicting the driver's driving state in a time window after the multiple time windows based on the fused features.

[0004] In a second aspect, an embodiment of the present application provides a driving state prediction device, the device comprising: an acquisition module, used to acquire multiple groups of state data, the multiple groups of state data comprising first state data within multiple time windows collected during the driver's driving and second state data collected at a historical moment before the driver starts driving; a first feature extraction module, used to perform feature extraction on the multiple groups of state data respectively to obtain initial features corresponding to the multiple groups of state data respectively; a second feature extraction module, used to perform time series processing on the initial features corresponding to the multiple groups of state data respectively to obtain time series features corresponding to the multiple groups of state data respectively; a weight determination module, used to determine the weights corresponding to the multiple groups of state data respectively based on the time series features corresponding to the multiple groups of state data respectively; a feature fusion module, used to perform weighted fusion on the time series features corresponding to the multiple groups of state data respectively based on the weights corresponding to the multiple groups of state data respectively to obtain fused features; a state prediction module, used to predict the driving state of the driver in a time window after the multiple time windows based on the fused features.

[0005] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the present application.

[0006] In a fourth aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any embodiment of the present application when executing the computer program.

[0007] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any embodiment of the present application.

[0008] The embodiment of the present application uses first state data of multiple time windows during the driver's driving and second state data obtained before the driver starts driving as input data for predicting the driver's state. By performing time series processing on initial features extracted from multiple groups of state data, it is possible to mine implicit time series features between each group of state data. The weights corresponding to each group of state data are determined based on the time series features corresponding to each group of state data, which can reflect the proportion of influence of each group of state data on the driving state. Based on the above weights, weighted fusion is performed on the time series features corresponding to each group of state data, and the driving state is predicted based on the fused features. The state data before driving and the state data during driving can be effectively fused, so that various factors that may affect the driving state can be more comprehensively considered when predicting the state, thereby improving the prediction accuracy.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into the specification and constitute a part of the present application. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solutions of the present application.

[0011] Figure 1 It is a flowchart of the method for predicting driving status according to an embodiment of the present application.

[0012] Figure 2 It is a structural diagram of the state prediction model of an embodiment of the present application.

[0013] Figure 3 It is a flowchart of the training method of the state prediction model of an embodiment of the present application.

[0014] Figure 4 It is a block diagram of a driving state prediction device according to an embodiment of the present application.

[0015] Figure 5 It is a schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0017] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a variety of.

[0018] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0020] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0021] The driver's driving state has a great impact on driving safety, so it is necessary to predict the driver's state. Related technologies usually predict the driver's state based on the driver's state data during driving, for example, by collecting video and physical data of the driver during driving to predict the driver's state. However, there are many factors that affect the driver's state. For example, the driver's physiological indicators on the day of driving (such as sleep quality, alcohol or drug intake, etc.) can reflect the driver's physiological condition on the day of driving. The physical examination records and skill assessment data obtained by the driver during the pre-job status assessment can reflect the driver's physiological condition, psychological condition, and skill proficiency. Information. All of the above information may affect the driver's driving state. Related technologies only consider the data during driving, and cannot comprehensively predict the driver's driving state, resulting in low accuracy of state prediction.

[0022] Based on this, the present application provides a method for predicting driving status, see Figure 1 , the method comprising:

[0023] Step S11: acquiring multiple sets of status data, wherein the multiple sets of status data include first status data in multiple time windows collected during the driver's driving and second status data collected at a historical moment before the driver starts driving;

[0024] Step S12: extracting features from the multiple groups of state data respectively to obtain initial features corresponding to the multiple groups of state data respectively;

[0025] Step S13: performing time series processing on the initial features respectively corresponding to the multiple sets of state data to obtain the time series features respectively corresponding to the multiple sets of state data;

[0026] Step S14: determining the weights corresponding to the multiple groups of state data respectively based on the time series features corresponding to the multiple groups of state data respectively;

[0027] Step S15: Based on the weights corresponding to the multiple groups of state data, respectively, weighted fusion is performed on the time series features corresponding to the multiple groups of state data to obtain fusion features;

[0028] Step S16: predicting the driving state of the driver in a time window after the multiple time windows based on the fusion features.

[0029] In step S11, the state data includes but is not limited to the driver's vital data, image data and / or voice data. The state data can be collected through various sensors. For example, the driver's vital data can be collected through a wearable device, the driver's image data can be collected through an image sensor in the cockpit, and the driver's voice data can be collected through a sound sensor in the cockpit. Among them, the vital data includes but is not limited to heart rate, blood pressure, blood oxygen, body temperature, etc., which can be used to analyze the driver's physiological health; image data can be used to determine the driver's expression and movement, so as to analyze whether the driver is fatigued driving, distracted driving, etc. during driving; voice data can be used to determine the driver's tone, speaking speed, tone, reaction to other sounds (such as the voice of the in-vehicle voice assistant or other passengers) and other information, so as to analyze the driver's emotional state and reaction speed.

[0030] The present application takes into account that the driver's driving state is not only related to the state data during driving, but may also be related to the state data before starting driving. Therefore, multiple sets of state data can be obtained, including first state data during the driver's driving and second state data before the driver starts driving.

[0031] Among them, the first state data may include time-series numerical data, which can reflect the continuous state of the driver within a period of time. The first state data obtained by continuous monitoring of the driver can be divided into multiple time windows. For example, it can be equally divided into N time windows according to the preset time length t, that is, the time length of the first state data in each time window is t, N is a positive integer, and t is a positive number. The value of t can be determined according to the data length required for data correlation analysis and the demand for computing resources. If the value of t is too small, the amount of information in the first state data in each time window is too small, and it may be difficult to obtain enough information from the first state data in each time window to predict the state; if the value of t is too large, it means that more data needs to be processed, which will increase the demand for computing resources. At the same time, the correlation of the data in a single time window will also be reduced, thereby introducing unnecessary noise and reducing the accuracy of subsequent state predictions.

[0032] The second state data may include instantaneous data, which may be collected at a certain historical moment before the driver starts driving. For example, when the driver has completed the pre-driving preparations and is about to start driving, the second state data may be collected once, and the second state data may reflect the instantaneous state of the driver at the moment of data collection. Optionally, the instantaneous state data is data collected by the driver before he starts driving on the day of driving.

[0033] In addition, the driver can also be state-evaluated in a certain state-evaluation cycle, and the evaluation content can include but is not limited to the driver's physical data, skill assessment data and / or psychological evaluation data. The second state data may include the state evaluation data obtained when the driver performs the most recent state evaluation before starting to drive. Optionally, the state evaluation data may be tabular data, and the tabular data may include discrete data and / or instantaneous numerical data. For example, the psychological evaluation data may include attribute items for evaluating the driver's anxiety level, and the attribute values ​​corresponding to the attribute items may include options such as "not in compliance", "compliant" and "very compliant", and each option is discrete data. For another example, the physical data may include attribute items corresponding to heart rate and attribute items corresponding to blood pressure, and the attribute values ​​corresponding to the above two attribute items are all instantaneous numerical data.

[0034] In order to comprehensively extract the factors that affect the driving state, each set of state data may include data of multiple dimensions. For example, the first state data includes the driver's physical sign data, image data and voice data during driving, and the second state data includes the driver's physical sign data and psychological evaluation data before starting to drive. In the case where the second state data includes the state evaluation data obtained when the driver performs the most recent state evaluation before starting to drive and the driver's instantaneous state data before starting to drive on the day of driving, the state evaluation data and the instantaneous state data may both include data of multiple dimensions. For example, the state evaluation data includes multiple state evaluation forms, such as an evaluation form for physical sign data and an evaluation form for psychological state, and each state evaluation form is data of one dimension; the instantaneous state data may include multiple physical sign data of the driver at a certain moment, and each physical sign data is data of one dimension.

[0035] In step S12, feature extraction may be performed on each group of status data to obtain initial features corresponding to each group of status data.

[0036] In some embodiments, each group of state data may be subjected to feature vectorization processing to obtain a feature vector corresponding to each group of state data, and then feature extraction may be performed based on the feature vector corresponding to each group of state data to obtain an initial feature corresponding to each group of state data.

[0037] Multiple groups of state data are heterogeneous in nature. For example, in the second state data, the state evaluation data belongs to table data, which may include discrete data and instantaneous numerical data. The instantaneous state data of the driver before starting to drive on the day of driving is instantaneous numerical data, and the first state data in each time window is time series numerical data. For the above heterogeneous data, feature vectorization can be performed in a way that matches its data type.

[0038] Specifically, if the data of a certain dimension in a certain set of state data is discrete data, the data of this dimension can be embedded to obtain the feature vector corresponding to this dimension. The embedding process can be implemented based on the word2vec or BERT model. If the data of a certain dimension in a certain set of state data is instantaneous numerical data, the data of this dimension can be linearly mapped to obtain the feature vector corresponding to this dimension. If the data of a certain dimension in a certain set of state data is time series numerical data, the data of this dimension can be nonlinearly mapped to obtain the feature vector corresponding to this dimension. For ease of processing, the dimensions of each feature vector obtained can be the same, assuming that they are all d-dimensional feature vectors.

[0039] Furthermore, for any set of status data, at least one dimension of data included in the set of status data may include multiple attribute items and their corresponding attribute values. At this time, the attribute values ​​corresponding to each attribute item can be feature vectorized to obtain a d-dimensional feature vector corresponding to each attribute value, and then the d-dimensional feature vectors corresponding to each attribute value are aggregated to obtain a feature vector corresponding to the dimension. Optionally, the above aggregation can be implemented based on a Transformer network. For example, the status evaluation data included in the second status data is tabular data, and the tabular data includes attribute items corresponding to the anxiety level and attribute items corresponding to the heart rate. The attribute value corresponding to the attribute item of the anxiety level can be converted into a d-dimensional feature vector, and the attribute value corresponding to the attribute item of the heart rate can be converted into a d-dimensional feature vector. Then, the two d-dimensional feature vectors are aggregated through the Transformer to obtain the feature vector corresponding to the tabular data.

[0040] In some embodiments, an attention mechanism may be used to extract features from each group of state data. Specifically, for any group of state data, after obtaining the feature vectors corresponding to multiple dimensions of the group of state data, the feature vectors corresponding to the multiple dimensions may be linearly processed to obtain linear features corresponding to the multiple dimensions, and the linear features corresponding to the multiple dimensions may be normalized to obtain weights corresponding to the multiple dimensions, and based on the weights corresponding to the multiple dimensions, the feature vectors corresponding to the multiple dimensions may be weighted to obtain the initial features corresponding to the group of state data. Assume that the feature vector corresponding to the i-th dimension is denoted as f i , then the weight w corresponding to the i-th dimension i It can be expressed as:

[0041] w i =softmax(Linear(f i ))

[0042] Among them, Linear represents linear processing, and softmax represents normalization processing. The initial features corresponding to this group of state data can be expressed as:

[0043]

[0044] By adopting the attention mechanism, it is possible to automatically identify and focus on information that is critical to the driver's state judgment, reduce noise interference, and improve prediction accuracy and robustness.

[0045] In step S13, the initial features corresponding to the multiple groups of state data may be processed in time series to obtain the time series features corresponding to the multiple groups of state data. In some embodiments, the initial features corresponding to each group of state data may be processed in time series by a bidirectional long short-term memory network (Bi-LSTM). Bi-LSTM can effectively capture the dependencies in long time series, thereby reflecting the impact of each group of state data on the driving state.

[0046] In step S14, the weights corresponding to the multiple groups of state data can be determined based on the time series features corresponding to the multiple groups of state data. The weight corresponding to each group of state data can reflect the influence of the group of state data on the driving state of the driver. In some embodiments, the weight corresponding to each group of state data can be obtained based on the attention mechanism.

[0047] Specifically, the time series features corresponding to each group of state data can be linearly processed to obtain the linear features corresponding to the group of state data, and the linear features corresponding to each group of state data can be normalized to obtain the weight corresponding to the group of state data. Assume that the time series features corresponding to the jth group of state data are denoted as h j , then the weight α corresponding to the jth group of state data j It can be written as:

[0048] α j =softmax(Linear(h j ))

[0049] Among them, Linear represents linear processing and softmax represents normalization processing. By adopting the attention mechanism, it is possible to automatically identify and focus on information that is critical to the driver's state judgment, reduce noise interference, and improve prediction accuracy and robustness.

[0050] In step S15, the weights corresponding to the multiple sets of state data can be used to perform weighted fusion on the time series features corresponding to the multiple sets of state data to obtain a fused feature. Continuing with the previous example, the fused feature r can be recorded as:

[0051]

[0052] In step S16, the driving state of the driver in the time window after the multiple time windows can be predicted based on the fusion features. For example, assuming that the first state data of the driver in N time windows and the second state data of the driver are obtained in the previous step, the category of the driver's driving state in the N+1th time window can be predicted based on the above state data in this step, such as the normal category, the distracted driving category, or the fatigue driving category.

[0053] In some embodiments, the method of the present application can be performed by a pre-trained state prediction model. Multiple sets of state data can be input into the state prediction model, and the state prediction model can be used to perform feature extraction, time series processing, weight determination, time series feature fusion, and state prediction on the multiple sets of state data.

[0054] See also Figure 2 In some embodiments, the state prediction model may include:

[0055] A first attention module is used to extract features from the multiple groups of state data respectively by using an attention mechanism to obtain initial features corresponding to the multiple groups of state data respectively;

[0056] A time series feature extraction module, used for performing time series processing on the initial features respectively corresponding to the multiple groups of state data to obtain the time series features respectively corresponding to the multiple groups of state data;

[0057] A second attention module is used to adopt an attention mechanism and determine the weights corresponding to the multiple groups of state data respectively based on the time series features corresponding to the multiple groups of state data respectively;

[0058] The state prediction module is used to perform weighted fusion of the time series features corresponding to the multiple groups of state data based on the weights corresponding to the multiple groups of state data, obtain fusion features, and predict the driving state of the driver in a time window after the multiple time windows based on the fusion features.

[0059] In some embodiments, see Figure 3 , the state prediction model can be trained based on the following methods:

[0060] Step S21: Acquire multiple sets of sample state data, wherein the multiple sets of sample state data include first sample state data in multiple time windows collected during the driver's driving and second sample state data collected at a historical moment before the driver starts driving;

[0061] The first sample status data in this embodiment corresponds to the first status data in the aforementioned embodiment, and the second sample status data corresponds to the second status data in the aforementioned embodiment. The specific details such as the types of multiple groups of sample status data and the method of obtaining multiple groups of sample status data can refer to the embodiment of multiple groups of status data in the aforementioned embodiment, and will not be repeated here.

[0062] Step S22: Acquire an abnormal state label of the first sample state data, where the abnormal state label is used to indicate an abnormal type of the driving state of the first sample state data and a target time window where the abnormal type is located;

[0063] Abnormal state labels can be generated by manual labeling. The abnormal type of driving state is used to indicate which abnormal driving state the driver has, such as distracted driving type or fatigue driving type. The target time window where the abnormal type is located is used to indicate in which time window the abnormal type occurs. Assuming that the driver is detected to be distracted in the kth time window (such as answering or making a phone call while driving), the target time window where the abnormal type is located is k.

[0064] Step S23: obtaining the predicted driving state of the driver in the time window after the multiple time windows obtained by the state prediction model based on the first sample state data in each time window before the target time window and the second sample state data;

[0065] Continuing with the previous example, assuming that an abnormal driving state is detected in the kth time window, the first sample state data from the 1st time window to the k-1th time window can be obtained, and the second sample state data can also be obtained. The state prediction model can obtain the predicted driving state in the k+1th time window based on these data.

[0066] Step S24: training the state prediction model based on the predicted driving state and the abnormal type of the driving state included in the abnormal state label.

[0067] Based on the predicted driving state and the abnormal type of the driving state included in the abnormal state label, the prediction loss of the state prediction model can be determined, so that the state prediction model can be adjusted based on the prediction loss. The structure and specific principle of the state prediction model can be referred to the above embodiment, and will not be repeated here.

[0068] In some embodiments, a cross entropy loss function may be established based on the predicted driving state and the abnormal type of the driving state included in the abnormal state label, and the state prediction model may be trained based on the cross entropy loss function. The cross entropy loss may be expressed as:

[0069]

[0070] Where M is the total number of driving states, including M-1 abnormal states and 1 normal state. i is the true value label, p i is the probability corresponding to the i-th driving state in the probability distribution output by the state prediction model.

[0071] The implementation details of the present application are illustrated below in conjunction with a specific embodiment. In this embodiment, it is assumed that the driver is a person whose profession is driving, such as a freight driver, a bus driver, etc. The driving state of the driver is closely related to the safety of personnel and property. Therefore, the present application proposes a method for predicting the driving state, which monitors and warns the driver's abnormal driving state based on data fusion. The present application can timely and comprehensively identify the driver's abnormal state by integrating and analyzing multi-source data, and then warn of possible risks. This helps to protect the life safety of drivers and passengers and reduce the loss of life and property caused by the driver's abnormal state. Among them, the multi-source data includes the state data collected by the driver during on-the-job driving, also known as on-the-job data (i.e., the first state data in the aforementioned embodiment); the state data of the driver on the day of taking up the post (i.e., the instantaneous state data of the driver before starting to drive on the day of driving in the aforementioned embodiment), also known as out-of-post data; and the state data collected when the driver is performing a state evaluation before officially taking up the post, which can be taken as the state data collected during the most recent state evaluation (i.e., the state evaluation data obtained when the driver in the aforementioned embodiment performs the most recent state evaluation before starting to drive), also known as pre-job data. Both the pre-job data and the post-job data are the second state data in the aforementioned embodiment.

[0072] At present, the technology mainly focuses on the status monitoring of drivers during their work, that is, the abnormal behaviors or physiological abnormalities exhibited by drivers during vehicle driving. However, in actual applications, the driver's pre-job data and out-of-job data may also provide effective information about abnormal conditions. Pre-job data includes physical examination records, skill assessment data, etc., which can be used to analyze the driver's basic health and skill adaptability; physiological indicators on the day of work, such as sleep quality, alcohol or drug intake, etc., may also reflect the driver's psychological or physiological state, which has a potential impact on driving safety.

[0073] Due to the lack of pre-job data and post-job data in the research data, relevant technologies have also failed to conduct effective correlation analysis on the data in the above multiple stages. Specifically, there is a lack of effective correlation mechanism for the data in the three stages of pre-job, on-job and post-job of drivers, making it difficult to form a comprehensive abnormal status warning.

[0074] In this process, how to highlight key data in multi-stage data association is a key challenge. For example, fatigue indicators and skill assessment results shown in pre-job physical examinations may become triggers for abnormal warnings under certain thresholds; during work, facial expressions, steering wheel operation habits and other features can be used to identify abnormal behavior in real time; and sleep quality and heart rate changes when leaving work can be used as indicators of potential risks while on the job. Existing technologies lack the screening and association of key data across stages.

[0075] Based on this, this application monitors and warns the driver's abnormal driving status based on data fusion. This application specifically includes the following steps:

[0076] Training phase

[0077] (1) Collect a large amount of multi-source data on drivers, including data from the pre-job, off-job, and on-job stages, as well as abnormal status labels (including abnormal type and the time point when the abnormality occurs). Pre-job data refers to the driver's status assessment data before officially taking up the job, such as physical examination data, psychological assessment data, etc.; off-job data refers to the driver's inspection data on the day of work, such as heart rate, blood pressure, blood oxygen, etc.; on-job data refers to the driver's monitoring data during work, including blood pressure, heart rate, etc. that are continuously monitored after taking up the job.

[0078] (2) The data in the above three stages are heterogeneous in nature: physical examination data and psychological assessment data are tabular data, which include discrete data (such as the "anxiety" item in the psychological scale is divided into: not meeting, meeting, and very meeting, and its options are discrete data) and instantaneous numerical data (such as the "weight" item in the physical examination data, whose value is an instantaneous value); heart rate, blood pressure, blood oxygen, etc. are also instantaneous numerical data; blood pressure, heart rate, etc. data that are continuously monitored after taking up the post belong to time series numerical data.

[0079] Therefore, for the heterogeneous data in the above three stages, the following scheme can be used to extract their feature vectors: for discrete data, map it into a d-dimensional feature vector. The mapping method can be pre-trained word feature vectors such as word2vec, or feature vectors output by pre-trained models such as BERT; for numerical data, use a linear mapping to map it into a d-dimensional feature vector; for tabular data, use the above-mentioned processing methods for discrete data and numerical data to convert all values ​​in the table into d-dimensional feature vectors, and then use the Transformer network to fuse these d-dimensional feature vectors to obtain the fused d-dimensional feature vector; for time series numerical data, divide it into N groups of time windows with a length of t, and use a nonlinear mapping to map the data in each window into a d-dimensional feature vector.

[0080] For the physical examination form and psychological scale in the pre-job data, Transformer is used to aggregate all the attributes in the physical examination form to obtain and For the skill test report, BERT is used to obtain

[0081] For the blood pressure, heart rate, alcohol intake, and sleep quality data in the post data, they are mapped as

[0082] For the blood pressure data and heart rate data in the on-the-job data, they are divided into N groups of time windows with a length of t, and the data in each window is mapped into a d-dimensional vector to obtain and

[0083] (3) Train a data fusion model based on dual attention mechanism.

[0084] First, construct training data (x, y) from the data collected in step (1) at the pre-job, post-job, and on-job stages and the abnormal state labels (including the abnormal type and the time node when the abnormal state occurs). Assuming that the time node when the abnormal state occurs is in the nth time window divided in step (2), then x in the training data includes: the 0th to n-1th time windows of the pre-job data, post-job data, and on-job data, and y is the abnormal type. Continuing with the previous example, x = [f 1 ,f 2 ,f 3 ,f 4 ,f 5 ,f 6 ,f 7 ,f 8 ,f 9 ],in, and

[0085] Next, for the pre-job data in x, the attention mechanism is used to aggregate the multi-source data to obtain The feature vector, where

[0086] [w 1 ,w 2 ,w 3 ]=softmax(Linear([f 1 ,f 2 ,f 3 ]))

[0087]

[0088] The same method is used for the out-of-post data in x to obtain the out-of-post information Eigenvectors:

[0089] [w 4 ,w 5 ,w 6 ,w 7 ]=softmax(Linear([f 4 ,f 5 ,f 6 ,f 7 ]))

[0090]

[0091] For each time window of on-the-job data in x, the attention mechanism is used to aggregate the multi-source data in the time window to obtain a d-dimensional feature vector containing the on-the-job information of the current time window. Therefore, for the 0th to n-1th time windows of on-the-job data in x, a total of n d-dimensional feature vectors are obtained, denoted as For the t-th time window, the aggregation formula is:

[0092] [w 8(t) ,w 9(t) ]=softmax(Linear([f 8(t) ,f 9(t) ]))

[0093]

[0094] In this stage, the attention mechanism is used to highlight the key data in different stages.

[0095] There is an obvious time relationship between the three stages of pre-job, post-job, and on-job, that is, pre-job is earlier than post-job and earlier than on-job. This application uses Bi-LSTM to construct this relationship. Specifically, the d-dimensional feature vector extracted from the pre-job data and the d-dimensional feature vector extracted from the post-job data will be spliced ​​before the n d-dimensional feature vectors extracted from the on-job data to obtain n+2 d-dimensional feature vectors. Then, the n+2 d-dimensional feature vectors are input to the Bi-LSTM to calculate the hidden state h in the above hidden state i This is the time series feature in the previous embodiment. Next, the attention mechanism is used to capture n+2 h i The relationship between (i.e., the fusion features in the aforementioned embodiment):

[0096]

[0097] A classifier (i.e., the state prediction module in the aforementioned embodiment) is used to calculate the probability distribution p through r, and the abnormal category corresponding to the maximum probability in the probability distribution is the driver abnormal state category predicted for the next time window. In the above process, LSTM constructs the temporal relationship of the data in the three stages of pre-job, on-job, and off-job, and the attention mechanism captures the correlation between different moments in the three stages.

[0098] Finally, the model is trained using the following loss function:

[0099]

[0100] Where M is the total number of driver states, including M-1 abnormal states and 1 normal state. i is the true value label, p i The probability corresponding to the i-th state in the model output probability distribution.

[0101] Reasoning Phase

[0102] The data from the pre-job, on-job and off-job stages are input into the trained state prediction model, and the state prediction model is used to predict the driver's driving state in the next time window.

[0103] See also Figure 4 The present application also provides a driving state prediction device, the device comprising:

[0104] An acquisition module 101, configured to acquire multiple sets of state data, the multiple sets of state data including first state data in multiple time windows acquired during a driver's driving and second state data acquired at a historical moment before the driver starts driving;

[0105] A first feature extraction module 102 is used to extract features from the multiple groups of state data respectively to obtain initial features corresponding to the multiple groups of state data respectively;

[0106] A second feature extraction module 103 is used to perform time series processing on the initial features respectively corresponding to the multiple sets of state data to obtain time series features respectively corresponding to the multiple sets of state data;

[0107] A weight determination module 104, configured to determine weights corresponding to the plurality of groups of state data respectively based on time series features corresponding to the plurality of groups of state data respectively;

[0108] A feature fusion module 105 is used to perform weighted fusion on the time series features corresponding to the multiple groups of state data based on the weights corresponding to the multiple groups of state data to obtain fused features;

[0109] The state prediction module 106 is used to predict the driving state of the driver in a time window after the multiple time windows based on the fusion features.

[0110] The functions or modules included in the device provided in the present application can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0111] An embodiment of the present application further provides a computer device, which comprises at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the foregoing embodiments is implemented.

[0112] Figure 5 A more specific hardware structure diagram of a computer device provided in an embodiment of the present application is shown, and the device may include: a processor 201, a memory 202, an input / output interface 203, a communication interface 204, and a bus 205. The processor 201, the memory 202, the input / output interface 203, and the communication interface 204 are connected to each other in communication within the device through the bus 205.

[0113] The processor 201 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided in the embodiments of the present application. The processor 201 can also include a graphics card, which can be a Nvidia titan X graphics card or a 1080Ti graphics card, etc.

[0114] The memory 202 can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of the present application is implemented by software or firmware, the relevant program code is stored in the memory 202 and is called and executed by the processor 201.

[0115] The input / output interface 203 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0116] The communication interface 204 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0117] The bus 205 comprises a pathway for transmitting information between the various components of the device (eg, the processor 201 , the memory 202 , the input / output interface 203 , and the communication interface 204 ).

[0118] It should be noted that, although the above device only shows the processor 201, the memory 202, the input / output interface 203, the communication interface 204 and the bus 205, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiment of the present application, and does not necessarily include all the components shown in the figure.

[0119] An embodiment of the present application provides a computer program product, including a computer program, which implements the method described in any embodiment of the present application when executed by a processor.

[0120] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the foregoing embodiments.

[0121] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0122] Each embodiment in the present application is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment scheme of the present application. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0123] The above is only a specific implementation of the embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present application. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present application.

Claims

1. A method for predicting a driving state, characterized in that: The method comprises: Acquire multiple sets of state data, the multiple sets of state data including first state data in multiple time windows collected during the driver's driving and second state data collected at a historical moment before the driver starts driving; Extracting features from the multiple groups of state data respectively to obtain initial features corresponding to the multiple groups of state data respectively; Performing time series processing on the initial features respectively corresponding to the multiple groups of state data to obtain the time series features respectively corresponding to the multiple groups of state data; Determine the weights corresponding to the multiple groups of state data respectively based on the time series features corresponding to the multiple groups of state data respectively; Based on the weights respectively corresponding to the multiple groups of state data, weighted fusion is performed on the time series features respectively corresponding to the multiple groups of state data to obtain fused features; The driving state of the driver in a time window after the multiple time windows is predicted based on the fused features.

2. The method according to claim 1, characterized in that Each set of status data includes data of multiple dimensions; the initial features corresponding to any set of status data are obtained based on the following method: Performing feature vectorization processing on data of multiple dimensions included in the group of status data to obtain feature vectors corresponding to the multiple dimensions respectively; Performing linear processing on the feature vectors corresponding to the multiple dimensions respectively to obtain linear features corresponding to the multiple dimensions respectively; Normalizing the linear features corresponding to the multiple dimensions to obtain weights corresponding to the multiple dimensions; Based on the weights respectively corresponding to the multiple dimensions, the feature vectors respectively corresponding to the multiple dimensions are weighted to obtain initial features corresponding to the group of state data.

3. The method according to claim 2, characterized in that The data of multiple dimensions in any group of state data include at least one of discrete data, instantaneous numerical data and time series numerical data; the feature vectorization processing is performed on the data of multiple dimensions included in the group of state data to obtain feature vectors corresponding to the multiple dimensions respectively, including: If the data of any dimension is discrete data, embed the data of that dimension to obtain the feature vector corresponding to that dimension; If the data of any dimension is instantaneous numerical data, linear mapping is performed on the data of that dimension to obtain the feature vector corresponding to that dimension; If the data in any dimension is time series numerical data, nonlinear mapping is performed on the data in that dimension to obtain the feature vector corresponding to that dimension.

4. The method according to claim 1, characterized in that: The second state data includes: The state evaluation data obtained when the driver performs the most recent state evaluation before starting to drive; and The instantaneous state data of the driver before starting driving on the driving day.

5. The method according to claim 1, characterized in that The determining, based on the time series features respectively corresponding to the multiple groups of state data, the weights respectively corresponding to the multiple groups of state data comprises: Performing linear processing on the time series features corresponding to each group of state data to obtain the linear features corresponding to the group of state data; The linear features corresponding to each group of state data are normalized to obtain the weight corresponding to the group of state data.

6. The method according to claim 1, characterized in that The method is performed by a pre-trained state prediction model; the state prediction model includes: A first attention module is used to extract features from the multiple groups of state data respectively by using an attention mechanism to obtain initial features corresponding to the multiple groups of state data respectively; A time series feature extraction module, used for performing time series processing on the initial features respectively corresponding to the multiple groups of state data to obtain the time series features respectively corresponding to the multiple groups of state data; A second attention module is used to adopt an attention mechanism and determine the weights corresponding to the multiple groups of state data respectively based on the time series features corresponding to the multiple groups of state data respectively; The state prediction module is used to perform weighted fusion of the time series features corresponding to the multiple groups of state data based on the weights corresponding to the multiple groups of state data, obtain fusion features, and predict the driving state of the driver in a time window after the multiple time windows based on the fusion features.

7. The method according to claim 6, characterized in that The state prediction model is trained based on the following method: Acquire multiple sets of sample state data, the multiple sets of sample state data including first sample state data in multiple time windows collected during the driver's driving and second sample state data collected at a historical moment before the driver starts driving; Acquire an abnormal state label of the first sample state data, where the abnormal state label is used to indicate an abnormal type of the driving state of the first sample state data and a target time window where the abnormal type is located; Acquire the predicted driving state of the driver in a time window after the multiple time windows obtained by the state prediction model based on first sample state data in each time window before the target time window and the second sample state data; The state prediction model is trained based on the predicted driving state and the abnormal type of the driving state included in the abnormal state label.

8. The method according to claim 7, characterized in that The training of the state prediction model based on the predicted driving state and the abnormal type of the driving state included in the abnormal state label includes: Establishing a cross entropy loss function based on the predicted driving state and the abnormal type of the driving state included in the abnormal state label; The state prediction model is trained based on the cross entropy loss function.

9. A driving state prediction device, characterized in that: The device comprises: An acquisition module, configured to acquire multiple sets of status data, the multiple sets of status data including first status data in multiple time windows acquired during a driver's driving and second status data acquired at a historical moment before the driver starts driving; A first feature extraction module is used to extract features from the multiple groups of state data respectively to obtain initial features corresponding to the multiple groups of state data respectively; A second feature extraction module is used to perform time series processing on the initial features respectively corresponding to the multiple groups of state data to obtain the time series features respectively corresponding to the multiple groups of state data; A weight determination module, used to determine the weights corresponding to the multiple groups of state data respectively based on the time series features corresponding to the multiple groups of state data respectively; A feature fusion module, configured to perform weighted fusion on the time series features corresponding to the multiple groups of state data based on the weights corresponding to the multiple groups of state data to obtain fused features; A state prediction module is used to predict the driving state of the driver in a time window after the multiple time windows based on the fusion features.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images