Method and apparatus for predicting driving state, medium, device, and program product

Abstract

A driving state prediction method and device, medium, equipment and program product, the method comprises: acquiring multiple sets of state data, including first state data collected in multiple time windows during driving of a driver and second state data collected at a historical moment before the driver starts driving; performing feature extraction on the multiple sets of state data to obtain initial features corresponding to the multiple sets of state data respectively; performing time series processing on the initial features corresponding to the multiple sets of state data respectively to obtain time series features corresponding to the multiple sets of state data respectively; determining weights corresponding to the multiple sets of state data respectively based on the time series features corresponding to the multiple sets of state data respectively; performing weighted fusion on the time series features corresponding to the multiple sets of state data respectively based on the weights corresponding to the multiple sets of state data respectively to obtain fusion features; and predicting a driving state of the driver in a time window after the multiple time windows based on the fusion features.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, medium, device, and program product for predicting driving status. Background Technology

[0002] Currently, cars, airplanes, and other means of transportation have become indispensable for people's travel. While these vehicles bring convenience, they also raise a series of traffic safety issues. Some of these issues are caused by drivers in poor driving conditions (such as fatigue or distraction). Therefore, it is necessary to predict driver condition to improve traffic safety. Current technologies typically predict driver condition based on data collected during driving. However, many factors influence driver condition, and the aforementioned methods cannot comprehensively capture all the factors affecting driver condition, resulting in low prediction accuracy. Summary of the Invention

[0003] In a first aspect, embodiments of this application provide a method for predicting driving state. The method includes: acquiring multiple sets of state data, the multiple sets of state data including first state data collected within multiple time windows during a driver's driving and second state data collected at historical moments before the driver begins driving; extracting features from the multiple sets of state data to obtain initial features corresponding to each set of state data; performing time-series processing on the initial features corresponding to each set of state data to obtain time-series features corresponding to each set of state data; determining weights corresponding to each set of state data based on the time-series features corresponding to each set of state data; performing weighted fusion on the time-series features corresponding to each set of state data based on the weights corresponding to each set of state data to obtain fused features; and predicting the driver's driving state within time windows following the multiple time windows based on the fused features.

[0004] Secondly, embodiments of this application provide a driving state prediction device, the device comprising: an acquisition module, configured to acquire multiple sets of state data, the multiple sets of state data including first state data collected within multiple time windows during the driver's driving and second state data collected at historical moments before the driver begins driving; a first feature extraction module, configured to extract features from the multiple sets of state data respectively to obtain initial features corresponding to the multiple sets of state data respectively; a second feature extraction module, configured to perform temporal processing on the initial features corresponding to the multiple sets of state data respectively to obtain temporal features corresponding to the multiple sets of state data respectively; a weight determination module, configured to determine the weights corresponding to the multiple sets of state data respectively based on the temporal features corresponding to the multiple sets of state data respectively; a feature fusion module, configured to perform weighted fusion of the temporal features corresponding to the multiple sets of state data respectively based on the weights corresponding to the multiple sets of state data respectively to obtain fused features; and a state prediction module, configured to predict the driving state of the driver within time windows after the multiple time windows based on the fused features.

[0005] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any embodiment of this application.

[0006] Fourthly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any embodiment of this application.

[0007] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this application.

[0008] This application embodiment uses first state data from multiple time windows during the driver's driving period and second state data acquired before the driver begins driving as input data for predicting the driver's state. By performing time-series processing on the initial features extracted from the multiple sets of state data, the implicit time-series features between the sets of state data can be mined. Based on the time-series features corresponding to the multiple sets of state data, the weights corresponding to the multiple sets of state data are determined, which can reflect the proportion of influence of each set of state data on the driving state. Based on the above weights, the time-series features corresponding to each set of state data are weighted and fused, and the driving state is predicted based on the fused features. This can effectively fuse the state data before driving and the state data during driving, so as to more comprehensively consider all aspects that may affect the driving state when making state prediction, thereby improving the prediction accuracy.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments consistent with this application and, together with the description, serve to illustrate the technical solutions of this application.

[0011] Figure 1 This is a flowchart of a method for predicting driving state according to an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of the state prediction model in an embodiment of this application.

[0013] Figure 3 This is a flowchart of the training method for the state prediction model according to an embodiment of this application.

[0014] Figure 4 This is a block diagram of a driving state prediction device according to an embodiment of this application.

[0015] Figure 5 This is a schematic diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0017] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0018] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0019] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0020] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0021] A driver's driving condition has a significant impact on driving safety, therefore, it is necessary to predict the driver's condition. Related technologies typically predict driver condition based on data collected during the driving process, such as video and physiological data. However, many factors influence a driver's condition. For example, physiological indicators on the day of driving (such as sleep quality, alcohol or drug intake) reflect the driver's physiological state, while pre-job assessments such as medical records and skills evaluation data reflect the driver's physiological condition, psychological state, and skill proficiency. All of these factors can affect the driver's driving condition. Current technologies that only consider data collected during driving cannot comprehensively predict the driver's condition, resulting in low accuracy in condition prediction.

[0022] Based on this, this application provides a method for predicting driving states, see [link to relevant documentation]. Figure 1 The method includes:

[0023] Step S11: Acquire multiple sets of state data, including first state data collected during multiple time windows while the driver is driving and second state data collected at historical moments before the driver starts driving.

[0024] Step S12: Extract features from the multiple sets of state data to obtain the initial features corresponding to the multiple sets of state data respectively;

[0025] Step S13: Perform time-series processing on the initial features corresponding to the multiple sets of state data to obtain the time-series features corresponding to the multiple sets of state data.

[0026] Step S14: Determine the weights corresponding to the multiple sets of state data based on the time-series features corresponding to each set of state data.

[0027] Step S15: Based on the weights corresponding to the multiple sets of state data, perform weighted fusion on the temporal features corresponding to the multiple sets of state data to obtain fused features;

[0028] Step S16: Based on the fusion features, predict the driver's driving state within the time window following the multiple time windows.

[0029] In step S11, the status data includes, but is not limited to, the driver's vital signs data, image data, and / or voice data. Status data can be collected through various sensors. For example, wearable devices can collect the driver's vital signs data, image sensors in the cockpit can collect the driver's image data, and sound sensors in the cockpit can collect the driver's voice data. Vital signs data includes, but is not limited to, heart rate, blood pressure, blood oxygen, and body temperature, which can be used to analyze the driver's physiological health; image data can be used to determine the driver's facial expressions and movements to analyze whether the driver is fatigued or distracted during driving; voice data can be used to determine the driver's tone, speech rate, intonation, and responses to other sounds (such as the voices of the in-vehicle voice assistant or other passengers) to analyze the driver's emotional state and reaction speed.

[0030] This application takes into account that the driver's driving state is related not only to the state data during driving, but also to the state data before driving begins. Therefore, multiple sets of state data can be obtained, including the first state data during the driver's driving and the second state data before the driver begins driving.

[0031] The first state data can include time-series numerical data, reflecting the driver's continuous state over a period of time. This first state data, obtained through continuous monitoring of the driver, can be divided into multiple time windows. For example, it can be divided into N time windows of a preset duration t, where the duration of the first state data within each time window is t, N is a positive integer, and t is a positive number. The value of t can be determined based on the data length required for correlation analysis and the computational resource demands. If t is too small, the amount of information in the first state data within each time window is insufficient, potentially making it difficult to extract enough information for state prediction. If t is too large, it means more data needs to be processed, increasing the demand for computational resources. Simultaneously, the correlation of data within a single time window will decrease, introducing unnecessary noise and reducing the accuracy of subsequent state predictions.

[0032] The second state data can include instantaneous data, which can be collected at a historical moment before the driver begins driving. For example, when the driver has completed the pre-driving preparations and is about to start driving, second state data can be collected once, reflecting the driver's instantaneous state at the time of data collection. Optionally, the aforementioned instantaneous state data is data collected by the driver before starting driving on the same day.

[0033] In addition, driver status assessments can be conducted at certain assessment cycles. The assessment content may include, but is not limited to, the driver's physical characteristics data, skills assessment data, and / or psychological assessment data. Secondary status data may include status assessment data obtained during the driver's most recent status assessment before starting to drive. Optionally, the status assessment data may be tabular data, which may include discrete data and / or instantaneous numerical data. For example, psychological assessment data may include attribute items for evaluating the driver's anxiety level, and the attribute values ​​corresponding to these attribute items may include options such as "does not meet," "meets," and "strongly meets," each of which is discrete data. As another example, physical characteristics data may include attribute items corresponding to heart rate and blood pressure, and the attribute values ​​corresponding to these two attribute items are instantaneous numerical data.

[0034] To comprehensively extract factors influencing driving state, each set of state data can include data from multiple dimensions. For example, the first state data includes the driver's physical signs, image data, and voice data during driving; the second state data includes the driver's physical signs and psychological assessment data before starting to drive. When the second state data includes state assessment data obtained from the driver's most recent state assessment before starting to drive, as well as the driver's instantaneous state data before starting to drive on the day of driving, both the state assessment data and the instantaneous state data can include data from multiple dimensions. For example, the state assessment data includes multiple state assessment tables, such as a table for assessing physical signs and a table for assessing psychological state, with each state assessment table representing one dimension of data; the instantaneous state data can include multiple physical signs of the driver at a specific moment, with each physical sign representing one dimension of data.

[0035] In step S12, features can be extracted from each group of state data to obtain the initial features corresponding to each group of state data.

[0036] In some embodiments, feature vectorization can be performed on each group of state data to obtain the feature vector corresponding to each group of state data, and then feature extraction can be performed based on the feature vector corresponding to each group of state data to obtain the initial features corresponding to each group of state data.

[0037] Multiple sets of state data are inherently heterogeneous. For example, in the second set of state data, the state assessment data is tabular data, which may include discrete data and instantaneous numerical data. The instantaneous state data of the driver before starting to drive on the day of driving is instantaneous numerical data, while the first state data within each time window is time-series numerical data. For the above heterogeneous data, feature vectorization processing can be performed in a way that matches its data type.

[0038] Specifically, if a certain dimension of the state data is discrete, embedding can be performed on that dimension to obtain the corresponding feature vector. Embedding can be implemented using word2vec or BERT models. If a certain dimension of the state data is instantaneous numerical data, a linear mapping can be performed on that dimension to obtain the corresponding feature vector. If a certain dimension of the state data is time-series numerical data, a non-linear mapping can be performed on that dimension to obtain the corresponding feature vector. For ease of processing, the dimensions of all obtained feature vectors can be the same, assumed to be d-dimensional feature vectors.

[0039] Furthermore, for any set of state data, the data in at least one dimension may include multiple attribute items and their corresponding attribute values. In this case, the attribute values ​​corresponding to each attribute item can be processed into feature vectors to obtain d-dimensional feature vectors corresponding to each attribute value. Then, the d-dimensional feature vectors corresponding to each attribute value are aggregated to obtain the feature vector corresponding to that dimension. Optionally, the above aggregation can be implemented based on a Transformer network. For example, if the state assessment data included in the second state data is tabular data, which includes attribute items corresponding to anxiety level and attribute items corresponding to heart rate, the attribute values ​​corresponding to the anxiety level attribute item can be converted into d-dimensional feature vectors, and the attribute values ​​corresponding to the heart rate attribute item can be converted into d-dimensional feature vectors. Then, the two d-dimensional feature vectors are aggregated using a Transformer to obtain the feature vector corresponding to the tabular data.

[0040] In some embodiments, an attention mechanism can be used to extract features from each set of state data. Specifically, for any set of state data, after obtaining the feature vectors corresponding to multiple dimensions of the set of state data, linear processing can be performed on the feature vectors corresponding to the multiple dimensions to obtain linear features corresponding to the multiple dimensions. Normalization processing is then performed on the linear features corresponding to the multiple dimensions to obtain the weights corresponding to the multiple dimensions. Based on the weights corresponding to the multiple dimensions, the feature vectors corresponding to the multiple dimensions are weighted to obtain the initial features corresponding to the set of state data. Let f be the feature vector corresponding to the i-th dimension. i Then the weight w corresponding to the i-th dimension i It can be represented as:

[0041] w i =softmax(Linear(f i ))

[0042] Here, Linear represents linear processing, and softmax represents normalization processing. The initial features corresponding to this set of state data can be represented as:

[0043]

[0044] By employing an attention mechanism, information crucial to driver condition assessment can be automatically identified and focused on, reducing noise interference and improving prediction accuracy and robustness.

[0045] In step S13, the initial features corresponding to multiple sets of state data can be processed temporally to obtain the temporal features corresponding to each set of state data. In some embodiments, a bidirectional long short-term memory network (Bi-LSTM) can be used to process the initial features corresponding to each set of state data temporally. Bi-LSTM can effectively capture dependencies in long-term sequences, thereby reflecting the impact of each set of state data on the driving state.

[0046] In step S14, the weights corresponding to the multiple sets of state data can be determined based on the temporal characteristics corresponding to each set of state data. The weights corresponding to each set of state data can reflect the proportion of influence of that set of state data on the driver's driving state. In some embodiments, the weights corresponding to each set of state data can be obtained based on an attention mechanism.

[0047] Specifically, the temporal features corresponding to each group of state data can be linearly processed to obtain the linear features corresponding to that group of state data. Then, the linear features corresponding to each group of state data can be normalized to obtain the weights corresponding to that group of state data. Let h be the temporal feature corresponding to the j-th group of state data. j Then the weight α corresponding to the j-th group of state data j It can be written as:

[0048] α j =softmax(Linear(h j ))

[0049] Here, Linear represents linear processing, and softmax represents normalization processing. By employing an attention mechanism, it is possible to automatically identify and focus on information crucial to driver state judgment, reduce noise interference, and improve prediction accuracy and robustness.

[0050] In step S15, the time-series features corresponding to the multiple sets of state data can be weighted and fused using their respective weights to obtain a fused feature. Continuing with the previous example, the fused feature r can be denoted as:

[0051]

[0052] In step S16, the driver's driving state within the time window following the multiple time windows can be predicted based on the fused features. For example, assuming that the driver's first state data and second state data within N time windows were obtained in the previous steps, this step can predict the category of the driver's driving state within the N+1 time window based on the above state data, such as normal category, distracted driving category, or fatigued driving category, etc.

[0053] In some embodiments, the method of this application can be executed by a pre-trained state prediction model. Multiple sets of state data can be input into the state prediction model, and the state prediction model can perform various steps such as feature extraction, temporal processing, weight determination, temporal feature fusion, and state prediction on the multiple sets of state data.

[0054] See Figure 2 In some embodiments, the state prediction model may include:

[0055] The first attention module is used to extract features from the multiple sets of state data using an attention mechanism to obtain the initial features corresponding to the multiple sets of state data respectively.

[0056] The temporal feature extraction module is used to perform temporal processing on the initial features corresponding to the multiple sets of state data respectively to obtain the temporal features corresponding to the multiple sets of state data respectively.

[0057] The second attention module is used to employ an attention mechanism and determine the weights corresponding to the multiple sets of state data based on the temporal features corresponding to the multiple sets of state data respectively.

[0058] The state prediction module is used to perform weighted fusion of the time-series features corresponding to the multiple sets of state data based on their respective weights to obtain fused features, and to predict the driving state of the driver within the time window following the multiple time windows based on the fused features.

[0059] In some embodiments, see Figure 3 The state prediction model can be trained in the following way:

[0060] Step S21: Obtain multiple sets of sample state data, including first sample state data collected during multiple time windows during the driver's driving and second sample state data collected at historical moments before the driver started driving.

[0061] The first sample state data in this embodiment corresponds to the first state data in the previous embodiment, and the second sample state data corresponds to the second state data in the previous embodiment. Specific details such as the type of multiple sets of sample state data and the method of obtaining multiple sets of sample state data can be referred to the embodiment of multiple sets of state data in the previous embodiment, and will not be repeated here.

[0062] Step S22: Obtain the abnormal status label of the first sample state data. The abnormal status label is used to indicate the abnormal type of the driving state of the first sample state data and the target time window in which the abnormal type is located.

[0063] Abnormal status labels can be generated manually. The abnormal driving state type indicates the specific type of abnormal driving state the driver exhibited, such as distracted driving or fatigued driving. The target time window for the abnormal type indicates the time window in which the abnormal type occurred. For example, if distracted driving (such as using a mobile phone while driving) is detected in the k-th time window, then the target time window for the abnormal type is k.

[0064] Step S23: Obtain the driver's predicted driving state in the time window after the multiple time windows, based on the first sample state data and the second sample state data in each time window before the target time window;

[0065] Continuing with the previous example, assuming an abnormal driving state is detected in the k-th time window, we can obtain the first sample state data from the 1st to the (k-1)th time window, and also obtain the second sample state data. The state prediction model can then use this data to obtain the predicted driving state for the (k+1)-th time window.

[0066] Step S24: Train the state prediction model based on the predicted driving state and the abnormal types of driving states included in the abnormal state labels.

[0067] Based on the predicted driving state and the abnormal types of driving states included in the abnormal state labels, the prediction loss of the state prediction model can be determined, and the state prediction model can be adjusted based on this prediction loss. The structure and specific principles of the state prediction model can be found in the foregoing embodiments, and will not be repeated here.

[0068] In some embodiments, a cross-entropy loss function can be established based on the predicted driving state and the anomaly type of the driving state included in the anomaly state label, and the state prediction model can be trained based on the cross-entropy loss function. The cross-entropy loss can be denoted as:

[0069]

[0070] Where M represents the total number of driving states, including M-1 abnormal states and 1 normal state. i p is the truth label. i The probability corresponding to the i-th driving state in the probability distribution output by the state prediction model.

[0071] The implementation details of this application are illustrated below with a specific embodiment. In this embodiment, it is assumed that the driver is a professional driver, such as a freight driver or bus driver. The driver's driving state is closely related to the safety of people and property. Therefore, this application proposes a method for predicting driving state, which monitors and warns of abnormal driving states based on data fusion. By integrating and analyzing multi-source data, this application can identify abnormal driving states in a timely and comprehensive manner, thereby providing early warnings of potential risks. This helps to protect the lives of drivers and passengers and reduce the loss of life and property caused by abnormal driving states. Among them, the multi-source data includes the state data collected by the driver during on-duty driving, also known as on-duty data (i.e., the first state data in the aforementioned embodiment); the state data of the driver on the day of starting work (i.e., the instantaneous state data of the driver before starting driving on the day of starting work in the aforementioned embodiment), also known as off-duty data; and the state data collected by the driver during the state assessment before officially starting work, which can be the state data collected during the most recent state assessment (i.e., the state assessment data obtained by the driver during the most recent state assessment before starting driving in the aforementioned embodiment), also known as pre-duty data. Both the pre-job data and the post-job data are the second state data in the aforementioned embodiments.

[0072] Current technology primarily focuses on monitoring drivers' condition during their shifts, specifically any abnormal behaviors or physiological abnormalities exhibited by drivers while driving. However, in practical applications, pre-shift and post-shift data can also provide valuable information about abnormal states. Pre-shift data includes medical examination records and skills assessment data, which can be used to analyze drivers' basic health and skill adaptability. Post-shift physiological indicators, such as sleep quality and alcohol or drug intake, may also reflect the driver's psychological or physiological state and have a potential impact on driving safety.

[0073] Due to the lack of pre-employment and post-employment data in the research data, the relevant technologies have also failed to conduct effective correlation analysis on the data from the aforementioned multiple stages. Specifically, the data from the pre-employment, on-the-job, and post-employment stages of drivers lack an effective correlation mechanism, making it difficult to form a comprehensive early warning of abnormal states.

[0074] In this process, highlighting key data across multiple stages of data correlation is a critical challenge. For example, fatigue indicators and skills assessment results from pre-employment medical examinations may trigger abnormal warnings at certain thresholds; during on-the-job activities, facial expressions and steering wheel operation habits can be used to identify abnormal behavior in real time; and sleep quality and heart rate changes upon leaving the job can serve as indicators of potential risks while on duty. Current technologies lack the ability to screen and correlate key data across stages.

[0075] Based on this, this application uses data fusion to monitor and issue early warnings for abnormal driving states of drivers. Specifically, this application includes the following steps:

[0076] Training phase

[0077] (1) Collect a large amount of multi-source data from drivers, including data from the pre-job, post-job, and on-job stages, as well as abnormal status labels (including abnormality types and the time points when abnormalities occur). Among them, pre-job data refers to the status assessment data of drivers before they officially start work, such as physical examination data and psychological assessment data; post-job data refers to the examination data of drivers on the day of work, such as heart rate, blood pressure, blood oxygen, etc.; on-job data refers to the monitoring data of drivers during work, including blood pressure, heart rate, etc. continuously monitored after starting work.

[0078] (2) The data in the above three stages are essentially heterogeneous: physical examination data and psychological assessment data are tabular data, which include discrete data (such as the "anxiety" item in the psychological scale, which is divided into: not agree, agree, and very agree, so its options are discrete data) and instantaneous numerical data (such as the "weight" item in the physical examination data, whose value is instantaneous); heart rate, blood pressure, blood oxygen, etc. are also instantaneous numerical data; blood pressure, heart rate, etc., which are continuously monitored after starting work are time-series numerical data.

[0079] Therefore, for the heterogeneous data in the above three stages, the following scheme can be used to extract their feature vectors: For discrete data, it is mapped to a d-dimensional feature vector. The mapping method can be pre-trained word feature vectors such as word2vec, or feature vectors output by pre-trained models such as BERT; For numerical data, a linear mapping is used to map it to a d-dimensional feature vector; For tabular data, the aforementioned processing methods for discrete and numerical data can be used to convert all values ​​in the table into d-dimensional feature vectors. Then, the Transformer network is used to fuse these d-dimensional feature vectors to obtain the fused d-dimensional feature vector; For time-series numerical data, it is divided into N time windows of length t, and a non-linear mapping is used to map the data in each window to a d-dimensional feature vector.

[0080] For the physical examination form and psychological scale in the pre-employment data, the Transformer was used to aggregate all attributes in the physical examination form to obtain... and For skills test reports, use BERT to obtain them.

[0081] For the blood pressure, heart rate, alcohol intake, and sleep quality data in the work attendance data, the mapping is as follows:

[0082] For the blood pressure and heart rate data in the on-duty data, divide them into N time windows of length t, and map the data in each window into a d-dimensional vector to obtain... and

[0083] (3) Train a data fusion model based on dual attention mechanism.

[0084] First, training data (x, y) is constructed from the data collected in step (1) for the pre-job, post-job, and on-job stages, as well as the abnormal status labels (including the abnormality type and the time node of the abnormality occurrence). Assuming the time node of the abnormality occurrence falls within the nth time window defined in step (2), then x in the training data includes the 0th to n-1th time windows of the pre-job, post-job, and on-job data, and y represents the abnormality type. Continuing with the previous example, x = [f1, f2, f3, f4, f5, f6, f7, f8, f9], where... and

[0085] Next, for the pre-employment data in x, an attention mechanism is used to aggregate the multi-source data to obtain data containing pre-employment information. eigenvectors, where,

[0086] [w1,w2,w3]=softmax(Linear([f1,f2,f3]))

[0087]

[0088] The same method is used to obtain the work stoppage data in x, which includes work stoppage information. Feature vector:

[0089] [w4,w5,w6,w7]=softmax(Linear([f4,f5,f6,f7]))

[0090]

[0091] For each time window of the on-duty data in x, an attention mechanism is used to aggregate the multi-source data within that time window, resulting in a d-dimensional feature vector containing the on-duty information for the current time window. Therefore, for the 0th to (n-1th)th time window of the on-duty data in x, a total of n d-dimensional feature vectors are obtained, denoted as . For the t-th time window, the aggregation formula is:

[0092] [w 8(t) ,w 9(t) ] = softmax(Linear([f8(t) ,f 9(t) ]))

[0093]

[0094] In this phase, attention mechanisms are used to highlight key data from different stages.

[0095] The aforementioned three stages—pre-job, post-job, and on-the-job—exhibit a clear temporal relationship, with pre-job preceding post-job preceding on-the-job. This application uses Bi-LSTM to construct this relationship. Specifically, the d-dimensional feature vectors extracted from pre-job and post-job data are concatenated before the n d-dimensional feature vectors extracted from on-the-job data, resulting in n+2 d-dimensional feature vectors. Then, these n+2 d-dimensional feature vectors are input into the Bi-LSTM to calculate the hidden state. h in the above hidden state i This refers to the temporal features described in the aforementioned embodiments. Next, an attention mechanism is used to capture n+2 h... i The relationship between them is obtained (i.e., the fusion feature in the aforementioned embodiments):

[0096]

[0097] A classifier (i.e., the state prediction module in the aforementioned embodiment) is used to calculate a probability distribution p using r. The anomaly category corresponding to the highest probability in this distribution is the predicted driver anomaly state category for the next time window. In the above process, LSTM constructs the temporal relationship of data in the three stages of pre-shift, on-the-job, and post-shift, and the attention mechanism captures the correlation between different moments in the three stages.

[0098] Finally, the model is trained using the following loss function:

[0099]

[0100] Where M represents the total number of driver states, including M-1 abnormal states and 1 normal state. i p is the truth label. i This outputs the probability of the i-th state in the probability distribution of the model.

[0101] Reasoning stage

[0102] Data from the pre-job, on-the-job, and post-job phases are input into a trained state prediction model, which then predicts the driver's driving state in the next time window.

[0103] See Figure 4 This application also provides a driving state prediction device, the device comprising:

[0104] The acquisition module 101 is used to acquire multiple sets of state data, including first state data collected during the driver's driving in multiple time windows and second state data collected at historical moments before the driver started driving.

[0105] The first feature extraction module 102 is used to extract features from the multiple sets of state data respectively to obtain the initial features corresponding to the multiple sets of state data respectively.

[0106] The second feature extraction module 103 is used to perform time-series processing on the initial features corresponding to the multiple sets of state data respectively to obtain the time-series features corresponding to the multiple sets of state data respectively.

[0107] The weight determination module 104 is used to determine the weights corresponding to the multiple sets of state data based on the time-series features corresponding to the multiple sets of state data respectively.

[0108] The feature fusion module 105 is used to perform weighted fusion of the time-series features corresponding to the multiple sets of state data based on the weights corresponding to the multiple sets of state data respectively, to obtain fused features;

[0109] The state prediction module 106 is used to predict the driving state of the driver within a time window following the plurality of time windows based on the fused features.

[0110] The apparatus provided in this application has functions or includes modules that can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0111] This application also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the foregoing embodiments.

[0112] Figure 5 This illustration shows a more specific hardware structure diagram of a computer device provided in an embodiment of this application. The device may include: a processor 201, a memory 202, an input / output interface 203, a communication interface 204, and a bus 205. The processor 201, memory 202, input / output interface 203, and communication interface 204 are interconnected internally via the bus 205.

[0113] The processor 201 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The processor 201 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

[0114] The memory 202 can be implemented in the form of read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented by software or firmware, the relevant program code is stored in the memory 202 and is called and executed by the processor 201.

[0115] Input / output interface 203 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0116] The communication interface 204 is used to connect the communication module (not shown in the figure) to enable communication and interaction between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0117] Bus 205 includes a pathway for transmitting information between various components of the device (e.g., processor 201, memory 202, input / output interface 203, and communication interface 204).

[0118] It should be noted that although the above-described device only shows the processor 201, memory 202, input / output interface 203, communication interface 204, and bus 205, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this application, and not necessarily all the components shown in the figures.

[0119] This application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any embodiment of this application.

[0120] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0121] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer devices. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0122] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0123] The above description is merely a specific implementation of the embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this application, and these improvements and modifications should also be considered within the protection scope of the embodiments of this application.

Claims

1. A method for predicting driving states, characterized in that, The method includes: Multiple sets of state data are acquired, including first state data collected within multiple time windows during the driver's driving and second state data collected at historical moments before the driver begins driving; the first state data is used to reflect the driver's continuous state over a period of time, and the second state data is used to reflect the driver's instantaneous state at the time of data acquisition; the state data includes the driver's vital signs data, image data, and / or voice data. Feature extraction is performed on the multiple sets of state data to obtain the initial features corresponding to the multiple sets of state data respectively; The initial features corresponding to the multiple sets of state data are subjected to time-series processing to obtain the time-series features corresponding to the multiple sets of state data. Based on the temporal characteristics corresponding to the multiple sets of state data, the weights corresponding to the multiple sets of state data are determined. Based on the weights corresponding to the multiple sets of state data, the time-series features corresponding to the multiple sets of state data are weighted and fused to obtain the fused features. Based on the fusion features, the driver's driving status is predicted within the time window following the multiple time windows; The method is performed by a pre-trained state prediction model; the state prediction model includes: The first attention module is used to extract features from the multiple sets of state data using an attention mechanism to obtain the initial features corresponding to the multiple sets of state data respectively. The temporal feature extraction module is used to perform temporal processing on the initial features corresponding to the multiple sets of state data respectively to obtain the temporal features corresponding to the multiple sets of state data respectively. The second attention module is used to employ an attention mechanism and determine the weights corresponding to the multiple sets of state data based on the temporal features corresponding to the multiple sets of state data respectively. The state prediction module is used to perform weighted fusion of the time-series features corresponding to the multiple sets of state data based on their respective weights to obtain fused features, and to predict the driver's driving state in the time window after the multiple time windows based on the fused features, thereby obtaining the category of the driver's driving state in the time window after the multiple time windows, including normal category, distracted driving category and fatigued driving category.

2. The method according to claim 1, characterized in that, Each set of state data includes data from multiple dimensions; the initial features corresponding to any set of state data are obtained based on the following method: The data of the state data includes multiple dimensions, which are processed into feature vectors to obtain feature vectors corresponding to each of the multiple dimensions. The feature vectors corresponding to the multiple dimensions are linearly processed to obtain the linear features corresponding to the multiple dimensions. The linear features corresponding to the multiple dimensions are normalized to obtain the weights corresponding to the multiple dimensions. Based on the weights corresponding to the multiple dimensions, the feature vectors corresponding to the multiple dimensions are weighted to obtain the initial features corresponding to the set of state data.

3. The method according to claim 2, characterized in that, The multiple dimensions of any set of state data include at least one of discrete data, instantaneous numerical data, and time-series numerical data; the step of performing feature vectorization processing on the multiple dimensions of the state data to obtain feature vectors corresponding to each of the multiple dimensions includes: If the data in any dimension is discrete, embedding is performed on the data in that dimension to obtain the feature vector corresponding to that dimension; If the data in any dimension is instantaneous numerical data, a linear mapping is performed on the data in that dimension to obtain the feature vector corresponding to that dimension; If the data in any dimension is time-series numerical data, a non-linear mapping is performed on the data in that dimension to obtain the corresponding feature vector.

4. The method according to claim 1, characterized in that, The second state data includes: The driver's condition assessment data obtained during the most recent condition assessment before starting to drive; and The instantaneous state data of the driver before starting to drive on the day of driving.

5. The method according to claim 1, characterized in that, The step of determining the weights corresponding to the multiple sets of state data based on their respective temporal features includes: The temporal features corresponding to each set of state data are linearly processed to obtain the linear features corresponding to that set of state data. The linear features corresponding to each set of state data are normalized to obtain the weights corresponding to that set of state data.

6. The method according to claim 1, characterized in that, The state prediction model is trained in the following manner: Acquire multiple sets of sample state data, including first sample state data collected within multiple time windows during the driver's driving and second sample state data collected at historical moments before the driver started driving. Obtain the abnormal status label of the first sample status data. The abnormal status label is used to indicate the abnormal type of the driving status of the first sample status data and the target time window in which the abnormal type is located. The state prediction model obtains the driver's predicted driving state in the time window after the multiple time windows based on the first sample state data and the second sample state data in each time window before the target time window. The state prediction model is trained based on the predicted driving state and the abnormal types of driving states included in the abnormal state labels.

7. The method according to claim 6, characterized in that, The step of training the state prediction model based on the predicted driving state and the abnormal types of driving states included in the abnormal state labels includes: Based on the predicted driving state and the abnormal types of driving states included in the abnormal state labels, a cross-entropy loss function is established. The state prediction model is trained based on the cross-entropy loss function.

8. A driving state prediction device, characterized in that, The device includes: The acquisition module is used to acquire multiple sets of state data, including first state data collected within multiple time windows during the driver's driving and second state data collected at historical moments before the driver started driving; the first state data is used to reflect the driver's continuous state over a period of time, and the second state data is used to reflect the driver's instantaneous state at the time of data acquisition; the state data includes the driver's vital signs data, image data, and / or voice data. The first feature extraction module is used to extract features from the multiple sets of state data respectively to obtain the initial features corresponding to the multiple sets of state data respectively. The second feature extraction module is used to perform time-series processing on the initial features corresponding to the multiple sets of state data respectively to obtain the time-series features corresponding to the multiple sets of state data respectively. The weight determination module is used to determine the weights corresponding to the multiple sets of state data based on the time-series features corresponding to the multiple sets of state data respectively. The feature fusion module is used to perform weighted fusion of the time-series features corresponding to the multiple sets of state data based on the weights corresponding to the multiple sets of state data respectively, to obtain fused features; The state prediction module is used to predict the driver's driving state within a time window following the multiple time windows based on the fused features; The device is applied to a pre-trained state prediction model; the state prediction model includes: The first attention module is used to extract features from the multiple sets of state data using an attention mechanism to obtain the initial features corresponding to the multiple sets of state data respectively. The temporal feature extraction module is used to perform temporal processing on the initial features corresponding to the multiple sets of state data respectively to obtain the temporal features corresponding to the multiple sets of state data respectively. The second attention module is used to employ an attention mechanism and determine the weights corresponding to the multiple sets of state data based on the temporal features corresponding to the multiple sets of state data respectively. The state prediction module is used to perform weighted fusion of the time-series features corresponding to the multiple sets of state data based on their respective weights to obtain fused features, and to predict the driver's driving state in the time window after the multiple time windows based on the fused features, thereby obtaining the category of the driver's driving state in the time window after the multiple time windows, including normal category, distracted driving category and fatigued driving category.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

Images