Multi-category unsafe driving behavior detection method, device and computer equipment

Abstract

This application relates to a multi-category unsafe driving behavior detection method, apparatus, and computer device. The method comprises: determining a target driving behavior category for detection of a target object within different driving behavior categories, generating a driving behavior detection task based on the target driving behavior category; obtaining driving behavior data of the target object; filtering the driving behavior data based on the driving behavior detection task to obtain driving behavior data relevant to the driving behavior detection task; integrating the driving behavior data relevant to the driving behavior detection task using an inter-modal data fusion algorithm to obtain modal set data; and inputting the modal set data into a temporal neural network model to analyze the temporal characteristics of the modal set data to obtain driving behavior detection results for the target object within the target driving behavior category. This method enables efficient, accurate, and diverse detection of unsafe driving behaviors.

Description

Technical Field

[0001] The present application relates to the field of driving safety technology, and in particular to a method, apparatus, and computer device for detecting multiple categories of unsafe driving behaviors. Background Art

[0002] In the field of driving safety technology, an unsafe driving behavior detection system is used to detect the driver's unsafe driving behavior to ensure the driving safety of the driver and passengers.

[0003] In the related art, in one type of unsafe driving behavior detection system, unsafe driving behavior is detected only based on image data. However, on the one hand, it is difficult to ensure the image quality in a dim environment; on the other hand, it is difficult to ensure multi-dimensional detection of the driver's unsafe driving behavior.

[0004] In another type of unsafe driving behavior detection system, unsafe driving behaviors are detected based on multimodal data. However, on the one hand, only detecting one specific category of unsafe driving behavior makes it impossible for one system to detect and warn of multiple unsafe driving behaviors. On the other hand, detection is only based on the driver's current state, without considering the temporal continuity and time dependence of driving behavior.

[0005] Based on this, it is difficult for the unsafe driving behavior detection system in the related art to detect the driver's unsafe driving behavior efficiently, accurately and diversely. Summary of the Invention

[0006] Based on this, it is necessary to provide a multi-category unsafe driving behavior detection method, device, computer equipment and computer-readable storage medium to address the above technical problems, so as to detect unsafe driving behaviors efficiently, accurately and diversely.

[0007] In a first aspect, the present application provides a multi-category unsafe driving behavior detection method, comprising:

[0008] Determining a target driving behavior category for detecting a target object among different driving behavior categories, and generating a driving behavior detection task corresponding to the target object based on the target driving behavior category;

[0009] Acquiring driving behavior data of the target subject, the driving behavior data being multimodal data including driving behavior physiological data and driving behavior operational data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target subject during driving, and the driving behavior operational data represents data describing the target subject's operational behavior of a vehicle during driving;

[0010] performing data screening on the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrating the driving behavior data related to the driving behavior detection task through a preset inter-modality data fusion algorithm to obtain modality set data;

[0011] Obtain a temporal neural network model corresponding to the driving behavior detection task, input the modal set data into the temporal neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0012] In a second aspect, the present application also provides a multi-category unsafe driving behavior detection device, including:

[0013] A task generation module is used to determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category;

[0014] an acquisition module, configured to acquire driving behavior data of the target subject, the driving behavior data being multimodal data including driving behavior physiological data and driving behavior operational data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target subject during driving, and the driving behavior operational data represents data describing the target subject's operational behavior of a vehicle during driving;

[0015] a data processing module, configured to screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrate the driving behavior data related to the driving behavior detection task using a preset inter-modality data fusion algorithm to obtain modality aggregate data;

[0016] The detection module is used to obtain a temporal neural network model corresponding to the driving behavior detection task, input the modal set data into the temporal neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0017] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above steps when executing the computer program.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above steps when executed by a processor.

[0019] The above-mentioned multi-category unsafe driving behavior detection method, apparatus, computer device and computer-readable storage medium firstly, in different driving behavior categories, diversely and selectively determine the target driving behavior category for detection of the target object, and generate a driving behavior detection task based on the target driving behavior category; secondly, obtain multimodal driving behavior data of the target object, and screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, thereby reducing the amount of data calculation and improving the detection accuracy and efficiency; thirdly, integrate the screened driving behavior data through an inter-modal data fusion algorithm to obtain modal set data, thereby improving the accuracy of the data; finally, input the modal set data into a temporal neural network model for time feature analysis to obtain a driving behavior detection result, thereby accurately, efficiently and adaptively obtaining the driving behavior detection result of the target object in the target driving behavior category in the temporal feature dimension based on the temporal continuity and time dependence of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 1 is a flow chart of a method for detecting multiple categories of unsafe driving behaviors in one embodiment;

[0022] Figure 2 2 is a structural block diagram of a device for detecting multiple categories of unsafe driving behaviors in one embodiment. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0024] In one embodiment, Figure 1 As shown, a multi-category unsafe driving behavior detection method is provided. This embodiment uses the method applied to a server as an example. It is understood that the method can also be applied to a terminal, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps S101 to S103.

[0025] Step S101 : determining a target driving behavior category for detecting a target object among different driving behavior categories, and generating a driving behavior detection task corresponding to the target object based on the target driving behavior category.

[0026] Among them, unsafe driving behaviors refer to behaviors taken by drivers during driving that may cause traffic accidents or affect traffic safety, such as speeding, running red lights, frequent lane changes, drunk driving, fatigue driving, distracted driving, etc.

[0027] Among them, the driving behavior category represents the classification of driving behavior according to specific behavioral characteristics, which is used to distinguish the driver's operating status or behavioral performance during the driving process; a driving behavior category may include at least one safe driving behavior state and one unsafe driving behavior state. For example, when the driving behavior category is "speeding", the driving behavior category may include the safe driving behavior state of "not speeding" and the unsafe driving behavior state of "speeding"; the target driving behavior category represents the driving behavior category to be detected.

[0028] The target object represents the specific driver who needs to undergo driving behavior detection.

[0029] Among them, the driving behavior detection task refers to the specific detection target and analysis task generated for the target object's driving behavior based on a specific driving behavior category, which is used to perform risk assessment on the target object's driving behavior.

[0030] For example, the target driving behavior category for detection of the target object can be determined by manual setting or intelligent calibration. For example, the driving behavior category judgment criteria for different application scenarios can be pre-set. For example, for a logistics fleet, "speeding", "fatigue driving", "frequent sudden braking" and other target driving behavior categories can be set as key detection targets; the trigger conditions corresponding to different driving behavior categories can be pre-selected and set. For example, the trigger condition corresponding to judging "whether speeding" can be set to "trigger when the vehicle speed exceeds the road speed limit", or the trigger condition corresponding to judging "whether fatigue driving" can be set to "trigger after continuous driving for more than 4 hours".

[0031] Step S102: Acquire driving behavior data of the target object. The driving behavior data is multimodal data including driving behavior physiological data and driving behavior operational data. The driving behavior physiological data represents data describing the physiological characteristics of the target object during driving, and the driving behavior operational data represents data describing the target object's operational behavior of the vehicle during driving.

[0032] The driving behavior data refers to a set of information describing the behavioral characteristics and status of the driver during driving.

[0033] The driving behavior physiological data refers to physiological signal data describing the driver's physiological state, such as eye movement data, heart rate data, breathing data, body temperature data, etc.

[0034] The driving behavior operation data refers to vehicle driving data that describes the driver's control behavior of the vehicle, such as speed, acceleration, steering wheel angle, and the distance between the vehicle and the side margins of the lane.

[0035] For example, driving physiological data can be collected through wearable devices (such as smart watches, smart bracelets, etc.), vehicle-mounted biosensors (such as cameras, heart rate monitors, etc.), and other devices; driving behavior operation data can be collected through vehicle-mounted sensors (such as speed sensors, steering wheel angle sensors, vehicle distance sensors, etc.).

[0036] In step S103, the driving behavior data is screened based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and the driving behavior data related to the driving behavior detection task is integrated through a preset inter-modal data fusion algorithm to obtain modal set data.

[0037] Among them, the inter-modal data fusion algorithm refers to an algorithm used to integrate and process data of different modalities (such as different data sources or different data types, etc.).

[0038] Among them, modal set data represents the unified data expression after feature extraction and fusion of different modal data through inter-modal data fusion algorithm.

[0039] For example, based on the specific scenario requirements of the driving behavior detection task, driving behavior data that is highly relevant to the driving behavior detection task can be screened out from the multimodal driving behavior data; wherein, the screening process includes parsing the goals and conditions of the driving behavior detection task, combining the data type, data characteristics and specific indicators of the driving behavior data, and screening and extracting valid data from the driving behavior data through preset matching rules.

[0040] For example, a preset inter-modal data fusion algorithm is used to integrate and process the filtered driving behavior data related to the driving behavior detection task; the integration process includes time alignment, feature extraction and weight allocation of driving behavior data of different modalities, and unified feature expression is achieved through feature splicing or deep learning models.

[0041] Step S104: obtain a temporal neural network model corresponding to the driving behavior detection task, input the modal set data into the temporal neural network model to analyze the temporal characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0042] Among them, the temporal neural network model represents a deep learning model for processing time series data. It obtains the driving behavior detection results corresponding to the target object by capturing the dependencies and dynamic change patterns of the data in the time dimension.

[0043] Among them, the temporal characteristics of the modal set data represent the dynamic change information presented by the modal set data in the time dimension, reflecting the changing trend of driving behavior in different time periods.

[0044] Among them, the driving behavior detection result represents the conclusive information obtained after performing time feature analysis on the input modal set data through the temporal neural network model, which is used to characterize the specific state or risk assessment of the target object corresponding to the target driving behavior category.

[0045] For example, the time features in the time dimension of the modal set data can be effectively captured through a long short-term memory network (LSTM) or a deep learning network (Transformer) based on an attention mechanism, and the time features in the modal set data can be deeply mined through a time series analysis method to extract the dynamic change patterns of the data and the time dependencies between the modalities. Through this analysis process, the behavior patterns, trend changes, and potential abnormal states of the target object during the driving process can be identified, thereby generating driving behavior detection results for the target driving behavior category.

[0046] Optionally, the driving behavior detection result is presented in the form of classification, risk score or abnormality mark, which specifically describes the driving behavior status of the target object under the driving behavior category.

[0047] Optionally, different driving behavior detection tasks correspond to different temporal neural network models, that is, different temporal neural network models are trained based on different driving behavior detection tasks, so that the matching temporal neural network model can be called according to the driving behavior detection task generated in the actual detection process, thereby realizing selective detection of different types of driving behaviors.

[0048] In the above-mentioned multi-category unsafe driving behavior detection method, first, in different driving behavior categories, the target driving behavior category for detection of the target object is diversely and selectively determined, and a driving behavior detection task is generated based on the target driving behavior category; secondly, multimodal driving behavior data of the target object is obtained, and the driving behavior data is screened based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, thereby reducing the amount of data calculation and improving the detection accuracy and efficiency; thirdly, the screened driving behavior data is integrated through an inter-modal data fusion algorithm to obtain modal set data, thereby improving the data accuracy; finally, the modal set data is input into a temporal neural network model for temporal feature analysis to obtain a driving behavior detection result, thereby accurately, efficiently, and adaptively obtaining the driving behavior detection result of the target object in the target driving behavior category in the temporal feature dimension based on the temporal continuity and temporal dependence of the data.

[0049] In an exemplary embodiment, before performing data screening on the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, the method further includes steps S201 to S203; performing data screening on the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, including step S204.

[0050] Step S201 : obtaining type label data corresponding to different driving behavior categories. The type label data corresponding to the driving behavior categories represent data for marking safe behavior states and unsafe behavior states in the driving behavior categories, respectively.

[0051] Exemplarily, a driving behavior category may correspond to at least two type label data, namely, type label data corresponding to a safe driving behavior state and type label data corresponding to an unsafe driving behavior state. For example, when the driving behavior category is "whether to exceed speed limit", the driving behavior category corresponds to type label data for marking the safe driving behavior state of "not exceeding speed limit" and type label data for marking the unsafe driving behavior state of "exceeding speed limit".

[0052] Step S202 : Calculate the Pearson correlation coefficient between the data features corresponding to each driving behavior data and each type of label data using the Pearson correlation coefficient algorithm to obtain the correlation between the data features corresponding to each driving behavior data and each type of label data.

[0053] The Pearson correlation coefficient algorithm represents a statistical method for measuring the degree of linear correlation between two variables; the Pearson correlation coefficient represents the degree of correlation between the data feature vector of the driving behavior data and the type label data of the driving behavior category.

[0054] For example, setting For the data feature vector corresponding to a certain mode of driving behavior data, set is the feature vector of each type of label data for a certain driving behavior category, then the correlation between X and Y is calculated as follows:

[0055] (1)

[0056] In formula (1), Indicates the Pearson correlation coefficient corresponding to X and Y, which is used to characterize the degree of correlation between X and Y; represents the i-th element in X, represents the i-th element in Y, represents the mean of each element in X, represents the mean of each element in Y, and n represents the number of data points.

[0057] Based on formula (1), the correlation between a certain data feature of the driving behavior data of a certain mode and the type label data of a certain driving behavior category is obtained, and then the correlation between the data feature and the driving behavior category is obtained; based on this, the driving behavior data of each mode and each driving behavior category are traversed through formula (1) to obtain the correlation between the data feature of the driving behavior data of any mode and any driving behavior category; a correlation coefficient matrix can be formed to record the correlation coefficient of each pair of data.

[0058] Step S203 : Based on the correlation between the data features of each driving behavior data and each type of label data, the data features of the driving behavior data that match each driving behavior category are obtained from the data features of each driving behavior data.

[0059] For example, based on data distribution and business needs, the correlation threshold conditions for pairing data features of driving behavior data and driving behavior categories can be determined. For example, a pairing relationship can be established between the data features of driving behavior data and driving behavior categories whose correlation coefficients are greater than or equal to a preset threshold; or the correlation coefficients of each pair of data can be sorted in descending order, and a pairing relationship can be established between the data features of driving behavior data and driving behavior categories whose correlation coefficients are less than or equal to the preset threshold.

[0060] In step S204, based on the target driving behavior category corresponding to the driving behavior detection task, target data features of the driving behavior data matching the target driving behavior category are determined, and based on the target data features, the driving behavior data are screened to obtain driving behavior data related to the driving behavior detection task.

[0061] Exemplarily, a target data feature that is paired with a target driving behavior category is determined, and based on the target data feature, the driving behavior data is screened, that is, the driving behavior data belonging to the target data feature is retained, and the driving behavior data belonging to data features other than the target data feature is removed, thereby screening out driving behavior data related to the driving behavior detection task.

[0062] For example, if the Pearson correlation coefficient of the "driver's eye state" data feature in the image modal data and the type label data of the "whether fatigue driving" driving behavior category meets the correlation threshold condition, then a pairing relationship is constructed between the "driver's eye state" data feature in the image modal data and the "whether fatigue driving" driving behavior category; in the driving behavior detection task corresponding to the "whether fatigue driving" driving behavior category, the image modal data corresponding to the "driver's eye state" data feature is preferentially selected, and then the data corresponding to other data features are selected.

[0063] In this embodiment, based on the Pearson correlation coefficient algorithm, the correlation between the data features of the driving behavior data of any modality and any driving behavior category is predetermined, thereby establishing a pairing relationship between each driving behavior category and the data features of the specified driving behavior data. Based on the pre-established pairing relationship, the driving behavior data is efficiently and accurately screened to obtain driving behavior data related to the driving behavior detection task, thereby reducing the amount of data calculation.

[0064] In an exemplary embodiment, the driving behavior data related to the driving behavior detection task is integrated through a preset inter-modality data fusion algorithm to obtain modality set data, including steps S301 to S304.

[0065] In step S301, each driving behavior data related to the driving behavior detection task is used as the first modal data, and each first modal data is dynamically time-warped to obtain each first modal data of the same time step, and each first modal data of the same time step is used as the second modal data.

[0066] Among them, Dynamic Time Warping (DTW) is a method for aligning time series data.

[0067] For example, since the first modal data of different modalities may have different sampling rates, durations or time offsets, it is necessary to adjust and match the first modal data on a time scale through a DTW algorithm to unify the various first modal data to the same time step to obtain the second modal data, ensuring that the subsequent feature extraction and encoding processing of the various second modal data can be performed in a consistent time frame.

[0068] Optionally, dynamic time warping can be used to automatically calculate the time gap between modal data and perform interpolation on low-frequency modal data so that all data maintain the same time step.

[0069] In step S302, feature extraction is performed on each piece of second modal data to obtain features corresponding to each piece of second modal data, and the features corresponding to each piece of second modal data are encoded to obtain each piece of second modal data in the same data format. Each piece of second modal data in the same data format is used as third modal data.

[0070] Exemplarily, features that effectively represent the data characteristics are extracted from the second modality data. For example, for facial video data, surface features such as image edges and colors, and deeper features such as the driver's emotions, are extracted. For eye movement data, features such as pupil size, gaze point coordinates, and pupil size are extracted. For acceleration data, features such as acceleration mean, variance, and frequency domain are extracted. Each extracted feature is then encoded, mapping the feature value to a specific numerical range or format. This allows the different second modality data to be converted into third modality data of the same data format.

[0071] In step S303 , each third modal data is integrated into the same representation space to obtain each third modal data in the same representation space, and each third modal data in the same representation space is used as initial modal set data.

[0072] Exemplarily, each third modality data can be integrated into the same structured framework, such as a table, matrix, tensor, etc., to obtain initial modality set data in the same representation space, and each data in the initial modality set data is presented in the same representation space in the same data format and time alignment.

[0073] Step S304 , performing back propagation processing on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data, thereby obtaining modal set data.

[0074] Here, backpropagation represents a data processing method for iteratively optimizing data, which is used to improve the quality of the initial modal set data.

[0075] Among them, inconsistent features refer to conflicting features in the initial modal set data caused by problems such as sensor noise and differences in data sources. For example, sensor acquisition noise causes data distortion, or the eye movement sensor and facial video sensor may simultaneously collect feature information of pupil size, but due to different data sources, these features may have inconsistent calculation results.

[0076] Among them, repeated features mean that in the initial modal set data, data from different modalities have the same or highly similar meanings, resulting in information redundancy; for example, the eye movement sensor and the facial video sensor may collect feature information of pupil size at the same time, but due to different data sources, these features may be repeated.

[0077] Exemplarily, the initial modal set data is subjected to back-propagation processing, that is, by defining an optimization objective (such as minimizing the redundancy or contradiction between features), the degree of conflict or redundancy between data features is calculated; low-quality features are eliminated through multiple iterations, and these inconsistent and repeated features are gradually removed, thereby gradually optimizing the initial modal set data to obtain optimized modal set data.

[0078] In this embodiment, first, the first modal data is dynamically time-warped to obtain second modal data of the same time step, thereby increasing the operability of the data in the dimension of time alignment; secondly, the features of the second modal data are encoded to obtain third modal data of the same data format, thereby increasing the operability of the data in the dimension of data format; thirdly, the third modal data are integrated into the same representation space to obtain initial modal set data in the same representation space, thereby ensuring the integrity and structuring of the data; finally, the initial modal set data is back-propagated to remove inconsistent features and repeated features to obtain modal set data with no conflicting and repeated information, thereby improving the data quality and contributing to the accuracy of the detection results.

[0079] In an exemplary embodiment, before performing dynamic time warping on each first modal data, the method further includes step S401.

[0080] In step S401, the relative spatial position of each first modal data in the same space is determined by a preset calibration algorithm, and based on the relative spatial position of each first modal data in the same space, each first modal data is mapped to the same coordinate system to obtain the first modal data in the same coordinate system.

[0081] Among them, the calibration algorithm refers to the algorithm used to determine the relative spatial position and relationship of different modal data sources (such as sensors). It maps all data into a unified spatial framework by analyzing the geometric layout, relative position or angular deviation between sensors.

[0082] Exemplarily, a calibration algorithm is used to calculate and determine the relative spatial position of each first modal data in the same space, wherein the first modal data of each modality (such as the driver's eye movement data, facial video data, steering wheel operation data, etc.) comes from different sensors, and these data may have different acquisition reference systems or coordinate representations in space. Through the calibration algorithm, the spatial relationship between the first modal data of each modality can be accurately estimated, such as the installation angle, acquisition range or positioning error of the sensor.

[0083] After determining the spatial relationship between the first modal data of each modality, all the first modal data are spatially aligned and mapped to the same coordinate system, for example, through a coordinate conversion matrix or a geometric transformation method, to ensure that the first modal data of different modalities can be represented and analyzed in a unified spatial framework.

[0084] In this embodiment, the first modality data from different sources are unified into the same coordinate system, so that the first modality data of different modalities can be processed in a unified spatial framework, thereby ensuring the operability of the data.

[0085] In an exemplary embodiment, the modal set data is input into a temporal neural network model to analyze the temporal characteristics of the modal set data to obtain a driving behavior detection result of the target object in the target driving behavior category, including steps S501 to S503.

[0086] Step S501 : Divide the modality set data into different time window data according to different time windows, take the time window data corresponding to the same time window as the same sample data, and obtain a sample data set based on the different sample data.

[0087] Among them, the time window represents the time period used to divide continuous time series data in time series analysis; the time window data corresponding to the same time window corresponds to all modal set data within a time window, that is, it is used to describe the behavioral characteristics of the target object within the time period to which the time window belongs.

[0088] Exemplarily, according to different time windows, the modal set data is divided into time window data corresponding to different time windows, that is, the modal set data is divided into different data blocks according to different time periods, and each data block can reflect the driving behavior characteristics within the time period; the time window data corresponding to the same time window are fused into a complete and independent sample data, and a sample data set is constructed based on the sample data corresponding to each time window.

[0089] Optionally, the modal set data can be divided into time window data corresponding to different time windows according to different time windows and time steps. The time step represents the time interval each time the time window is moved, which determines the coverage density of the time window. The shorter the time step, the more overlap between the time windows and the denser the data coverage.

[0090] In step S502 , the sample data sets are input into a temporal neural network model for time feature analysis to obtain a driving behavior detection result of the target object in the target driving behavior category.

[0091] Exemplarily, the temporal neural network model performs time feature analysis on the sample data set, that is, performs time feature analysis on the sample data corresponding to each time window in the sample data set, so as to capture the dynamic change pattern of driving behavior between different time periods.

[0092] In step S503, if the driving behavior detection result indicates that the target object is in an unsafe driving behavior state in the target driving behavior category, a warning message corresponding to the driving behavior detection result is generated and displayed in a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category.

[0093] Among them, the warning information refers to a prompt or warning content generated when it is detected that the target object is in an unsafe driving behavior state in the target driving behavior category, which is used to remind the target object to take measures to correct or avoid potential dangers.

[0094] For example, the warning information may include a specific description of the driving behavior detection results (such as "currently in a fatigue driving state") and recommended measures (such as "recommended to take a break" or "slow down"); the generated warning information can be displayed through preset reminder methods, such as in-vehicle voice reminders, screen prompts or vibration feedback, etc., to attract the driver's attention; the warning information will continue to be displayed until it is detected that the driver has returned to a safe driving behavior state, thereby ensuring driving safety.

[0095] In this embodiment, on the one hand, the modal set data is divided into different time window data according to different time windows, and the time window data corresponding to the same time window is used as the same sample data. Sample data sets are obtained based on different sample data, and the sample data sets are respectively input into the time series neural network model for time feature analysis, so as to efficiently and accurately capture the dynamic change rules of driving behavior between different time periods to improve the reliability of the driving behavior detection results; on the other hand, warning information is generated based on the driving behavior detection results and displayed through a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category, thereby improving the safety of the driving process based on the real-time feedback mechanism.

[0096] In an exemplary embodiment, after obtaining the driving behavior detection result of the target object in the target driving behavior category, the method further includes steps S601 to S603.

[0097] Step S601 : normalize the driving behavior prediction result of the target object to obtain a normalized driving behavior prediction result.

[0098] For example, the driving behavior prediction results of the target object are normalized and mapped to a uniform interval (e.g., [0, 1] or [-1, 1]) in the form of a time series to eliminate differences in data dimensions, ranges, and distributions.

[0099] In step S602, the normalized driving behavior prediction results are divided into different first time window data according to different time windows, and the modal set data are divided into different second time window data according to different time windows. The normalized first time window data and second time window data corresponding to the same time window are integrated to obtain integrated data corresponding to the same time window.

[0100] Exemplarily, the normalized driving behavior prediction results and modal set data are divided into different time window data according to different time windows, and the first time window data corresponding to the normalized driving behavior prediction results in different time windows and the second time window data corresponding to the modal set data in different time windows are obtained; the first time window data and the second time window data corresponding to the same time window are integrated to generate integrated data corresponding to the time window, thereby obtaining the integrated data corresponding to each time window.

[0101] Step S603: Obtain a prediction time series neural network model corresponding to the driving behavior detection task, input the integrated data corresponding to different time windows into the prediction time series neural network model for time feature analysis, and obtain the driving behavior prediction results of the target object in the target driving behavior category in the future time period.

[0102] Among them, the predictive time series neural network model represents a deep learning model for processing time series data. It obtains the driving behavior prediction results of the target object in the future period by capturing the dependencies and dynamic change patterns of the data in the time dimension.

[0103] Among them, the driving behavior prediction result represents the conclusive information obtained after the time feature analysis of the input integrated data through the predictive time series neural network model, which is used to characterize the specific state or risk assessment corresponding to the target driving behavior category of the target object in the future time period.

[0104] Exemplarily, the integrated data corresponding to different time windows are input into the predictive time series neural network model for time feature analysis, that is, according to the dependency relationship of the integrated data corresponding to different time windows in the time series, the predictive time series neural network model extracts time features and performs trend analysis on the integrated data to predict the driving behavior state corresponding to the target driving behavior category of the target object in the future time period.

[0105] In this embodiment, the time window data corresponding to the driving behavior prediction results and the modal set data are integrated according to different time windows, so as to comprehensively obtain the integrated data corresponding to each time window. Based on the integrated data corresponding to each time window, time feature analysis is performed to obtain the driving behavior prediction results of the target object in the future time period, thereby improving the scenario diversity of driving behavior detection.

[0106] In an exemplary embodiment, before performing data screening on the driving behavior data based on the driving behavior detection task, the method further includes steps S701 to S703.

[0107] Step S701 : normalizing the driving behavior data using a preset linear normalization method to obtain normalized driving behavior data.

[0108] Exemplarily, the driving behavior data is normalized by a linear normalization method to map the values ​​of driving behavior data of different modes into a unified interval range, so that the normalized driving behavior data has better numerical consistency.

[0109] In step S702, the normalized driving behavior data is interpolated in time series in a preset time series model using an interpolation method based on a machine learning model to obtain time series data corresponding to the normalized driving behavior data.

[0110] For example, in a preset time series model, such as linear regression, Gaussian process model, or autoregressive moving average model (ARMA), the normalized driving behavior data is treated as a time series and trained, and the data values ​​of the missing time points in the time series are predicted. Based on the predicted data values ​​of the missing time points, the time series is interpolated to obtain more complete and uniform sequence data in the time dimension.

[0111] In step S703, the time series data is clustered and divided into different clusters by using an outlier removal method based on a clustering algorithm. The outliers in the time series data are determined based on the distance from the time series data in different clusters to the cluster center, and the outliers are removed to obtain the preprocessed driving behavior data.

[0112] For example, cluster analysis is performed on time series data using a clustering algorithm (such as K-means or DBSCAN) to divide the time series data into several clusters. That is, by analyzing the similarity of data features, similar data points are grouped into the same cluster. Furthermore, the abnormality of the data is judged based on the distance from the data points in each cluster to the cluster center. That is, data points with excessively large distances are regarded as outliers, and the outliers are removed or corrected to obtain preprocessed driving behavior data.

[0113] In this embodiment, based on pre-processing operations such as normalization processing, interpolation processing, and outlier deletion processing on the driving behavior data, the data accuracy and data reliability of the driving behavior data are improved.

[0114] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0115] Based on the same inventive concept, embodiments of the present application also provide a multi-category unsafe driving behavior detection device for implementing the multi-category unsafe driving behavior detection method described above. The implementation solution provided by this device is similar to the implementation solution described in the method described above. Therefore, the specific limitations of one or more embodiments of the multi-category unsafe driving behavior detection device provided below can be found in the limitations of the multi-category unsafe driving behavior detection method described above and will not be repeated here.

[0116] In an exemplary embodiment, Figure 2 As shown, a multi-category unsafe driving behavior detection device is provided, including: a task generation module 201, an acquisition module 202, a data processing module 203 and a detection module 204, wherein:

[0117] A task generation module 201 is used to determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category;

[0118] Acquisition module 202, configured to acquire driving behavior data of a target subject, wherein the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operational data. The driving behavior physiological data represents data describing the physiological characteristics of the target subject during driving, and the driving behavior operational data represents data describing the target subject's operational behavior of the vehicle during driving.

[0119] The data processing module 203 is used to screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrate the driving behavior data related to the driving behavior detection task using a preset inter-modality data fusion algorithm to obtain modality set data;

[0120] The detection module 204 is used to obtain the time series neural network model corresponding to the driving behavior detection task, input the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0121] In an exemplary embodiment, the device also includes a pairing module, which is used to: obtain type label data corresponding to different driving behavior categories, the type label data corresponding to the driving behavior category represents data for respectively labeling safe behavior states and unsafe behavior states in the driving behavior category; calculate the Pearson correlation coefficient between the data features corresponding to each driving behavior data and each type label data through a Pearson correlation coefficient algorithm, and obtain the correlation between the data features of each driving behavior data and each type label data; based on the correlation between the data features of each driving behavior data and each type label data, obtain the data features of the driving behavior data that match each driving behavior category from the data features of each driving behavior data; the data processing module 203 is also used to: determine the target data features of the driving behavior data that match the target driving behavior category based on the target driving behavior category corresponding to the driving behavior detection task, and based on the target data features, perform data screening on the driving behavior data to obtain driving behavior data related to the driving behavior detection task.

[0122] In an exemplary embodiment, the data processing module 203 is also used to: take each driving behavior data related to the driving behavior detection task as the first modal data, perform dynamic time warping on each first modal data to obtain each first modal data of the same time step, and use each first modal data of the same time step as the second modal data; perform feature extraction on each second modal data to obtain the features corresponding to each second modal data, perform encoding on the features corresponding to each second modal data to obtain each second modal data of the same data format, and use each second modal data of the same data format as the third modal data; integrate each third modal data into the same representation space to obtain each third modal data in the same representation space, and use each third modal data in the same representation space as the initial modal set data; perform back propagation processing on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data to obtain modal set data.

[0123] In an exemplary embodiment, the data processing module 203 is also used to: determine the relative spatial position of each first modal data in the same space through a preset calibration algorithm; based on the relative spatial position of each first modal data in the same space, map each first modal data to the same coordinate system to obtain the first modal data in the same coordinate system.

[0124] In an exemplary embodiment, the detection module 204 is also used to: divide the modal set data into different time window data according to different time windows, take the time window data corresponding to the same time window as the same sample data, and obtain a sample data set based on different sample data; input the sample data sets into the temporal neural network model for time feature analysis to obtain the driving behavior detection result of the target object in the target driving behavior category; if the driving behavior detection result indicates that the target object is in an unsafe driving behavior state in the target driving behavior category, then generate warning information corresponding to the driving behavior detection result and display the warning information through a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category.

[0125] In an exemplary embodiment, the detection module 204 is further used to: normalize the driving behavior prediction results of the target object to obtain normalized driving behavior detection results; divide the normalized driving behavior detection results into different first time window data according to different time windows, divide the modal set data into different second time window data according to different time windows, integrate the normalized first time window data and the second time window data corresponding to the same time window to obtain integrated data corresponding to the same time window; obtain a prediction time series neural network model corresponding to the driving behavior detection task, input the integrated data corresponding to different time windows into the prediction time series neural network model for time feature analysis, and obtain the driving behavior prediction results of the target object in the target driving behavior category in the future time period.

[0126] In an exemplary embodiment, the data processing module 203 is also used to: normalize the driving behavior data using a preset linear normalization method to obtain normalized driving behavior data; perform time series interpolation processing on the normalized driving behavior data in a preset time series model using an interpolation method based on a machine learning model to obtain time series data corresponding to the normalized driving behavior data; perform cluster analysis on the time series data and divide it into different clusters using an outlier removal method based on a clustering algorithm, determine the outliers in the time series data based on the distance from the time series data in different clusters to the cluster center, and remove the outliers to obtain the preprocessed driving behavior data.

[0127] Each module in the aforementioned multi-category unsafe driving behavior detection device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a computer device's memory in the form of software, allowing the processor to call and execute the corresponding operations of each module.

[0128] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in any of the above embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above embodiments are implemented.

[0130] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0131] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multi-category unsafe driving behavior detection method, characterized in that: The method comprises: Determining a target driving behavior category for detecting a target object among different driving behavior categories, and generating a driving behavior detection task corresponding to the target object based on the target driving behavior category; Acquiring driving behavior data of the target subject, the driving behavior data being multimodal data including driving behavior physiological data and driving behavior operational data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target subject during driving, and the driving behavior operational data represents data describing the target subject's operational behavior of a vehicle during driving; performing data screening on the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrating the driving behavior data related to the driving behavior detection task through a preset inter-modality data fusion algorithm to obtain modality set data; Obtaining a temporal neural network model corresponding to the driving behavior detection task, inputting the modal set data into the temporal neural network model to analyze the temporal characteristics of the modal set data, and obtaining a driving behavior detection result of the target object in the target driving behavior category; The driving behavior data related to the driving behavior detection task are integrated by a preset inter-modal data fusion algorithm to obtain modal set data, including: Each driving behavior data related to the driving behavior detection task is respectively used as the first modal data, and each first modal data is dynamically time-warped to obtain each first modal data of the same time step, and each first modal data of the same time step is respectively used as the second modal data; feature extraction is performed on each second modal data to obtain features corresponding to each second modal data, and encoding is performed on the features corresponding to each second modal data to obtain each second modal data of the same data format, and each second modal data of the same data format is respectively used as the third modal data; each third modal data is integrated into the same representation space to obtain each third modal data in the same representation space, and each third modal data in the same representation space is used as initial modal set data; back-propagation is performed on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data to obtain modal set data.

2. The method according to claim 1, characterized in that Before filtering the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, the method further includes: Acquire type label data corresponding to different driving behavior categories, where the type label data corresponding to the driving behavior categories represent data for respectively labeling safe behavior states and unsafe behavior states in the driving behavior categories; By using the Pearson correlation coefficient algorithm, the Pearson correlation coefficients of the data features corresponding to each driving behavior data and each type of label data are calculated to obtain the correlation between the data features of each driving behavior data and each type of label data; Based on the correlation between the data features of each driving behavior data and each type of label data, the data features of the driving behavior data that match each driving behavior category are obtained from the data features of each driving behavior data; The performing data screening on the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task includes: Based on the target driving behavior category corresponding to the driving behavior detection task, target data features of the driving behavior data matching the target driving behavior category are determined; based on the target data features, the driving behavior data are screened to obtain driving behavior data related to the driving behavior detection task.

3. The method according to claim 1, characterized in that Before performing dynamic time warping on each first modal data, the method further includes: The relative spatial position of each first modal data in the same space is determined by a preset calibration algorithm. Based on the relative spatial position of each first modal data in the same space, each first modal data is mapped to the same coordinate system to obtain the first modal data in the same coordinate system.

4. The method according to claim 1, wherein The step of inputting the modal set data into the temporal neural network model to analyze the temporal characteristics of the modal set data to obtain a driving behavior detection result of the target object in the target driving behavior category includes: Dividing the modal set data into different time window data according to different time windows, taking the time window data corresponding to the same time window as the same sample data, and obtaining a sample data set based on the different sample data; Inputting the sample data sets into the temporal neural network model respectively to perform time feature analysis to obtain a driving behavior detection result of the target object in the target driving behavior category; If the driving behavior detection result indicates that the target object is in an unsafe driving behavior state in the target driving behavior category, a warning message corresponding to the driving behavior detection result is generated and the warning message is displayed in a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category.

5. The method according to claim 1, wherein After obtaining the driving behavior detection result of the target object in the target driving behavior category, the method further includes: Normalizing the driving behavior prediction result of the target object to obtain a normalized driving behavior detection result; Dividing the normalized driving behavior detection results into different first time window data according to different time windows, dividing the modal set data into different second time window data according to different time windows, and integrating the normalized first time window data and second time window data corresponding to the same time window to obtain integrated data corresponding to the same time window; Obtain a prediction time series neural network model corresponding to the driving behavior detection task, input the integrated data corresponding to different time windows into the prediction time series neural network model for time feature analysis, and obtain the driving behavior prediction result of the target object in the target driving behavior category in the future time period.

6. The method according to claim 1, wherein Before the driving behavior data is screened based on the driving behavior detection task, the method further includes: Normalizing the driving behavior data using a preset linear normalization method to obtain normalized driving behavior data; By using an interpolation method based on a machine learning model, in a preset time series model, the normalized driving behavior data is interpolated in time series to obtain time series data corresponding to the normalized driving behavior data; The time series data is clustered and divided into different clusters by an outlier removal method based on a clustering algorithm. The outliers in the time series data are determined based on the distance from the time series data in different clusters to the cluster center, and the outliers are removed to obtain preprocessed driving behavior data.

7. A multi-category unsafe driving behavior detection device, characterized in that: The device comprises: A task generation module is used to determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category; an acquisition module, configured to acquire driving behavior data of the target subject, the driving behavior data being multimodal data including driving behavior physiological data and driving behavior operational data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target subject during driving, and the driving behavior operational data represents data describing the target subject's operational behavior of a vehicle during driving; a data processing module, configured to screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrate the driving behavior data related to the driving behavior detection task using a preset inter-modality data fusion algorithm to obtain modality aggregate data; a detection module, configured to obtain a temporal neural network model corresponding to the driving behavior detection task, input the modal set data into the temporal neural network model, analyze the temporal characteristics of the modal set data, and obtain a driving behavior detection result of the target object in the target driving behavior category; Among them, the data processing module is also used to: take each driving behavior data related to the driving behavior detection task as the first modal data, perform dynamic time warping processing on each first modal data to obtain each first modal data of the same time step, and use each first modal data of the same time step as the second modal data; perform feature extraction on each second modal data to obtain the features corresponding to each second modal data, perform encoding processing on the features corresponding to each second modal data to obtain each second modal data of the same data format, and use each second modal data of the same data format as the third modal data; integrate each third modal data into the same representation space to obtain each third modal data in the same representation space, and use each third modal data in the same representation space as the initial modal set data; perform back propagation processing on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data to obtain modal set data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images