Multi-class unsafe driving behavior detection method and device and computer equipment

Abstract

The invention relates to a multi-class unsafe driving behavior detection method and device and computer equipment. The method comprises the steps of determining a target driving behavior category for detecting a target object in different driving behavior categories, and generating a driving behavior detection task based on the target driving behavior category; obtaining driving behavior data of the target object; performing data screening on the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrating the driving behavior data related to the driving behavior detection task through an inter-modal data fusion algorithm to obtain modal set data; and inputting the modal set data into the time sequence neural network model to analyze the time characteristics of the modal set data to obtain a driving behavior detection result of the target object in the target driving behavior category. By adopting the method, the unsafe driving behaviors can be efficiently, accurately and diverse detected.

Description

Technical Field

[0001] The present application relates to the field of driving safety technology, and in particular to a method, device and computer equipment for detecting multiple categories of unsafe driving behaviors. Background Art

[0002] In the field of driving safety technology, an unsafe driving behavior detection system is used to detect the driver's unsafe driving behavior to ensure the driving safety of the driver and passengers.

[0003] In the related art, in one type of unsafe driving behavior detection system, unsafe driving behavior is detected only based on image data. However, on the one hand, it is difficult to ensure the image quality in a dim environment; on the other hand, it is difficult to ensure multi-dimensional detection of the driver's unsafe driving behavior.

[0004] In another type of unsafe driving behavior detection system, unsafe driving behaviors are detected based on multimodal data. However, on the one hand, only one specific category of unsafe driving behavior is detected, and it is impossible for one system to detect and warn multiple unsafe driving behaviors. On the other hand, detection is only based on the driver's current state, and the time continuity and time dependence of driving behavior are not taken into account.

[0005] Based on this, it is difficult for the unsafe driving behavior detection system in the related art to detect the driver's unsafe driving behavior efficiently, accurately and diversely. Summary of the invention

[0006] Based on this, it is necessary to provide a multi-category unsafe driving behavior detection method, device, computer equipment and computer-readable storage medium to address the above technical problems, so as to detect unsafe driving behaviors efficiently, accurately and diversely.

[0007] In a first aspect, the present application provides a multi-category unsafe driving behavior detection method, comprising: Determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category; Acquire driving behavior data of the target object, wherein the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operation data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target object during driving, and the driving behavior operation data represents data describing the target object's operation behavior of a vehicle during driving; Based on the driving behavior detection task, the driving behavior data is screened to obtain driving behavior data related to the driving behavior detection task, and the driving behavior data related to the driving behavior detection task is integrated through a preset inter-modality data fusion algorithm to obtain modality set data; Obtain a time series neural network model corresponding to the driving behavior detection task, input the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0008] In a second aspect, the present application also provides a multi-category unsafe driving behavior detection device, comprising: A task generation module, used to determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category; an acquisition module, configured to acquire driving behavior data of the target object, wherein the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operation data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target object during driving, and the driving behavior operation data represents data describing the target object's operation behavior of a vehicle during driving; a data processing module, configured to screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrate the driving behavior data related to the driving behavior detection task through a preset inter-modality data fusion algorithm to obtain modality set data; The detection module is used to obtain the time series neural network model corresponding to the driving behavior detection task, input the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0009] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above steps when executing the computer program.

[0010] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above steps when executed by a processor.

[0011] The above-mentioned multi-category unsafe driving behavior detection method, device, computer equipment and computer-readable storage medium firstly, in different driving behavior categories, the target driving behavior category for detecting the target object is determined in a diverse and selective manner, and a driving behavior detection task is generated based on the target driving behavior category; secondly, multi-modal driving behavior data of the target object is obtained, and the driving behavior data is screened based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, thereby reducing the amount of data calculation and improving the detection accuracy and efficiency; thirdly, the screened driving behavior data is integrated through an inter-modal data fusion algorithm to obtain modal set data, thereby improving the accuracy of the data; finally, the modal set data is input into a temporal neural network model for time feature analysis to obtain a driving behavior detection result, thereby accurately, efficiently and adaptively obtaining the driving behavior detection result of the target object in the target driving behavior category in the time feature dimension based on the time continuity and time dependence of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 1 is a flow chart of a method for detecting multiple categories of unsafe driving behaviors in one embodiment; Figure 2 1 is a structural block diagram of a device for detecting multiple categories of unsafe driving behaviors in one embodiment. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0015] In one embodiment, Figure 1 As shown, a multi-category unsafe driving behavior detection method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps S101 to S103.

[0016] Step S101 : determining a target driving behavior category for detecting a target object among different driving behavior categories, and generating a driving behavior detection task corresponding to the target object based on the target driving behavior category.

[0017] Among them, unsafe driving behavior refers to the behavior taken by the driver during the driving process that may cause traffic accidents or affect traffic safety, such as speeding, running red lights, frequent lane changes, drunk driving, fatigue driving, distracted driving, etc.

[0018] Among them, the driving behavior category represents the classification of driving behavior according to specific behavioral characteristics, which is used to distinguish the driver's operating status or behavioral performance during driving; a driving behavior category may include at least one safe driving behavior state and one unsafe driving behavior state. For example, when the driving behavior category is "speeding", the driving behavior category may include the safe driving behavior state of "not speeding" and the unsafe driving behavior state of "speeding"; the target driving behavior category represents the driving behavior category to be detected.

[0019] The target object represents the specific driver who needs to undergo driving behavior detection.

[0020] Among them, the driving behavior detection task refers to the specific detection target and analysis task generated for the target object's driving behavior based on a specific driving behavior category, which is used to perform risk assessment on the target object's driving behavior.

[0021] Exemplarily, the target driving behavior category for detection of the target object can be determined by manual setting or intelligent calibration. For example, the driving behavior category determination criteria for different application scenarios can be pre-set. For example, for a logistics fleet, "speeding", "fatigue driving", "frequent emergency braking" and other target driving behavior categories can be set as key detection targets; the trigger conditions corresponding to different driving behavior categories can be pre-selected and set. For example, the trigger condition corresponding to judging "whether speeding" can be set to "trigger when the vehicle speed exceeds the road speed limit", or the trigger condition corresponding to judging "whether fatigue driving" can be set to "trigger after continuous driving for more than 4 hours".

[0022] Step S102, obtaining driving behavior data of the target object, where the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operational data, where the driving behavior physiological data represents data describing the physiological characteristics of the target object during driving, and the driving behavior operational data represents data describing the target object's operational behavior of a vehicle during driving.

[0023] The driving behavior data refers to a set of information describing the behavior characteristics and states exhibited by the driver during driving.

[0024] The driving behavior physiological data refers to physiological signal data describing the driver's physiological state, such as eye movement data, heart rate data, breathing data, body temperature data, etc.

[0025] The driving behavior operation data refers to the vehicle driving data describing the driver's control behavior of the vehicle, such as speed, acceleration, steering wheel angle, and the distance between the vehicle and the side margins of the lane.

[0026] For example, driving physiological data can be collected through wearable devices (such as smart watches, smart bracelets, etc.), vehicle-mounted biosensors (such as cameras, heart rate monitors, etc.), and other devices; driving behavior operation data can be collected through vehicle-mounted sensors (such as speed sensors, steering wheel angle sensors, vehicle distance sensors, etc.).

[0027] Step S103, based on the driving behavior detection task, the driving behavior data is screened to obtain driving behavior data related to the driving behavior detection task, and the driving behavior data related to the driving behavior detection task is integrated through a preset inter-modal data fusion algorithm to obtain modal set data.

[0028] Among them, the inter-modal data fusion algorithm refers to an algorithm used to integrate and process data of different modalities (such as different data sources or different data types, etc.).

[0029] Among them, modal set data represents the unified data expression after feature extraction and fusion of different modal data through inter-modal data fusion algorithm.

[0030] Exemplarily, according to the specific scenario requirements of the driving behavior detection task, driving behavior data that is highly relevant to the driving behavior detection task can be screened out from the multimodal driving behavior data; wherein the screening process includes parsing the goals and conditions of the driving behavior detection task, combining the data type, data characteristics and specific indicators of the driving behavior data, and screening and extracting valid data from the driving behavior data through preset matching rules.

[0031] Exemplarily, a preset inter-modal data fusion algorithm is used to integrate and process the screened driving behavior data related to the driving behavior detection task; wherein the integration process includes time alignment, feature extraction and weight allocation of driving behavior data of different modalities, and unified feature expression is achieved through feature splicing or deep learning models.

[0032] Step S104, obtaining a time series neural network model corresponding to the driving behavior detection task, inputting the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtaining the driving behavior detection result of the target object in the target driving behavior category.

[0033] Among them, the time series neural network model represents a deep learning model for processing time series data. It obtains the driving behavior detection results corresponding to the target object by capturing the dependencies and dynamic change patterns of the data in the time dimension.

[0034] Among them, the temporal characteristics of the modal set data represent the dynamic change information presented by the modal set data in the time dimension, reflecting the changing trend of driving behavior in different time periods.

[0035] Among them, the driving behavior detection result represents the conclusive information obtained after performing time characteristic analysis on the input modal set data through the time series neural network model, which is used to characterize the specific state or risk assessment of the target object corresponding to the target driving behavior category.

[0036] For example, the time characteristics in the time dimension of the modal set data can be effectively captured through a long short-term memory network (LSTM) or a deep learning network (Transformer) based on an attention mechanism, and the time characteristics in the modal set data can be deeply mined through a time series analysis method to extract the dynamic change patterns of the data and the time dependencies between the modalities. Through this analysis process, the behavior patterns, trend changes, and potential abnormal states of the target object during driving can be identified, thereby generating driving behavior detection results for the target driving behavior category.

[0037] Optionally, the driving behavior detection result is presented in the form of classification, risk score or abnormal mark, which specifically describes the driving behavior status of the target object under the driving behavior category.

[0038] Optionally, different driving behavior detection tasks correspond to different time series neural network models, that is, different time series neural network models are trained based on different driving behavior detection tasks, so that according to the driving behavior detection tasks generated in the actual detection process, the matching time series neural network model is called, thereby realizing the selective detection of different types of driving behaviors.

[0039] In the above-mentioned multi-category unsafe driving behavior detection method, first, in different driving behavior categories, the target driving behavior category for detecting the target object is determined in a diverse and selective manner, and a driving behavior detection task is generated based on the target driving behavior category; secondly, multi-modal driving behavior data of the target object is obtained, and the driving behavior data is screened based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, thereby reducing the amount of data calculation and improving the detection accuracy and efficiency; thirdly, the screened driving behavior data is integrated through an inter-modal data fusion algorithm to obtain modal set data, thereby improving the accuracy of the data; finally, the modal set data is input into a temporal neural network model for time feature analysis to obtain a driving behavior detection result, thereby accurately, efficiently and adaptively obtaining the driving behavior detection result of the target object in the target driving behavior category in the time feature dimension based on the time continuity and time dependence of the data.

[0040] In an exemplary embodiment, before performing data screening on the driving behavior data based on the driving behavior detection task to obtain the driving behavior data related to the driving behavior detection task, the method further includes steps S201 to S203; performing data screening on the driving behavior data based on the driving behavior detection task to obtain the driving behavior data related to the driving behavior detection task, including step S204.

[0041] Step S201 , obtaining type label data corresponding to different driving behavior categories, where the type label data corresponding to the driving behavior category represents data for marking the safe behavior state and the unsafe behavior state in the driving behavior category, respectively.

[0042] Exemplarily, a driving behavior category may correspond to at least two type label data, namely, type label data corresponding to a safe driving behavior state and type label data corresponding to an unsafe driving behavior state. For example, when the driving behavior category is "speeding", the driving behavior category corresponds to type label data marking the safe driving behavior state of "not speeding" and type label data marking the unsafe driving behavior state of "speeding".

[0043] Step S202, using the Pearson correlation coefficient algorithm, calculate the Pearson correlation coefficients between the data features corresponding to each driving behavior data and each type of label data, and obtain the correlations between the data features of each driving behavior data and each type of label data.

[0044] Among them, the Pearson correlation coefficient algorithm represents a statistical method for measuring the degree of linear correlation between two variables; the Pearson correlation coefficient represents the degree of correlation between the data feature vector of the driving behavior data and the type label data of the driving behavior category.

[0045] For example, setting For the data feature vector corresponding to a certain mode of driving behavior data, set is the feature vector of each type of label data of a certain driving behavior category, then the correlation between X and Y is calculated as follows: (1) In formula (1), Indicates the Pearson correlation coefficient corresponding to X and Y, which is used to characterize the degree of correlation between X and Y; represents the i-th element in X, represents the i-th element in Y, represents the mean of each element in X, represents the mean of each element in Y, and n represents the number of data points.

[0046] Based on formula (1), the correlation between a data feature of a certain mode of driving behavior data and the type label data of a certain driving behavior category is obtained, and then the correlation between the data feature and the driving behavior category is obtained; based on this, the driving behavior data of each mode and each driving behavior category are traversed through formula (1) to obtain the correlation between the data feature of any mode of driving behavior data and any driving behavior category; a correlation coefficient matrix can be formed to record the correlation coefficient of each pair of data.

[0047] Step S203 , based on the correlation between the data features of each driving behavior data and each type of label data, the data features of the driving behavior data that match each driving behavior category are obtained from the data features of each driving behavior data.

[0048] Exemplarily, the correlation threshold conditions for pairing data features of driving behavior data and driving behavior categories can be determined based on data distribution and business needs. For example, a pairing relationship can be established between data features of driving behavior data and driving behavior categories whose correlation coefficients are greater than or equal to a preset threshold; or the correlation coefficients of each pair of data can be sorted in descending order, and a pairing relationship can be established between data features of driving behavior data and driving behavior categories whose correlation coefficients are less than or equal to a preset threshold.

[0049] Step S204, based on the target driving behavior category corresponding to the driving behavior detection task, determine the target data features of the driving behavior data matching the target driving behavior category, and based on the target data features, perform data screening on the driving behavior data to obtain driving behavior data related to the driving behavior detection task.

[0050] Exemplarily, a target data feature that is paired with a target driving behavior category is determined, and based on the target data feature, the driving behavior data is screened, that is, the driving behavior data belonging to the target data feature is retained, and the driving behavior data belonging to data features other than the target data feature is removed, thereby screening out driving behavior data related to the driving behavior detection task.

[0051] For example, if the Pearson correlation coefficient between the "driver's eye state" data feature in the image modal data and the type label data of the "whether fatigue driving" driving behavior category meets the correlation threshold condition, then a pairing relationship between the "driver's eye state" data feature in the image modal data and the "whether fatigue driving" driving behavior category is constructed; in the driving behavior detection task corresponding to the "whether fatigue driving" driving behavior category, the image modal data corresponding to the "driver's eye state" data feature is preferentially selected, and then the data corresponding to other data features are selected.

[0052] In this embodiment, based on the Pearson correlation coefficient algorithm, the correlation between the data features of the driving behavior data of any modality and any driving behavior category is determined in advance, so as to construct a pairing relationship between each driving behavior category and the data features of the specified driving behavior data. Based on the pre-constructed pairing relationship, the driving behavior data is efficiently and accurately screened to obtain the driving behavior data related to the driving behavior detection task, thereby reducing the amount of data calculation.

[0053] In an exemplary embodiment, the driving behavior data related to the driving behavior detection task is integrated through a preset inter-modality data fusion algorithm to obtain modality set data, including steps S301 to S304.

[0054] Step S301, taking each driving behavior data related to the driving behavior detection task as the first modal data, performing dynamic time warping on each first modal data to obtain each first modal data of the same time step, and taking each first modal data of the same time step as the second modal data.

[0055] Among them, Dynamic Time Warping (DTW) is a method for aligning time series data.

[0056] For example, since the first modal data of different modalities may have different sampling rates, durations or time offsets, it is necessary to adjust and match the first modal data on a time scale through a DTW algorithm so as to unify the various first modal data to the same time step to obtain the second modal data, thereby ensuring that the subsequent feature extraction and encoding processing of the various second modal data can be performed in a consistent time frame.

[0057] Optionally, through dynamic time warping, the time gap between each modal data can be automatically calculated, and the low-frequency modal data can be interpolated so that all data maintain the same time step.

[0058] Step S302: extract features from each piece of second modal data to obtain features corresponding to each piece of second modal data, encode the features corresponding to each piece of second modal data to obtain each piece of second modal data in the same data format, and use each piece of second modal data in the same data format as third modal data.

[0059] Exemplarily, features that can effectively represent data characteristics are extracted from the second modality data. For example, for facial video data, surface features such as image edges and colors and deep features such as driver emotions are extracted. For eye movement data, features such as pupil size, gaze point coordinates and pupil size are extracted. For acceleration data, features such as acceleration mean, variance, and frequency domain are extracted. Each extracted feature is encoded, that is, the feature value is mapped to a specific numerical range or format, so that the second modality data of different modalities are converted into the third modality data of the same data format.

[0060] Step S303: Integrate each third modality data into the same representation space to obtain each third modality data in the same representation space, and use each third modality data in the same representation space as initial modality set data.

[0061] Exemplarily, each third modality data can be integrated into the same structured framework, such as a table, matrix, tensor, etc., to obtain initial modality set data in the same representation space, and each data in the initial modality set data is presented in the same representation space in the same data format and time alignment.

[0062] Step S304, performing back propagation processing on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data, and obtaining the modal set data.

[0063] Among them, back propagation represents a data processing method for iteratively optimizing data, which is used to improve the quality of the initial modal set data.

[0064] Among them, inconsistent features refer to conflicting features in the initial modal set data caused by problems such as sensor noise and differences in data sources. For example, sensor acquisition noise causes data distortion, or the eye movement sensor and facial video sensor may simultaneously collect feature information on pupil size, but due to different data sources, these features may have inconsistent calculation results.

[0065] Among them, repeated features mean that in the initial modal set data, data from different modalities have the same or highly similar meanings, resulting in information redundancy; for example, eye movement sensors and facial video sensors may collect feature information of pupil size at the same time, but due to different data sources, these features may be repeated.

[0066] Exemplarily, the initial modal set data is subjected to back-propagation processing, that is, by defining an optimization objective (such as minimizing the redundancy or contradiction between features), the degree of conflict or redundancy between data features is calculated; low-quality features are eliminated through multiple iterations, and these inconsistent and repeated features are gradually removed, thereby gradually optimizing the initial modal set data to obtain optimized modal set data.

[0067] In this embodiment, first, the first modal data is dynamically time-warped to obtain second modal data of the same time step, thereby increasing the operability of the data in the dimension of time alignment; secondly, the features of the second modal data are encoded to obtain third modal data of the same data format, thereby increasing the operability of the data in the dimension of data format; thirdly, the third modal data are integrated into the same representation space to obtain initial modal set data in the same representation space, thereby ensuring the integrity and structuring of the data; finally, the initial modal set data is back-propagated to remove inconsistent features and repeated features to obtain modal set data with no information conflict and no duplication, thereby improving data quality and contributing to the accuracy of detection results.

[0068] In an exemplary embodiment, before performing dynamic time warping processing on each first modal data, the method further includes step S401.

[0069] Step S401, determining the relative spatial position of each first modal data in the same space through a preset calibration algorithm, and mapping each first modal data to the same coordinate system based on the relative spatial position of each first modal data in the same space to obtain the first modal data in the same coordinate system.

[0070] Among them, the calibration algorithm refers to the algorithm used to determine the relative spatial position and relationship of different modal data sources (such as sensors). It maps all data into a unified spatial framework by analyzing the geometric layout, relative position or angle deviation between sensors.

[0071] Exemplarily, a calibration algorithm is used to calculate and determine the relative spatial position of each first modal data in the same space, wherein the first modal data of each modality (such as the driver's eye movement data, facial video data, steering wheel operation data, etc.) comes from different sensors, and these data may have different acquisition reference systems or coordinate representations in space. Through the calibration algorithm, the spatial relationship between the first modal data of each modality, such as the installation angle, acquisition range or positioning error of the sensor, can be accurately estimated.

[0072] After determining the spatial relationship between the first modal data of each modality, all the first modal data are spatially aligned and mapped to the same coordinate system, for example, through a coordinate transformation matrix or a geometric transformation method, thereby ensuring that the first modal data of different modalities can be represented and analyzed in a unified spatial framework.

[0073] In this embodiment, the first modality data from different sources are unified into the same coordinate system, so that the first modality data of different modalities can be processed in a unified spatial framework, thereby ensuring the operability of the data.

[0074] In an exemplary embodiment, the modal set data is input into a temporal neural network model to analyze the temporal characteristics of the modal set data to obtain a driving behavior detection result of the target object in the target driving behavior category, including steps S501 to S503.

[0075] Step S501, divide the modal set data into different time window data according to different time windows, take the time window data corresponding to the same time window as the same sample data, and obtain a sample data set based on different sample data.

[0076] Among them, the time window represents the time period used to divide the continuous time series data in time series analysis; the time window data corresponding to the same time window corresponds to all modal set data within a time window, that is, it is used to describe the behavioral characteristics of the target object within the time period to which the time window belongs.

[0077] Exemplarily, according to different time windows, the modal set data is divided into time window data corresponding to different time windows, that is, the modal set data is divided into different data blocks according to different time periods, and each data block can reflect the driving behavior characteristics within the time period; the time window data corresponding to the same time window are merged into a complete and independent sample data, and a sample data set is constructed based on the sample data corresponding to each time window.

[0078] Optionally, the modal set data can be divided into time window data corresponding to different time windows according to different time windows and time steps, wherein the time step represents the time interval each time the time window is moved, which determines the coverage density of the time window. The shorter the time step, the more overlaps between time windows and the denser the data coverage.

[0079] Step S502: input the sample data sets into the time series neural network model for time feature analysis to obtain the driving behavior detection result of the target object in the target driving behavior category.

[0080] Exemplarily, the time series neural network model performs time feature analysis on the sample data set, that is, performs time feature analysis on the sample data corresponding to each time window in the sample data set, so as to capture the dynamic changes of driving behavior between different time periods.

[0081] Step S503, if the driving behavior detection result indicates that the target object is in an unsafe driving behavior state in the target driving behavior category, a warning message corresponding to the driving behavior detection result is generated and the warning message is displayed in a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category.

[0082] Among them, the warning information refers to a prompt or warning content generated when it is detected that the target object is in an unsafe driving behavior state in the target driving behavior category, which is used to remind the target object to take measures to correct or avoid potential dangers.

[0083] Exemplarily, the warning information may include a specific description of the driving behavior detection results (such as "currently in a fatigue driving state") and recommended measures (such as "recommended to take a rest" or "slow down"); the generated warning information can be displayed through preset reminder methods, such as in-car voice reminders, screen prompts or vibration feedback, etc., to attract the driver's attention; the warning information will continue to be displayed until it is detected that the driver has returned to a safe driving behavior state, thereby ensuring driving safety.

[0084] In this embodiment, on the one hand, the modal set data is divided into different time window data according to different time windows, the time window data corresponding to the same time window is used as the same sample data, and a sample data set is obtained based on the different sample data. The sample data sets are respectively input into the time series neural network model for time feature analysis, so as to efficiently and accurately capture the dynamic change law of driving behavior between different time periods to improve the reliability of the driving behavior detection results; on the other hand, warning information is generated based on the driving behavior detection results and displayed through a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category, thereby improving the safety of the driving process based on the real-time feedback mechanism.

[0085] In an exemplary embodiment, after obtaining the driving behavior detection result of the target object in the target driving behavior category, the method further includes steps S601 to S603.

[0086] Step S601 : normalize the driving behavior prediction result of the target object to obtain a normalized driving behavior prediction result.

[0087] Exemplarily, the driving behavior prediction results of the target object are normalized and mapped to a uniform interval (eg, [0, 1] or [-1, 1]) in the form of a time series to eliminate differences in data dimensions, ranges, and distributions.

[0088] Step S602: divide the normalized driving behavior prediction results into different first time window data according to different time windows, divide the modal set data into different second time window data according to different time windows, and integrate the normalized first time window data and second time window data corresponding to the same time window to obtain integrated data corresponding to the same time window.

[0089] Exemplarily, the normalized driving behavior prediction results and modal set data are divided into different time window data according to different time windows, respectively, to obtain the first time window data corresponding to the normalized driving behavior prediction results in different time windows, and the second time window data corresponding to the modal set data in different time windows; the first time window data and the second time window data corresponding to the same time window are integrated to generate integrated data corresponding to the time window, thereby obtaining the integrated data corresponding to each time window.

[0090] Step S603, obtain the prediction time series neural network model corresponding to the driving behavior detection task, input the integrated data corresponding to different time windows into the prediction time series neural network model for time feature analysis, and obtain the driving behavior prediction result of the target object in the target driving behavior category in the future time period.

[0091] Among them, the predictive time series neural network model represents a deep learning model for processing time series data. It obtains the driving behavior prediction results of the target object in the future period by capturing the dependencies and dynamic change patterns of the data in the time dimension.

[0092] Among them, the driving behavior prediction result represents the conclusive information obtained after the time feature analysis of the input integrated data through the predictive time series neural network model, which is used to characterize the specific state or risk assessment corresponding to the target driving behavior category of the target object in the future period.

[0093] Exemplarily, the integrated data corresponding to different time windows are input into the predictive time series neural network model for time feature analysis. That is, according to the dependency relationship of the integrated data corresponding to different time windows in the time series, the predictive time series neural network model performs time feature extraction and trend analysis on the integrated data to predict the driving behavior state corresponding to the target driving behavior category of the target object in the future time period.

[0094] In this embodiment, the time window data corresponding to the driving behavior prediction results and the modal set data are integrated according to different time windows, so as to comprehensively obtain the integrated data corresponding to each time window, and the time feature analysis is performed based on the integrated data corresponding to each time window to obtain the driving behavior prediction results of the target object in the future period, thereby improving the scene diversity of driving behavior detection.

[0095] In an exemplary embodiment, before performing data screening on the driving behavior data based on the driving behavior detection task, the method further includes steps S701 to S703.

[0096] Step S701 , normalizing the driving behavior data using a preset linear normalization method to obtain normalized driving behavior data.

[0097] Exemplarily, the driving behavior data is normalized by a linear normalization method to map the values ​​of driving behavior data of different modes into a unified interval range, so that the normalized driving behavior data has better numerical consistency.

[0098] Step S702, through the interpolation method based on the machine learning model, in a preset time series model, the normalized driving behavior data is interpolated in time series to obtain time series data corresponding to the normalized driving behavior data.

[0099] Exemplarily, in a preset time series model, such as linear regression, Gaussian process model or autoregressive moving average model (ARMA), the normalized driving behavior data is taken as a time series and trained, and the data values ​​of the missing time points in the time series are predicted. Based on the predicted data values ​​of the missing time points, the time series is interpolated to obtain more complete and uniform sequence data in the time dimension.

[0100] Step S703, clustering analysis is performed on the time series data and divided into different clusters through an outlier removal method based on a clustering algorithm, outliers in the time series data are determined according to the distance from the time series data in different clusters to the cluster center, and outliers are removed to obtain preprocessed driving behavior data.

[0101] Exemplarily, cluster analysis is performed on time series data through a clustering algorithm (such as K-means or DBSCAN) to divide the time series data into several clusters, that is, by analyzing the similarity of data features, similar data points are grouped into the same cluster; secondly, the abnormality of the data is judged based on the distance from the data point in each cluster to the cluster center, that is, data points with too large distance are regarded as outliers, and the outliers are removed or corrected to obtain the preprocessed driving behavior data.

[0102] In this embodiment, based on pre-processing operations such as normalization processing, interpolation processing, and outlier deletion processing of the driving behavior data, the data accuracy and data reliability of the driving behavior data are improved.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides a multi-category unsafe driving behavior detection device for implementing the multi-category unsafe driving behavior detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-category unsafe driving behavior detection device embodiments provided below can refer to the limitations of the multi-category unsafe driving behavior detection method above, and will not be repeated here.

[0105] In an exemplary embodiment, Figure 2 As shown, a multi-category unsafe driving behavior detection device is provided, including: a task generation module 201, an acquisition module 202, a data processing module 203 and a detection module 204, wherein: A task generation module 201 is used to determine a target driving behavior category for detecting a target object among different driving behavior categories, and to generate a driving behavior detection task corresponding to the target object based on the target driving behavior category; An acquisition module 202 is used to acquire driving behavior data of a target object, where the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operation data, where the driving behavior physiological data represents data describing physiological characteristics of the target object during driving, and the driving behavior operation data represents data describing the target object's operation behavior of a vehicle during driving; The data processing module 203 is used to screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrate the driving behavior data related to the driving behavior detection task through a preset inter-modality data fusion algorithm to obtain modality set data; The detection module 204 is used to obtain the time series neural network model corresponding to the driving behavior detection task, input the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

[0106] In an exemplary embodiment, the device also includes a pairing module, which is used to: obtain type label data corresponding to different driving behavior categories, the type label data corresponding to the driving behavior category represents data for marking the safe behavior state and the unsafe behavior state in the driving behavior category respectively; calculate the Pearson correlation coefficient between the data features corresponding to each driving behavior data and each type label data respectively through the Pearson correlation coefficient algorithm, and obtain the correlation between the data features of each driving behavior data and each type label data respectively; based on the correlation between the data features of each driving behavior data and each type label data respectively, obtain the data features of the driving behavior data that match each driving behavior category respectively in the data features of each driving behavior data; the data processing module 203 is also used to: based on the target driving behavior category corresponding to the driving behavior detection task, determine the target data features of the driving behavior data that match the target driving behavior category, and based on the target data features, perform data screening on the driving behavior data to obtain driving behavior data related to the driving behavior detection task.

[0107] In an exemplary embodiment, the data processing module 203 is also used to: take each driving behavior data related to the driving behavior detection task as the first modal data, perform dynamic time warping on each first modal data to obtain each first modal data of the same time step, and take each first modal data of the same time step as the second modal data; perform feature extraction on each second modal data to obtain the features corresponding to each second modal data, perform encoding on the features corresponding to each second modal data to obtain each second modal data of the same data format, and take each second modal data of the same data format as the third modal data; integrate each third modal data into the same representation space to obtain each third modal data in the same representation space, and take each third modal data in the same representation space as the initial modal set data; perform back propagation processing on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data to obtain modal set data.

[0108] In an exemplary embodiment, the data processing module 203 is also used to: determine the relative spatial position of each first modal data in the same space through a preset calibration algorithm, and map each first modal data to the same coordinate system based on the relative spatial position of each first modal data in the same space to obtain the first modal data in the same coordinate system.

[0109] In an exemplary embodiment, the detection module 204 is also used to: divide the modal set data into different time window data according to different time windows, take the time window data corresponding to the same time window as the same sample data, and obtain a sample data set based on different sample data; input the sample data sets into the time series neural network model for time feature analysis to obtain the driving behavior detection result of the target object in the target driving behavior category; if the driving behavior detection result indicates that the target object is in an unsafe driving behavior state in the target driving behavior category, then generate warning information corresponding to the driving behavior detection result and display the warning information through a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category.

[0110] In an exemplary embodiment, the detection module 204 is also used to: normalize the driving behavior prediction results of the target object to obtain normalized driving behavior detection results; divide the normalized driving behavior detection results into different first time window data according to different time windows, divide the modal set data into different second time window data according to different time windows, integrate the normalized first time window data and the second time window data corresponding to the same time window, and obtain integrated data corresponding to the same time window; obtain the prediction time series neural network model corresponding to the driving behavior detection task, and input the integrated data corresponding to different time windows into the prediction time series neural network model for time feature analysis to obtain the driving behavior prediction results of the target object in the target driving behavior category in the future time period.

[0111] In an exemplary embodiment, the data processing module 203 is also used to: normalize the driving behavior data by a preset linear normalization method to obtain normalized driving behavior data; perform time series interpolation processing on the normalized driving behavior data in a preset time series model by an interpolation method based on a machine learning model to obtain time series data corresponding to the normalized driving behavior data; perform cluster analysis on the time series data and divide it into different clusters by an outlier removal method based on a clustering algorithm, determine the outliers in the time series data according to the distance from the time series data in different clusters to the cluster center, and remove the outliers to obtain the preprocessed driving behavior data.

[0112] Each module in the above-mentioned multi-category unsafe driving behavior detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0113] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in any of the above embodiments when executing the computer program.

[0114] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above embodiments are implemented.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0116] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-category unsafe driving behavior detection method, characterized in that: The method comprises: Determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category; Acquire driving behavior data of the target object, wherein the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operation data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target object during driving, and the driving behavior operation data represents data describing the target object's operation behavior of a vehicle during driving; Based on the driving behavior detection task, the driving behavior data is screened to obtain driving behavior data related to the driving behavior detection task, and the driving behavior data related to the driving behavior detection task is integrated through a preset inter-modality data fusion algorithm to obtain modality set data; Obtain a time series neural network model corresponding to the driving behavior detection task, input the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

2. The method according to claim 1, characterized in that Before the driving behavior data is screened based on the driving behavior detection task to obtain the driving behavior data related to the driving behavior detection task, the method further includes: Acquire type label data corresponding to different driving behavior categories, wherein the type label data corresponding to the driving behavior category represents data for marking safe behavior states and unsafe behavior states in the driving behavior category respectively; By using the Pearson correlation coefficient algorithm, the Pearson correlation coefficients of the data features corresponding to each driving behavior data and each type of label data are calculated to obtain the correlation between the data features of each driving behavior data and each type of label data; Based on the correlation between the data features of each driving behavior data and each type of label data, the data features of the driving behavior data respectively matching each driving behavior category are obtained from the data features of each driving behavior data; The step of screening the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task includes: Based on the target driving behavior category corresponding to the driving behavior detection task, the target data features of the driving behavior data matching the target driving behavior category are determined, and based on the target data features, the driving behavior data are screened to obtain the driving behavior data related to the driving behavior detection task.

3. The method according to claim 1, characterized in that The driving behavior data related to the driving behavior detection task are integrated by a preset inter-modality data fusion algorithm to obtain modality set data, including: Taking each driving behavior data related to the driving behavior detection task as first modal data, performing dynamic time warping processing on each first modal data to obtain each first modal data of the same time step, and taking each first modal data of the same time step as second modal data; Extract features from each of the second modal data to obtain features corresponding to each of the second modal data, encode the features corresponding to each of the second modal data to obtain each of the second modal data in the same data format, and use each of the second modal data in the same data format as the third modal data; Integrate each third modality data into the same representation space to obtain each third modality data in the same representation space, and use each third modality data in the same representation space as initial modality set data; Back propagation processing is performed on the initial modal set data to remove inconsistent features and repeated features in the initial modal set data to obtain modal set data.

4. The method according to claim 3, characterized in that Before performing dynamic time warping processing on each first modal data, the method further includes: The relative spatial position of each first modal data in the same space is determined by a preset calibration algorithm, and based on the relative spatial position of each first modal data in the same space, each first modal data is mapped to the same coordinate system to obtain the first modal data in the same coordinate system.

5. The method according to claim 1, characterized in that The step of inputting the modal set data into the temporal neural network model to analyze the temporal characteristics of the modal set data to obtain a driving behavior detection result of the target object in the target driving behavior category includes: Dividing the modality set data into different time window data according to different time windows, taking the time window data corresponding to the same time window as the same sample data, and obtaining a sample data set based on the different sample data; Inputting the sample data sets into the temporal neural network model respectively to perform time feature analysis to obtain the driving behavior detection result of the target object in the target driving behavior category; If the driving behavior detection result indicates that the target object is in an unsafe driving behavior state in the target driving behavior category, a warning message corresponding to the driving behavior detection result is generated and the warning message is displayed in a preset reminder method until it is detected that the target object is in a safe driving behavior state in the target driving behavior category.

6. The method according to claim 1, characterized in that After obtaining the driving behavior detection result of the target object in the target driving behavior category, the method further includes: Normalizing the driving behavior prediction result of the target object to obtain a normalized driving behavior detection result; Dividing the normalized driving behavior detection results into different first time window data according to different time windows, dividing the modal set data into different second time window data according to different time windows, and integrating the normalized first time window data and the second time window data corresponding to the same time window to obtain integrated data corresponding to the same time window; Obtain a prediction time series neural network model corresponding to the driving behavior detection task, input the integrated data corresponding to different time windows into the prediction time series neural network model for time feature analysis, and obtain the driving behavior prediction result of the target object in the target driving behavior category in the future time period.

7. The method according to claim 1, characterized in that Before the driving behavior data is screened based on the driving behavior detection task, the method further includes: The driving behavior data is normalized by a preset linear normalization method to obtain normalized driving behavior data; By using an interpolation method based on a machine learning model, in a preset time series model, the normalized driving behavior data is interpolated in time series to obtain time series data corresponding to the normalized driving behavior data; The time series data is clustered and divided into different clusters by an outlier removal method based on a clustering algorithm. The outliers in the time series data are determined according to the distances from the time series data in different clusters to the cluster center, and the preprocessed driving behavior data are obtained by removing the outliers.

8. A multi-category unsafe driving behavior detection device, characterized in that: The device comprises: A task generation module, used to determine a target driving behavior category for detecting a target object among different driving behavior categories, and generate a driving behavior detection task corresponding to the target object based on the target driving behavior category; an acquisition module, configured to acquire driving behavior data of the target object, wherein the driving behavior data is multimodal data including driving behavior physiological data and driving behavior operation data, wherein the driving behavior physiological data represents data describing physiological characteristics of the target object during driving, and the driving behavior operation data represents data describing the target object's operation behavior on a vehicle during driving; a data processing module, configured to screen the driving behavior data based on the driving behavior detection task to obtain driving behavior data related to the driving behavior detection task, and integrate the driving behavior data related to the driving behavior detection task through a preset inter-modality data fusion algorithm to obtain modality set data; The detection module is used to obtain the time series neural network model corresponding to the driving behavior detection task, input the modal set data into the time series neural network model to analyze the time characteristics of the modal set data, and obtain the driving behavior detection result of the target object in the target driving behavior category.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images