Driving behavior classification method and device based on multi-modal fusion

Abstract

The invention provides a driving behavior classification method and device based on multi-modal fusion. The method comprises the following steps: acquiring time sequence data of a vehicle, wherein the time sequence data comprises motion state data collected according to a time domain in the vehicle running process; determining a time domain branch vector corresponding to the time sequence data, the time domain branch vector being obtained by performing residual connection based on a time sequence feature vector and a feature vector corresponding to the time sequence data; determining a frequency domain branch vector corresponding to the time sequence data, the frequency domain branch vector being obtained by performing residual connection based on a frequency domain feature vector and a feature vector corresponding to the time sequence data; and determining a driving behavior classification result of the vehicle according to a weighted fusion vector of the time domain branch vector and the frequency domain branch vector. Through the scheme, the accuracy of the driving behavior classification result is improved, and the calculation consumption is reduced.

Description

Technical Field

[0001] The present application relates to the field of driving behavior classification, and in particular to a driving behavior classification method, device, electronic device and computer-readable storage medium based on multimodal fusion. Background Art

[0002] With the rapid development of information technology, intelligent driving assistance systems (IDAS) have gradually become one of the core research areas in the technological evolution of the modern automotive industry. Advanced driver assistance systems (ADAS) are complex intelligent systems that integrate active and passive safety functions. Currently, driving behavior classification, such as through ADAS, allows for dynamic adjustment of vehicle control strategies based on driving status and behavior classification, contributing to safety, energy conservation, and carbon emission reduction. Therefore, an accurate driving behavior classification system is crucial for improving road safety and sustainable transportation.

[0003] However, the analysis process of driving status and driving behavior requires the collection of dynamic data of the vehicle or driver, such as driver physiological data, driver images, roadside images, etc., which has high data collection costs, analysis process complexity, computing resource consumption, and processing time. Summary of the Invention

[0004] In view of this, the present application provides a driving behavior classification method, device, equipment and computer-readable storage medium based on multimodal fusion, which solves the driving behavior classification problem through a time-frequency domain multimodal fusion method, improves the accuracy of driving behavior classification, and reduces computational overhead.

[0005] The present application is introduced below from multiple aspects, and the implementation methods and beneficial effects of the following multiple aspects can be referenced to each other.

[0006] In the first aspect, the present application provides a driving behavior classification method based on multimodal fusion, including: obtaining time series data of a vehicle, the time series data including motion state data collected in the time domain during the operation of the vehicle; determining the time domain branch vector corresponding to the time series data, wherein the time domain branch vector is obtained by residual connection based on a time series feature vector and a feature vector corresponding to the time series data, and the time series feature vector is obtained after inputting the time series data into a large kernel convolution layer; determining the frequency domain branch vector corresponding to the time series data, wherein the frequency domain branch vector is obtained by residual connection based on a frequency domain feature vector and a feature vector corresponding to the time series data, and the frequency domain feature vector is obtained by extracting the frequency domain data through an incremental convolution kernel, and the frequency domain data is obtained by performing time-frequency conversion on the time series data; and determining the driving behavior classification result of the vehicle based on a weighted fusion vector of the time domain branch vector and the frequency domain branch vector.

[0007] According to the implementation methods of this application, these frequency features are combined with feature fusion technology to construct a more complex and sophisticated model, significantly improving its performance and generalization capabilities. This model integrates the advantages of both time and frequency domain features to ultimately output a predicted category. Furthermore, through parallel processing in both the time and frequency domains, the model integrates dynamic temporal changes with global spectral characteristics, overcoming the limitations of single modal information and significantly improving the ability to characterize driving behavior.

[0008] In a possible implementation of the first aspect above, the time series data of the vehicle includes one or more of the number of lanes collected in chronological order, the position, acceleration, roll angle, pitch angle, yaw angle and vehicle speed of the vehicle.

[0009] In a possible implementation of the first aspect above, the large-kernel convolution layer includes convolution layers with convolution kernel sizes of 51, 49, 47 and 15, respectively, and the residual connection of the time domain branch vector is completed through a one-dimensional convolution layer with a convolution kernel size of 1.

[0010] In a possible implementation of the first aspect above, the convolution kernel sizes of the incremental convolution kernel are 1, 3, 5, 9 and 17 respectively, and the residual connection of the frequency domain branch vector is completed through a one-dimensional convolution layer with a convolution kernel size of 1.

[0011] In a possible implementation of the first aspect above, the driving behavior classification result of the vehicle is determined: the time domain branch vector and the frequency domain branch vector are respectively subjected to a one-dimensional convolution operation and a Sigmoid operation to obtain corresponding time series features and frequency domain features; the time series features and the frequency domain features are subjected to residual connection and full connection operations to obtain a fully connected output result, and the fully connected output result is used to determine the driving behavior classification result of the vehicle.

[0012] In a possible implementation of the first aspect above, the frequency domain data is obtained by performing a wavelet transform on the time series data, where the wavelet transform includes a Morlet transform.

[0013] In a possible implementation of the first aspect, the time-domain branch vector performs a residual connection on the time series feature vector and the feature vector corresponding to the time series data based on the following formula:

[0014] X out =ReLU(conv1(x t ))+X4

[0015] Among them, ReLU represents the ReLU activation function, conv1(*) is a one-dimensional convolution with a convolution kernel size of 1, "+" is an element-wise addition operation, and x t is the time series data, X4 is the time series feature vector output by the large kernel convolution layer, X out The time domain branch vector is output; the frequency domain branch vector performs residual connection on the frequency domain feature vector and the feature vector corresponding to the time series data based on the following formula:

[0016] F out =conv1(x t )+F5

[0017] Among them, F5 is the frequency domain feature vector, F out is the frequency domain branch vector output.

[0018] In the second aspect, the present application provides an attention training device based on dynamic sensory stimulation, including: an acquisition unit, used to: acquire time series data of a vehicle, the time series data including motion state data collected in the time domain during the operation of the vehicle; a processing unit, used to: determine the time domain branch vector corresponding to the time series data, wherein the time domain branch vector is obtained by residual connection based on a time series feature vector and a feature vector corresponding to the time series data, and the time series feature vector is obtained after the time series data is input into a large kernel convolution layer; determine the frequency domain branch vector corresponding to the time series data, wherein the frequency domain branch vector is obtained by residual connection based on a frequency domain feature vector and a feature vector corresponding to the time series data, and the frequency domain feature vector is obtained by extracting the frequency domain data through an incremental convolution kernel, and the frequency domain data is obtained by performing time-frequency conversion on the time series data; determine the driving behavior classification result of the vehicle according to the weighted fusion vector of the time domain branch vector and the frequency domain branch vector.

[0019] In a third aspect, the present application provides a device for driving behavior classification, the device comprising: a memory for storing instructions executed by one or more processors of the device, and a processor, which is one of the processors of the device, for executing the driving behavior classification method based on multimodal fusion disclosed in any aspect of the first aspect above.

[0020] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores instructions. When the instructions are executed on a computer, the computer executes the steps of the driving behavior classification method based on multimodal fusion described in the first aspect.

[0021] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer, the computer is caused to perform the steps of multimodal fusion-based driving behavior classification described in the first aspect. Alternatively, a computer program is provided. When the computer program is executed on a computer, the computer is caused to perform the steps of multimodal fusion-based driving behavior classification described in the first aspect.

[0022] The possible implementation methods and technical effects obtained in the above-mentioned second to fifth aspects are similar to the corresponding technical means and technical effects obtained in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the process of a driving behavior classification method based on multimodal fusion provided in an embodiment of the present application;

[0024] Figure 2 This is a process diagram of another driving behavior classification method based on multimodal fusion provided in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a driving behavior classification method provided in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of multimodal fusion provided in an embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of a device provided in an embodiment of the present application;

[0028] Figure 6 It is a structural diagram of a system on chip (SoC) provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] First, the prior art and existing technical problems involved in this application are introduced.

[0031] With the rapid development of information technology, intelligent driving assistance systems (IDAS) have gradually become one of the core research areas in the technological evolution of the modern automotive industry. Advanced driver assistance systems (ADAS) are complex intelligent systems that integrate active and passive safety functions. Through a multimodal sensor network (including visual sensors for traffic sign recognition and signal light detection, and radar sensors for real-time ranging and speed measurement), the system dynamically perceives the vehicle's surroundings and performs data fusion processing. Furthermore, based on this environmental perception data, the system can establish a driving situation recognition model, providing the driver with decision-making support information with temporal and spatial characteristics. Risk prediction algorithms can also be used to provide pre-collision warnings, significantly improving driving safety and ensuring proper driving operation.

[0032] The popularity and intelligence of these systems are increasing, but it is reported that the number of deaths caused by traffic accidents is still increasing each year, and most traffic accidents are caused by improper driving behavior of drivers. Based on the above problems, the research on driving behavior classification methods in ADAS systems has become an important topic. By identifying and classifying driving behavior states in real time, ADAS can dynamically optimize vehicle control parameters to achieve three goals: (1) improve driving safety performance; (2) reduce energy consumption; and (3) reduce exhaust emissions. Therefore, building a high-precision, low-cost, and more practical driving behavior classification system has important theoretical and practical value for achieving road traffic safety and sustainable development of the transportation system.

[0033] Existing methods for classifying driving behavior based on vehicle dynamics face several key technical challenges, including insufficient data utilization, defects in multimodal fusion, and limitations of time series processing methods.

[0034] For example, current research methods for extracting and utilizing features from vehicle dynamics data primarily focus on the time series level, limiting their scope. Another example is multimodal fusion, where current approaches generally rely on supplementary data such as driver physiological data, roadside surveillance images, and driver visual features. This significantly increases system complexity, including increased data acquisition costs, greater difficulty in model training, increased computing resource requirements, and increased processing latency.

[0035] For another example, in terms of fusion strategies, existing research primarily employs simple linear fusion methods such as feature addition and / or concatenation, which struggle to effectively model the nonlinear relationships between multimodal data and thus limit the representational capabilities of features. In another example, in time series processing, the current mainstream approach uses a fixed-size sliding window to segment continuous time series data. This not only fails to adapt to the timescale differences between different driving behaviors, but also suffers from insufficient accuracy in identifying behavior boundaries. Determining the optimal window size is extremely time-consuming and computationally intensive.

[0036] In order to solve the above technical problems, the present application proposes a driving behavior classification method 100 based on multimodal fusion. Figure 1 is a flow chart of the method 100 for classifying driving behaviors based on multimodal fusion, as shown in FIG. Figure 1 As shown, the method includes steps 110 to 140. The method 100 processes time series data through two different approaches, namely, time domain branching and frequency domain branching, to obtain a driving behavior classification result, thereby improving the accuracy of driving behavior classification and reducing computational overhead.

[0037] Step 110: Obtain time series data of vehicles.

[0038] Specifically, the time series data may include motion state data collected in the time domain during the operation of the vehicle.

[0039] For example, the motion state data may include one or more of the vehicle's acceleration collected in time sequence (such as three-axis acceleration, that is, the acceleration values ​​along the x-axis (front and rear direction), y-axis (left and right lateral direction), and z-axis (vertical direction) of the vehicle coordinate system, which can be used to describe the vehicle's longitudinal acceleration / deceleration, lateral deviation, and vertical vibration.), roll angle (the vehicle's rotation angle around the x-axis, which can be used to reflect the vehicle's left and right tilt state (such as the body roll when turning)), pitch angle (the vehicle's rotation angle around the y-axis, which can be used to reflect the vehicle's front and rear pitch state (such as the nodding action during acceleration or braking)), yaw angle (the vehicle's rotation angle around the z-axis, which can be used to reflect changes in the vehicle's driving direction (such as changes in the heading angle during turning)) and vehicle speed (the vehicle's current driving speed, which can be calculated from the wheel speed sensor or GPS data).

[0040] For another example, motion state data may also include vehicle control parameters, such as the steering wheel angle (reflecting the driver's steering operation intention), brake pressure (the pressure applied by the brake pedal, used to judge the braking intensity), throttle opening (throttle pedal position, used to reflect acceleration requirements), gear status (the vehicle's current gear (such as forward gear, reverse gear, etc.)), etc.

[0041] Step 120: Determine the time-domain branch vector corresponding to the time series data.

[0042] Specifically, the time domain branch vector is obtained by performing residual connection on a time series feature vector and a feature vector corresponding to the time series data. The time series feature vector is obtained after the time series data is input into a large kernel convolution layer.

[0043] A large kernel convolution layer refers to a convolution operation layer in deep learning (especially in computer vision) that uses larger convolution kernels (such as 7×7, 9×9, or even larger). Unlike the small convolution kernels (such as 3×3) commonly used in traditional convolutional neural networks (CNNs), large kernel convolution layers significantly improve the model's expressive power by increasing the receptive field and capturing more global contextual information. Residual connections, which pass inputs directly to deeper layers of the network rather than relying solely on the results of layer-by-layer forward propagation, can address the vanishing gradient problem that occurs as the network deepens.

[0044] Step 120 extracts global temporal features (i.e., time-series feature vectors) from the time series data using a large-kernel convolutional layer. The extracted global temporal features are then fused with the original temporal features (i.e., feature vectors corresponding to the time series data) using a residual connection. This allows the temporal processing process in this application to adapt to the time scale differences of different driving behaviors and is more efficient. The detailed process of step 120 will be described in detail in the following examples, and will not be elaborated on here.

[0045] Step 130: Determine the frequency domain branch feature vector of the time series data.

[0046] Specifically, the frequency domain branch vector is obtained by residual connection based on the frequency domain feature vector and the feature vector corresponding to the time series data. The frequency domain feature vector is obtained by extracting the frequency domain data through an incremental convolution kernel, and the frequency domain data is obtained by time-frequency conversion of the time series data.

[0047] Increasing convolution kernels refer to the parallel use of multiple convolution kernels of increasing sizes (e.g., 1×1, 3×3, 5×5) in a single layer to fuse multi-scale features, expand the receptive field, and extract global semantics. Frequency domain data can be obtained by performing Fourier transform (FFT), discrete cosine transform (DCT), wavelet transform (WT), and other methods on time series data.

[0048] Step 130 converts the time series data into frequency domain data, extracts features from the frequency domain data using an incremental convolution kernel, and finally fuses the frequency domain feature vector with the original time series features (i.e., the feature vector corresponding to the time series data). This effectively reveals the periodic patterns and frequency characteristics in the frequency domain data. The following examples will describe the detailed process of step 130, which will not be elaborated here.

[0049] Step 140: Determine the driving behavior classification result of the vehicle according to the weighted fusion vector of the time domain branch feature vector and the frequency domain branch feature vector.

[0050] Specifically, in the embodiment of the present application, after extracting features from both the time domain and the frequency domain, the two obtained feature vectors are weighted and fused. Finally, the driving behavior classification result of the vehicle is determined based on the weighted and fused results.

[0051] For example, in an embodiment of the present application, the weighted fusion process may be to perform a one-dimensional convolution operation and a Sigmoid operation on the time domain branch vector and the frequency domain branch vector, respectively, to obtain corresponding time series features and frequency domain features. Furthermore, the time series features and the frequency domain features are subjected to residual connection and full connection operations to obtain a fully connected output result, which is used to determine the vehicle's driving behavior classification result.

[0052] Method 100 utilizes parallel processing in both the time and frequency domains to integrate dynamic time series changes with global spectral characteristics, overcoming the limitations of single-modal information and significantly improving the ability to characterize driving behavior. Furthermore, in method 100, frequency domain analysis plays a crucial role in time series analysis, effectively revealing periodic patterns and frequency components in the data. Combining these frequency characteristics with feature fusion techniques enables the construction of more complex and sophisticated models, significantly improving both performance and generalization.

[0053] The following combination Figures 2 to 4 An example of an embodiment corresponding to method 100 is introduced. Figure 2is a flow chart of an example method for driving behavior classification based on multimodal fusion provided in an embodiment of the present application, that is, a schematic diagram of an example method 200 of the method 100; Figure 3 is a schematic diagram of a method 200 provided in an embodiment of the present application; Figure 4 It is a schematic diagram of multimodal fusion provided in an embodiment of the present application.

[0054] First combine Figure 2 ,like Figure 2 As shown, the method 200 includes steps 210 to 230. For ease of understanding, the present application can divide the framework of the method 200 into three modules: a global temporal feature extraction module, a frequency domain feature extraction module, and a temporal-frequency domain attention fusion module.

[0055] Step 210: The global temporal feature extraction module extracts more temporal global information through large kernel convolution.

[0056] Corresponding to step 110, the vehicle dynamics data at time t can be recorded as x t =(A t1 ,A t2 ,A t3 ,…). The time series data of the vehicle in step 110 may include multiple data columns (for example, 36 data columns may be selected in this application), and the data in the data columns may include acceleration, roll angle, pitch angle, yaw angle and vehicle speed measured along the x-axis, y-axis and z-axis. t =(A xt ,A yt ,A zt ,…) as input. For example, driving time series data may include timestamps, speeds, latitude coordinates, altitude coordinates, etc., which are arranged in order. In this case, these data can be displayed in each column of the driving time series data:

[0057]

[0058] Among them, the columns where 8.83 and 9.82 are timestamp data, the columns where 64.5 and 63.6 are vehicle speed data, and the same applies to the data in other columns.

[0059] Then, corresponding to step 120, as Figure 3 As shown, data x t It is fed into the global temporal feature extraction module as input. The feature extraction process of the large kernel convolution layer is shown in the following formula (1):

[0060]

[0061] Among them, ReLU represents the ReLU activation function, conv 51(*) represents a one-dimensional convolution with a kernel size of 51, X i Represents the obtained feature vector of layer i. ki Represents the size of the convolution kernel in different convolution layers. In the embodiment of the present application, Figure 3 As shown in the figure, the convolution kernel sizes of the large kernel convolution layers can be set as follows: the convolution kernel size of the first layer is 51, and the convolution kernel sizes of the second, third and fourth layers are 49, 47 and 15 respectively.

[0062] After feature extraction, Figure 3 As shown, the extracted feature vector X4 and time series data x t The corresponding original feature vectors are residually connected. The residual connection formula is shown in the following formula (2):

[0063] X out =ReLU(conv1(x t ))+X4, (2)

[0064] Here conv1(*) represents a one-dimensional convolution with a kernel size of 1, + represents an element-wise addition operation, and X out Represents the time domain branch vector of the output.

[0065] Step 220: The frequency domain feature extraction module converts the time series features into the frequency domain, and then uses incremental convolution to extract features from the frequency domain data.

[0066] Corresponding to step 130, as Figure 3 As shown in Figure 2, the frequency domain feature extraction module first converts the time series data into frequency domain data using the wavelet transform method. The wavelet transform formula is shown in the following formula (3):

[0067] F=Wavelet(x t ), (3)

[0068] Where F represents the frequency domain feature after conversion, and Wavelet represents the wavelet transform method used. Here, the wavelet transform can use Morlet transform, which can be expressed as shown in formula (4):

[0069]

[0070] Here, Φ(t) represents the wavelet function, and a, b, and t are the scale factor, translation factor, and time translation factor, respectively. After converting the time series data to frequency domain data, the frequency domain data is sent to the frequency domain feature extraction module for feature extraction. This process can be expressed as shown in Equation (5):

[0071]

[0072] Among them, F iis the frequency domain feature vector extracted by each convolution kernel, conv ki (*) represents two-dimensional convolution with different convolution kernel sizes. In the incremental convolution kernel, such as Figure 3 As shown in Figure 2, when i=1 to 5, the convolution kernel sizes are 1, 3, 5, 9, and 17 respectively. After extracting the frequency domain feature vector F5, the time series data x t The corresponding original eigenvector and the extracted frequency domain eigenvector F5 are residually connected to obtain the frequency domain branch vector F out , the process is shown in formula (6):

[0073] F out =Conv1(x t )+F5, (6)

[0074] Step 230: The time-frequency attention fusion module fuses the features extracted from the time and frequency domains, and outputs the predicted category of the driving behavior through a fully connected layer.

[0075] In the embodiment of the present application, the time series features and frequency domain features are fused in this module. Figure 4 As shown, first, the time domain branch vector obtained in step 210 and the frequency domain branch vector obtained in step 220 are respectively subjected to a one-dimensional convolution operation to adjust the dimension, and then a Sigmoid operation is performed, as shown in equations (7) and (8):

[0076] F c =Sigmoid(conv1(F out ))×F out , (7)

[0077] X c =Sigmoid(conv1(X out ))×X out , (8)

[0078] Among them, F c and X c Represent the enhanced time series and frequency domain features respectively.

[0079] After that, the residual connection is used to perform element-wise multiplication with the original features to highlight the most important features. Finally, the features are connected and input into the fully connected layer to obtain the driving behavior category. The process is shown in the following equations (9) and (10):

[0080] Fusion=Concat(F c ,X c ), (9)

[0081] Final=FC(Fusion), (10)

[0082] Concat represents the concatenation operation, FC represents the fully connected operation, and Final represents the output driving behavior category. The resulting Final value can be used to determine the driving behavior classification result. For example, the value of Final can be divided into three values ​​(such as 0, 1, and 2), which can represent normal driving, aggressive driving, and drowsy driving, respectively.

[0083] Frequency domain analysis is crucial in time series analysis, effectively revealing periodic patterns and frequency components in data. In Methods 100 and 200, these frequency features are combined with feature fusion techniques to construct more complex and sophisticated models, significantly improving their performance and generalization capabilities. This model integrates the strengths of both time and frequency domain features to ultimately output a predicted category. Furthermore, Methods 100 and 200 employ parallel processing in both the time and frequency domains, integrating dynamic time series changes with global spectral characteristics. This overcomes the limitations of single-modal information and significantly enhances the ability to characterize driving behavior.

[0084] Tables 1 to 3 show the data analysis results in the examples of this application.

[0085] Table 1 shows a comparison of the performance on a motorway. As shown in Table 1, the accuracy (Acc), precision (Pre), recall (Rec), and F1 score (F1) of the embodiment of the present application are all higher than those of other existing models. Furthermore, the frames per second (FPS) of the embodiment of the present application is higher than those of other existing models, indicating that the model has a faster response speed.

[0086] Table 1 Comparison of results on the UAH-Drive dataset (motorway roads)

[0087]

[0088]

[0089] Table 2 shows a comparison of the effects on the Secondary path. As shown in Table 2, the accuracy (Acc), precision / precision (Pre), recall / recall (Rec), and F1 score (F1 Score, F1) of the embodiment of the present application are also higher than most existing models, and the frames per second (FPS) of the embodiment of the present application is higher than other existing models.

[0090] Table 2. Comparison of results on the UAH-Drive dataset (Secondary roads)

[0091]

[0092]

[0093] Table 3 shows the comparison of the effects on all roads. As shown in Table 3, the accuracy (Acc), precision / precision (Pre), recall / recall (Rec) and F1 score (F1 Score, F1) of the embodiment of the present application are also higher than most existing models, and the frames per second (FPS) of the embodiment of the present application is higher than other existing models.

[0094] Table 3. Comparison of results on the UAH-Drive dataset (all roads)

[0095]

[0096]

[0097] In addition, Tables 1 to 3 also show that the parameter amount of the model of the embodiment of the present application is 19.53M, and the floating point number of the model is 8.96G. It can be concluded that the model selected in this application is a lightweight model, corresponding to low computing overhead, low memory usage, high real-time performance and low training cost.

[0098] Therefore, it can be seen from Tables 1 to 3 that the method proposed in this application has high accuracy and is very lightweight, and its effect surpasses most current methods.

[0099] Now refer to Figure 5, shown is a block diagram of a device 500 according to one embodiment of the present application. The device 500 may include one or more processors 501 coupled to a controller hub 503. For at least one embodiment, the controller hub 503 communicates with the processor 501 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a quickpath interconnect (QPI), or a similar connection 510. The processor 501 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 503 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on separate chips) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0100] The device 500 may also include a coprocessor 502 and a memory 504 coupled to the controller hub 503. Alternatively, one or both of the memory and the GMCH may be integrated within the processor, with the memory 504 and the coprocessor 502 directly coupled to the processor 501 and the controller hub 503, with the controller hub 503 and the IOH being in a single chip. The memory 504 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. In one embodiment, the coprocessor 1002 is a special-purpose processor, such as, for example, a high-throughput MIC processor (manyintegrated core, MIC), a network or communication processor, a compression engine, a graphics processor, a general purpose computing on GPU (GPGPU), or an embedded processor, etc. The optional nature of the coprocessor 502 is indicated by a dotted line in Figure 5 middle.

[0101] The memory 504, as a computer-readable storage medium, may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, the memory 504 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as one or more hard disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0102] In one embodiment, device 500 may further include a network interface controller (NIC) 506. NIC 506 may include a transceiver for providing a radio interface for device 500, thereby communicating with any other suitable devices (e.g., a front-end module, an antenna, etc.). In various embodiments, NIC 506 may be integrated with other components of device 500. NIC 506 may implement the functionality of the communication unit in the above-described embodiments.

[0103] Device 500 may further include input / output (I / O) devices 505. I / O 505 may include: a user interface designed to enable a user to interact with device 500; a peripheral component interface designed to enable peripheral components to interact with device 500; and / or sensors designed to determine environmental conditions and / or location information related to device 500.

[0104] It is worth noting that Figure 5 This is for illustrative purposes only. Figure 5 It is shown that the device 500 includes multiple components such as a processor 501, a controller hub 503, a memory 504, etc. However, in actual applications, the device using the various methods of the present application may only include a part of the components of the device 500, for example, it may only include the processor 501 and NIC506. Figure 5 The properties of the optional components are shown with dashed lines. According to some embodiments of the present application, the memory 504 as a computer-readable storage medium stores instructions that, when executed on a computer, cause the device 500 to perform the attention training method according to the above-described embodiment. For details, please refer to the method of the above-described embodiment and will not be repeated here.

[0105] Now refer to Figure 6 , which is a block diagram of a system on chip (SoC) 600 according to an embodiment of the present application. Figure 6In FIG, similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Figure 6 In the embodiment, SoC 600 includes: an interconnect unit 650 coupled to an application processor 610; a system agent unit 680; a bus controller unit 690; an integrated memory controller unit 640; a set of one or more coprocessors 620, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 630; and a direct memory access (DMA) unit 660. In one embodiment, the coprocessors 620 include specialized processors, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0106] The static random access memory (SRAM) unit 630 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions. The instructions may include: when executed by at least one unit in the processor, causing the Soc 600 to perform the attention training method according to the above embodiment. For details, please refer to the method of the above embodiment, which will not be repeated here.

[0107] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0108] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0109] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0110] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to floppy disks, optical disks, optical discs, compact disc read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random-access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0111] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such a feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0112] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0113] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0114] Although the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.

Claims

1. A driving behavior classification method based on multimodal fusion, characterized in that: include: Acquiring time series data of a vehicle, wherein the time series data includes motion state data of the vehicle collected in a time domain during operation; Determine a time-domain branch vector corresponding to the time series data, wherein the time-domain branch vector is obtained by performing a residual connection between a time-series feature vector and a feature vector corresponding to the time series data, and the time-series feature vector is obtained by inputting the time series data into a large-kernel convolution layer; Determining a frequency domain branch vector corresponding to the time series data, wherein the frequency domain branch vector is obtained by performing a residual connection based on a frequency domain feature vector and a feature vector corresponding to the time series data, the frequency domain feature vector is obtained by extracting the frequency domain data using an incremental convolution kernel, and the frequency domain data is obtained by performing a time-frequency conversion on the time series data; A driving behavior classification result of the vehicle is determined according to a weighted fusion vector of the time domain branch vector and the frequency domain branch vector.

2. The method according to claim 1, characterized in that The time series data of the vehicle includes one or more of the number of lanes, the position, acceleration, roll angle, pitch angle, yaw angle and vehicle speed of the vehicle collected in time sequence.

3. The method according to claim 1 or 2, characterized in that The large-kernel convolution layer includes convolution layers with convolution kernel sizes of 51, 49, 47, and 15, respectively. The residual connection of the time-domain branch vector is completed through a one-dimensional convolution layer with a convolution kernel size of 1.

4. The method according to claim 1 or 2, characterized in that The convolution kernel sizes of the incremental convolution kernel are 1, 3, 5, 9 and 17 respectively, and the residual connection of the frequency domain branch vector is completed through a one-dimensional convolution layer with a convolution kernel size of 1.

5. The method according to claim 1 or 2, characterized in that Determining the driving behavior classification result of the vehicle: Perform a one-dimensional convolution operation and a Sigmoid operation on the time domain branch vector and the frequency domain branch vector respectively to obtain corresponding time series features and frequency domain features; The time series features and the frequency domain features are subjected to residual connection and full connection operations to obtain a full connection output result, and the full connection output result is used to determine the driving behavior classification result of the vehicle.

6. The method according to claim 1 or 2, characterized in that The frequency domain data is obtained by performing wavelet transform on the time series data, and the wavelet transform includes Morlet transform.

7. The method according to claim 1 or 2, characterized in that The time domain branch vector performs residual connection on the time series feature vector and the feature vector corresponding to the time series data based on the following formula: X out =RELU(conv1(x t ))+X4 Among them, ReLU represents the ReLU activation function, conv1(*) is a one-dimensional convolution with a convolution kernel size of 1, "+" is an element-wise addition operation, and x t is the time series data, X4 is the time series feature vector output by the large kernel convolution layer, X out is the time domain branch vector output; The frequency domain branch vector performs residual connection on the frequency domain feature vector and the feature vector corresponding to the time series data based on the following formula: F out =conv1(x t )+F5 Among them, F5 is the frequency domain feature vector output by the incremental convolution kernel, F out is the frequency domain branch vector output.

8. A driving behavior classification device based on multimodal fusion, characterized in that: include: An acquisition unit is configured to: acquire time series data of a vehicle, wherein the time series data includes motion state data of the vehicle collected in a time domain during operation; a processing unit, configured to determine a time-domain branch vector corresponding to the time series data, wherein the time-domain branch vector is obtained by performing a residual connection between a time-series feature vector and a feature vector corresponding to the time series data, and the time-series feature vector is obtained by inputting the time series data into a large-kernel convolution layer; Determining a frequency domain branch vector corresponding to the time series data, wherein the frequency domain branch vector is obtained by performing a residual connection based on a frequency domain feature vector and a feature vector corresponding to the time series data, the frequency domain feature vector is obtained by extracting the frequency domain data using an incremental convolution kernel, and the frequency domain data is obtained by performing a time-frequency conversion on the time series data; A driving behavior classification result of the vehicle is determined according to a weighted fusion vector of the time domain branch vector and the frequency domain branch vector.

9. A device for classifying driving behavior, characterized in that: include: a memory for storing instructions executable by the processor; A processor, wherein the processor is configured to implement the method according to any one of claims 1 to 7 when executing the instructions.

10. A computer-readable storage medium storing instructions, characterized in that: When the instruction is executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 7.

Images