Multi-modal data fusion method and device for driver behavior prediction
By acquiring image, audio and vehicle status data, preprocessing and feature extraction, and using attention network to perform multimodal feature fusion, the problem of information interference and high computational complexity of multimodal data fusion in the prior art is solved. Achieve more accurate and flexible driver behavior predictions.
Patent Information
- Application Number
- CN202410967443.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Among the existing driver behavior prediction methods, single modal data input cannot meet the technical needs, and multimodal data fusion has problems of information interference and high computational complexity, and the existing methods are highly subjective and poorly scientific.
By acquiring image data, audio data and vehicle status data, preprocessing and feature extraction, multimodal feature fusion is used to adaptively adjust weights, avoid manual intervention, and realize information integration of different data sources.
It improves the accuracy and adaptability of driver behavior prediction, reduces the amount of calculation, enhances the scientificity and flexibility of the method, and can more comprehensively capture driver behavior characteristics.
Smart Images

Figure CN118916834A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of data processing technology, and in particular to a multimodal data fusion method for driver behavior prediction. Background Art
[0002] With the continuous development of deep learning, methods for predicting driver behavior are emerging in an endless stream. Single-modal data input can no longer meet technical requirements. The processing of multimodal data has become one of the hot topics of current research. The fusion methods of multimodal data are mainly divided into data-level fusion, feature-level fusion, decision-level fusion, etc. Specific methods include: vector concatenation, weighted fusion, attention mechanism fusion, subspace learning, etc.
[0003] The data-level and feature-level fusion methods have the problem of interference between different modal information, and the amount of data after fusion is large, resulting in a high model complexity; although the decision-level fusion can reduce the amount of calculation and improve the running speed, the correlation between the modes is not considered, which may lead to inaccurate results. Among the commonly used specific methods: vector concatenation ignores the unique statistical properties of the mode through simple feature connection; weighted fusion considers the weights of different features compared to vector concatenation, but because it is based on the definition of rules and weight assignment based on human prior consciousness, it is more subjective and less scientific. Although the attention mechanism and subspace learning method have achieved good results in fusion, they are also feature-level fusion methods. In the task of predicting driver driving behavior, there are many types of data and a large amount of data. It is difficult to adapt to actual application scenarios by directly using such methods. Therefore, a better solution is urgently needed. Summary of the invention
[0004] In view of this, the embodiments of this specification provide a multimodal data fusion method for driver behavior prediction. One or more embodiments of this specification also relate to a multimodal data fusion device for driver behavior prediction, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.
[0005] According to a first aspect of an embodiment of this specification, a multimodal data fusion method for driver behavior prediction is provided, comprising:
[0006] Acquire initial data; wherein the initial data includes image data, audio data and vehicle status data;
[0007] Determine a data preprocessing rule based on the category of the initial data, and process the initial data based on the data preprocessing rule to determine the multimodal features;
[0008] Data fusion is performed based on the categories of multimodal features to determine fusion features.
[0009] In a possible implementation, obtaining initial data includes:
[0010] Acquire image data through the in-vehicle camera;
[0011] Acquire audio data through the microphone in the car;
[0012] Acquire vehicle status data through vehicle sensors;
[0013] Initial data is determined based on the image data, the audio data, and the vehicle status data.
[0014] In a possible implementation, the initial data is processed based on a data preprocessing rule to determine the multimodal features, including:
[0015] Crop the image data, and extract features from the cropped data based on the time window to determine initial image features;
[0016] Perform dimensionality reduction processing on the initial image features to determine the target image features;
[0017] Converting audio data into text information, and determining audio text coefficients based on the text information;
[0018] Determine a Mel-frequency cepstral coefficient tensor based on the audio data, and perform MLP processing based on the Mel-frequency cepstral coefficient tensor to determine sound features;
[0019] Based on the vehicle state data, the initial vehicle state features are extracted through a convolutional neural network, and MLP processing is performed based on the initial vehicle state features to determine the target vehicle state features;
[0020] The video frame data is determined based on the image data, and features are extracted from the video frame data through a deep learning model to determine semantic features. MLP processing is performed based on the semantic features to determine video features.
[0021] In a possible implementation, data fusion is performed based on the category of multimodal features to determine fusion features, including:
[0022] The target image features, audio text coefficients and sound features are fused through the first attention network to obtain the first intermediate features.
[0023] In a possible implementation, data fusion is performed based on the category of multimodal features to determine fusion features, including:
[0024] The video features, audio text coefficients and sound features are fused through the first attention network to obtain the second intermediate features.
[0025] In a possible implementation, data fusion is performed based on the category of multimodal features to determine fusion features, including:
[0026] Based on the first intermediate feature, the second intermediate feature and the target vehicle state feature, they are fused through the second attention network to obtain a fused feature.
[0027] In a possible implementation, the method further includes:
[0028] Perform model training based on fusion features and determine the target deep learning model;
[0029] Driver behavior prediction based on targeted deep learning models.
[0030] According to a second aspect of an embodiment of this specification, a multimodal data fusion device for driver behavior prediction is provided, comprising:
[0031] A data acquisition module is configured to acquire initial data; wherein the initial data includes image data, audio data and vehicle status data;
[0032] A feature extraction module is configured to determine a data preprocessing rule based on the category of the initial data, and process the initial data based on the data preprocessing rule to determine a multimodal feature;
[0033] The feature fusion module is configured to perform data fusion based on the categories of multimodal features and determine fusion features.
[0034] According to a third aspect of an embodiment of this specification, a computing device is provided, including:
[0035] Memory and processor;
[0036] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the multimodal data fusion method for driver behavior prediction are implemented.
[0037] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the multimodal data fusion method for driver behavior prediction are implemented.
[0038] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multimodal data fusion method for driver behavior prediction.
[0039] The embodiments of this specification provide a multimodal data fusion method and device for driver behavior prediction, wherein the multimodal data fusion method for driver behavior prediction includes: obtaining initial data; wherein the initial data includes image data, audio data, and vehicle status data; determining data preprocessing rules based on the category of the initial data, and processing the initial data based on the data preprocessing rules to determine multimodal features; performing data fusion based on the category of the multimodal features to determine fusion features. By obtaining initial data; determining data preprocessing rules based on the category of the initial data, and processing the initial data based on the data preprocessing rules to determine multimodal features; performing data fusion based on the category of the multimodal features to determine fusion features, it is possible to integrate information from different data sources together to more comprehensively and accurately capture the characteristics of driver behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 is a flowchart of a multimodal data fusion method for driver behavior prediction provided by an embodiment of this specification;
[0041] Figure 2 is a schematic diagram of a multimodal data fusion method for driver behavior prediction provided by an embodiment of this specification;
[0042] Figure 3 It is a schematic diagram of multi-layer cross-mixed multi-modal data fusion of a multi-modal data fusion method for driver behavior prediction provided by an embodiment of this specification;
[0043] Figure 4 It is a schematic diagram of intermediate feature fusion of a multimodal data fusion method for driver behavior prediction provided by an embodiment of this specification;
[0044] Figure 5 It is a feature fusion schematic diagram of a multimodal data fusion method for driver behavior prediction provided by an embodiment of this specification;
[0045] Figure 6 It is a structural schematic diagram of a multimodal data fusion device for driver behavior prediction provided by an embodiment of this specification;
[0046] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0047] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0048] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0049] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0050] First, the terms involved in one or more embodiments of this specification are explained.
[0051] Multilayer Perceptron (MLP) is a basic artificial neural network model, whose structure is a multi-layer structure composed of multiple neurons.
[0052] PCA (Principal Component Analysis), also known as principal component analysis, is the most widely used data dimensionality reduction algorithm.
[0053] ORB: Oriented Fast and Rotated Brief, can be used to quickly create feature vectors for key points in an image, which can be used to identify objects in the image.
[0054] In this specification, a multimodal data fusion method for driver behavior prediction is provided. This specification also relates to a multimodal data fusion device for driver behavior prediction, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0055] See also Figure 1 , Figure 1 A flowchart of a multimodal data fusion method for driver behavior prediction provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0056] Step 101: Acquire initial data; wherein the initial data includes image data, audio data and vehicle status data.
[0057] In a possible implementation, obtaining initial data includes: obtaining image data through an in-vehicle camera; obtaining audio data through an in-vehicle microphone; obtaining vehicle status data through a vehicle sensor; and determining initial data based on the image data, audio data, and vehicle status data.
[0058] In practical applications, see Figure 2 ,First, collect multimodal data during driving, including images, videos, audio, and vehicle sensor data.
[0059] For example, in-vehicle cameras, microphones, and vehicle sensors are used to collect image data, video data, audio data, and vehicle status data, where the vehicle status data includes: vehicle speed, steering angle, brake pressure, accelerator pedal position, and lane departure information.
[0060] The embodiments of this specification use multimodal data to monitor the driver's driving condition from multiple aspects such as image, video, audio and vehicle status, which can increase the prediction accuracy.
[0061] Step 102: Determine a data preprocessing rule based on the category of the initial data, and process the initial data based on the data preprocessing rule to determine the multimodal features.
[0062] In practical applications, see Figure 2 ,Next, the data are preprocessed and feature extracted to automatically learn and ,extract high-level feature representation of driver behavior.
[0063] In a possible implementation, the initial data is processed based on data preprocessing rules to determine multimodal features, including: cropping the image data, and extracting features from the cropped data based on a time window to determine initial image features; performing dimensionality reduction processing on the initial image features to determine target image features; converting the audio data into text information, and determining the audio text coefficients based on the text information; determining a Mel-frequency cepstral coefficient tensor based on the audio data, and performing MLP processing based on the Mel-frequency cepstral coefficient tensor to determine sound features; extracting initial vehicle state features through a convolutional neural network based on vehicle state data, and performing MLP processing based on the initial vehicle state features to determine target vehicle state features; determining video frame data based on the image data, extracting features from the video frame data through a deep learning model to determine semantic features, and performing MLP processing based on the semantic features to determine video features.
[0064] In practical applications, after collecting data, each modal information is processed and features are extracted. For image data, the driver's facial expression features are mainly collected. The main steps are as follows:
[0065] 1. Image cropping: crop the sampled image with the driver’s face as the center;
[0066] 2. Feature extraction: The ORB (Oriented FAST and RotatedBRIEF) method is used to extract features from images within a time window;
[0067] 3. Dimensionality reduction: Use principal component analysis (PCA) technology to reduce the dimension of the features obtained in the previous step to obtain a one-dimensional feature vector f1.
[0068] Furthermore, the video data mainly collects the driver's actions. The main steps include:
[0069] 1. Preprocess the video data: including frame rate control, color space conversion, video compression, etc., to reduce the amount of data and improve processing efficiency.
[0070] 2. Video decomposition: Decompose the video into a series of image frames, each frame represents a time point in the video.
[0071] 3. Feature extraction: Using deep learning models (such as convolutional neural networks) to extract features from video frames can obtain high-level semantic feature representations. For example, you can use pre-trained deep learning models (such as VGG, ResNet, etc.) to extract feature vectors of video frames, and after MLP processing, you can get video features f2 with the same dimension as f1.
[0072] Furthermore, for audio data, the audio data is divided into two parts: text information and sound features. The former is to analyze the content of the speaker, and the latter is to analyze the speaker's tone and speaking speed to assist in judging the driver's emotional state. The main steps include:
[0073] 1. Text information: Use the deep learning speech recognition model CTC to convert audio signals into text information, provide text information about the speech content contained in the audio, use the word embedding technology based on shallow neural networks to convert the data from various sources into computable high-dimensional vectors, and then use a fully connected network to train the high-order vector features of the text, and output a text content danger index (0-9), which is set as the audio text coefficient k1.
[0074] 2. Voice features: Language can express emotions through changes in tone, speed and intonation. Different emotions in speech will change the sound quality and frequency features. Mel-frequency cepstral coefficients (MFCC) are commonly used voice features in emotion recognition. The Mel-frequency cepstral coefficient tensor of an audio segment is calculated and processed by MLP to obtain the voice feature f3 with the same dimension as f1.
[0075] Furthermore, for vehicle status data, vehicle speed, steering angle, brake pressure, accelerator pedal position, and lane departure information are obtained through vehicle sensors, and vehicle status features are extracted through a convolutional neural network, and vehicle status feature f4 is obtained through MLP processing.
[0076] Step 103: Perform data fusion based on the categories of multimodal features to determine fusion features.
[0077] In a possible implementation, data fusion is performed based on the category of multimodal features to determine fusion features, including: fusing target image features, audio text coefficients, and sound features through a first attention network to obtain a first intermediate feature.
[0078] In practical applications, see Figure 3 , the image feature f1 is fused with the audio text coefficient k1 and the sound feature f3 using the attention network to obtain the intermediate feature M1.
[0079] Specifically, Figure 4 As shown, first, the text coefficient k is repeatedly filled to obtain a vector K with the same dimension as f1 and f2, and then f1, K, and f3 are input into the MLP to obtain the initial weight coefficients W1, W2, and W3. The three weights are processed by the softmax function to obtain the weight coefficients α1, α2, and α3, so that the sum of the three is 1. Finally, the weight coefficients are weighted and fused with the features f1, K, and f3 to obtain the intermediate feature M1. This step fuses facial expression features with audio features and is mainly used to judge the driver's fatigue state.
[0080] In the multi-layer cross-mixed multimodal data fusion method, the embodiment of this specification uses a mixed fusion method of feature-level data and decision-level data (wherein the audio text coefficient is the decision-level data). Using decision-level data while obtaining more information can reduce the amount of calculation and improve the speed.
[0081] In a possible implementation, data fusion is performed based on the category of multimodal features to determine fusion features, including: fusing video features, audio text coefficients, and sound features through a first attention network to obtain a second intermediate feature.
[0082] In practical applications, see Figure 3, the video feature f2 is fused with the audio text coefficient k1 and the sound feature f3 using the same attention network as in the previous step to obtain the intermediate feature M2. The fusion of action features and audio features is mainly used to determine whether the driver has abnormal behavior caused by emotional excitement and other dangerous behaviors.
[0083] In the multi-layer cross-mixed data multimodal fusion method of the embodiment of this specification, the audio data information is reused according to the task characteristics, which can effectively solve the problem of mutual interference between different modal data in feature-level fusion.
[0084] In a possible implementation, data fusion is performed based on the category of multimodal features to determine the fusion feature, including: based on the first intermediate feature, the second intermediate feature and the target vehicle state feature, the fusion is performed through a second attention network to obtain the fusion feature.
[0085] In practical applications, the intermediate features M1 and M2 are fused with the vehicle state feature f4 using the attention mechanism 2 to obtain the fused feature: F1. Figure 5 As shown in the figure, firstly, the intermediate features M1, M2, and the vehicle state feature f3 are input into the hidden layer H: H = tanh (W1·M1+W2·M2+W3·f3+b), where W1, W2, and W3 are weight matrices, which are used to map the features of each modality to the hidden layer respectively; M1 and M2 are intermediate features with a dimension of D; f3 is a vehicle state feature with a dimension of D; and b is a bias vector. The H layer can integrate the features of each modality and the vehicle state features for subsequent calculation of the attention weight. Then, the output result of the H layer is used as input, and the initial attention weight is obtained using MLP, and then the weight coefficient is normalized using softmax to obtain the attention weight coefficients α4, α5, and α6. Finally, the three weight coefficients are weightedly fused with M1, M2, and M3 to obtain the fused feature F1. The vehicle state is crucial to the evaluation of driving behavior, so the intermediate features are directly fused with the vehicle state features here.
[0086] The method of the embodiment of this specification does not have artificially designed weights, avoiding the problem of defining rules for weight assignment based on human a priori consciousness, which is highly subjective and less scientific. The method can adaptively adjust the weights according to the data, so that it can adapt to a variety of different scenarios and has strong flexibility.
[0087] In a possible implementation, it also includes: performing model training based on fusion features to determine a target deep learning model; and predicting driver behavior based on the target deep learning model.
[0088] In practical applications, for the fused data F1, deep learning methods such as deep neural network (DNN), deep reinforcement learning (Deep Reinforcement Learning) and other methods can be used to train the model to complete different tasks according to actual needs, such as the evaluation of dangerous driving behavior of drivers.
[0089] The embodiments of this specification provide a multimodal data fusion method and device for driver behavior prediction, wherein the multimodal data fusion method for driver behavior prediction includes: obtaining initial data; wherein the initial data includes image data, audio data, and vehicle status data; determining data preprocessing rules based on the category of the initial data, and processing the initial data based on the data preprocessing rules to determine multimodal features; performing data fusion based on the category of the multimodal features to determine fusion features. By obtaining initial data; determining data preprocessing rules based on the category of the initial data, and processing the initial data based on the data preprocessing rules to determine multimodal features; performing data fusion based on the category of the multimodal features to determine fusion features, it is possible to integrate information from different data sources together to more comprehensively and accurately capture the characteristics of driver behavior.
[0090] Corresponding to the above method embodiment, this specification also provides a multimodal data fusion device embodiment for driver behavior prediction. Figure 6 FIG. 1 is a schematic diagram showing a structure of a multimodal data fusion device for driver behavior prediction provided by an embodiment of the present specification. Figure 6 As shown, the device comprises:
[0091] The data acquisition module 601 is configured to acquire initial data; wherein the initial data includes image data, audio data and vehicle status data;
[0092] The feature extraction module 602 is configured to determine a data preprocessing rule based on the category of the initial data, and process the initial data based on the data preprocessing rule to determine a multimodal feature;
[0093] The feature fusion module 603 is configured to perform data fusion based on the categories of multimodal features and determine fusion features.
[0094] In a possible implementation, the data acquisition module 601 is further configured to:
[0095] Acquire image data through the in-vehicle camera;
[0096] Acquire audio data through the microphone in the car;
[0097] Acquire vehicle status data through vehicle sensors;
[0098] Initial data is determined based on the image data, the audio data, and the vehicle status data.
[0099] In a possible implementation, the feature extraction module 602 is further configured to:
[0100] Crop the image data, and extract features from the cropped data based on the time window to determine initial image features;
[0101] Perform dimensionality reduction processing on the initial image features to determine the target image features;
[0102] Converting audio data into text information, and determining audio text coefficients based on the text information;
[0103] Determine a Mel-frequency cepstral coefficient tensor based on the audio data, and perform MLP processing based on the Mel-frequency cepstral coefficient tensor to determine sound features;
[0104] Based on the vehicle state data, the initial vehicle state features are extracted through a convolutional neural network, and MLP processing is performed based on the initial vehicle state features to determine the target vehicle state features;
[0105] The video frame data is determined based on the image data, and features are extracted from the video frame data through a deep learning model to determine semantic features. MLP processing is performed based on the semantic features to determine video features.
[0106] In a possible implementation, the feature fusion module 603 is further configured to:
[0107] The target image features, audio text coefficients and sound features are fused through the first attention network to obtain the first intermediate features.
[0108] In a possible implementation, the feature fusion module 603 is further configured to:
[0109] The video features, audio text coefficients and sound features are fused through the first attention network to obtain the second intermediate features.
[0110] In a possible implementation, the feature fusion module 603 is further configured to:
[0111] Based on the first intermediate feature, the second intermediate feature and the target vehicle state feature, they are fused through the second attention network to obtain a fused feature.
[0112] In a possible implementation, the feature fusion module 603 is further configured to:
[0113] Perform model training based on fusion features and determine the target deep learning model;
[0114] Driver behavior prediction based on targeted deep learning models.
[0115] The embodiments of this specification provide a multimodal data fusion method and device for driver behavior prediction, wherein the multimodal data fusion device for driver behavior prediction includes: obtaining initial data; wherein the initial data includes image data, audio data, and vehicle status data; determining data preprocessing rules based on the category of the initial data, and processing the initial data based on the data preprocessing rules to determine multimodal features; performing data fusion based on the category of the multimodal features to determine fusion features. By obtaining initial data; determining data preprocessing rules based on the category of the initial data, and processing the initial data based on the data preprocessing rules to determine multimodal features; performing data fusion based on the category of the multimodal features to determine fusion features, it is possible to integrate information from different data sources together to more comprehensively and accurately capture the characteristics of driver behavior.
[0116] The above is a schematic scheme of a multimodal data fusion device for driver behavior prediction of this embodiment. It should be noted that the technical scheme of the multimodal data fusion device for driver behavior prediction and the technical scheme of the multimodal data fusion method for driver behavior prediction belong to the same concept, and the details not described in detail in the technical scheme of the multimodal data fusion device for driver behavior prediction can be referred to the description of the technical scheme of the multimodal data fusion method for driver behavior prediction.
[0117] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.
[0118] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0119] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0120] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 may also be a mobile or stationary server.
[0121] Among them, the processor 720 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned multimodal data fusion method for driver behavior prediction. The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned multimodal data fusion method for driver behavior prediction belong to the same concept. The details of the technical scheme of the computing device that are not described in detail can be found in the description of the technical scheme of the above-mentioned multimodal data fusion method for driver behavior prediction.
[0122] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal data fusion method for driver behavior prediction.
[0123] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimodal data fusion method for driver behavior prediction belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the multimodal data fusion method for driver behavior prediction.
[0124] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multimodal data fusion method for driver behavior prediction.
[0125] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the multimodal data fusion method for driver behavior prediction described above belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the multimodal data fusion method for driver behavior prediction described above.
[0126] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0127] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0128] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0130] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A multimodal data fusion method for driver behavior prediction, characterized in that: include: Acquire initial data; wherein the initial data includes image data, audio data and vehicle status data; Determining a data preprocessing rule based on the category of the initial data, and processing the initial data based on the data preprocessing rule to determine a multimodal feature; Data fusion is performed based on the categories of the multimodal features to determine fusion features.
2. The method according to claim 1, characterized in that The obtaining of initial data comprises: Acquire image data through the in-vehicle camera; Acquire audio data through the microphone in the car; Acquire vehicle status data through vehicle sensors; Initial data is determined based on the image data, the audio data, and the vehicle status data.
3. The method according to claim 1, characterized in that The processing of the initial data based on the data preprocessing rule to determine the multimodal features includes: Cropping the image data, and performing feature extraction on the cropped data based on a time window to determine initial image features; Performing dimensionality reduction processing on the initial image features to determine target image features; Converting the audio data into text information, and determining an audio text coefficient based on the text information; Determine a Mel-frequency cepstral coefficient tensor based on the audio data, and perform MLP processing based on the Mel-frequency cepstral coefficient tensor to determine sound features; Based on the vehicle state data, the initial vehicle state features are extracted through a convolutional neural network, and MLP processing is performed based on the initial vehicle state features to determine the target vehicle state features; The video frame data is determined based on the image data, features are extracted from the video frame data through a deep learning model to determine semantic features, and MLP processing is performed based on the semantic features to determine video features.
4. The method according to claim 3, characterized in that The performing data fusion based on the category of the multimodal features to determine the fusion features includes: The target image feature, the audio text coefficient and the sound feature are fused through a first attention network to obtain a first intermediate feature.
5. The method according to claim 4, characterized in that The performing data fusion based on the category of the multimodal features to determine the fusion features includes: The video feature, the audio text coefficient and the sound feature are fused through the first attention network to obtain a second intermediate feature.
6. The method according to claim 5, characterized in that The performing data fusion based on the category of the multimodal features to determine the fusion features includes: Based on the first intermediate feature, the second intermediate feature and the target vehicle state feature, a fusion feature is obtained by fusing them through a second attention network.
7. The method according to claim 1, characterized in that Also includes: Perform model training based on the fusion features to determine a target deep learning model; Driver behavior prediction is performed based on the target deep learning model.
8. A multimodal data fusion device for driver behavior prediction, characterized in that: include: A data acquisition module is configured to acquire initial data; wherein the initial data includes image data, audio data and vehicle status data; a feature extraction module, configured to determine a data preprocessing rule based on the category of the initial data, and process the initial data based on the data preprocessing rule to determine a multimodal feature; The feature fusion module is configured to perform data fusion based on the categories of the multimodal features and determine fusion features.
9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the multimodal data fusion method for driver behavior prediction described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal data fusion method for driver behavior prediction as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Driving decision-making system and method fusing vision and sensor information
CN111311945A
Behavior recognition method and system based on multi-modal feature fusion
CN115205729A
Flotation control parameter prediction method and system based on multi-modal time sequence information
CN116184843A
Method for generating driver state text in intelligent cabin environment and related equipment
CN116935063A
Man-machine interaction system and method based on multi-modal feature fusion
CN117193524A
