Multi-view data fusion driver behavior prediction method and device
Through the triple camera, multi-view data is collected and combined with the fully connected network layer, the driver's movements and eye status are analyzed, and the problems of limited viewing angle and poor robustness of a single camera are solved, achieving more accurate driver behavior prediction and safety monitoring.
Patent Information
- Application Number
- CN202410967440.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-18
AI Technical Summary
When monitoring driver abnormal behavior, the limited viewing angle of a single camera leads to occlusion problems, making it difficult to fully and accurately predict drivers' behavior, especially when wearing glasses or sunglasses, which are poorly robust and physiological signal monitoring. Wearable devices are required, and the application is inconvenient.
Three cameras are used to collect multi-view data, and through frame sampling and feature fusion, combined with the fully connected network layer, the driver's overall movement and eye state are analyzed to generate more accurate behavior predictions.
It effectively avoids occlusion problems, improves the accuracy of behavior prediction, can comprehensively monitor drivers' abnormal behavior, promptly provide early warnings, and ensure travel safety.
Smart Images

Figure CN118918568A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of data processing technology, and in particular to a driver behavior prediction method using multi-view data fusion. Background Art
[0002] Abnormal behaviors of drivers in the car include: smoking, making phone calls, eating, drinking, and driving while fatigued, etc. Any of the above behaviors will have an adverse effect on the safe driving of the vehicle, especially fatigue driving, which will greatly prolong the driver's reaction time. In an emergency, it is difficult for the driver to make standardized operations in time, which will lead to traffic accidents. Therefore, in the field of assisted driving, real-time monitoring of abnormal behaviors of drivers is an important means to ensure the safety of drivers' travel. At present, the methods for monitoring abnormal behaviors of drivers mainly include monitoring methods based on physiological signals and monitoring methods based on vision. The monitoring technology based on physiological signals mainly identifies the fatigue and distraction state of the driver through real-time monitoring of physiological signals such as electrocardiogram (ECG), electroencephalogram (EEG) and skin galvanic response. The monitoring technology based on vision mainly captures the driver's facial expressions, eye movements and body movements through the camera, and uses convolutional neural network (CNN) and other deep learning models for analysis to provide abnormal behavior prediction.
[0003] Methods for monitoring physiological signals often require wearing instruments and equipment, which are difficult to apply in actual scenarios; and existing solutions based on visual information mostly focus on the first four or the last of abnormal behaviors such as smoking, making phone calls, eating, drinking, and fatigue driving, that is, they only focus on the driver's actions or whether the driver is fatigued. However, such methods cannot fully and accurately predict the driver's behavior; at the same time, most existing methods use a single camera to collect image data, so due to the limited viewing angle, it is easy to cause occlusion and inaccurate behavior detection; at the same time, due to the complexity of the actual situation, the driver may wear glasses, sunglasses, etc., and the robustness of the existing eye key point positioning method is poor. Therefore, a better solution is urgently needed. Summary of the invention
[0004] In view of this, the embodiments of this specification provide a multi-view data fusion driver behavior prediction method. One or more embodiments of this specification also involve a multi-view data fusion driver behavior prediction device, a computing device, a computer readable storage medium and a computer program to solve the technical defects existing in the prior art.
[0005] According to a first aspect of an embodiment of this specification, a multi-view data fusion driver behavior prediction method is provided, comprising:
[0006] Acquire image data; wherein the image data includes a first side image, a second side image, and a front image;
[0007] Perform frame sampling based on the image data to determine frame data;
[0008] Determine a multi-view feature vector based on the frame data;
[0009] Feature fusion is performed based on multi-view feature vectors to determine the fusion features, and the fusion features are input into the fully connected network layer to determine the driver behavior results.
[0010] In a possible implementation, acquiring image data includes:
[0011] Acquire a first side image of the driver through a first camera;
[0012] acquiring a second side image of the driver through a second camera;
[0013] Acquiring a frontal image of the driver through a third camera;
[0014] Image data is determined based on the first lateral image, the second lateral image, and the frontal image.
[0015] In a possible implementation, performing frame sampling based on image data to determine frame data includes:
[0016] determining a first interval and a second interval;
[0017] Perform frame sampling based on the first side image, the second side image and the first interval to determine action frame data;
[0018] Frame sampling is performed based on the frontal image and the second interval to determine eye frame data.
[0019] In a possible implementation, determining a multi-view feature vector based on frame data includes:
[0020] Perform motion analysis on the motion frame data to determine the motion feature vector;
[0021] Eye feature extraction is performed on the eye frame data to determine the eye feature vector.
[0022] In a possible implementation, performing motion analysis on the motion frame data to determine the motion feature vector includes:
[0023] Extract skeleton sequence data based on action frame data;
[0024] Determine angle data based on skeleton sequence data, and input the angle data into a convolutional model to determine initial motion features;
[0025] Feature fusion is performed based on initial action features corresponding to the first side image and the second side image to determine an action feature vector.
[0026] In a possible implementation, extracting eye features from eye frame data to determine an eye feature vector includes:
[0027] Perform feature extraction based on eye frame data to determine initial eye features;
[0028] Perform attention calculation based on the initial eye features and determine the attention weight;
[0029] Perform feature fusion based on the attention weight and the original feature map corresponding to the eye frame data to obtain the target eye feature;
[0030] The eye state is evaluated based on the target eye features to determine the eye feature vector.
[0031] In a possible implementation, performing feature fusion based on multi-view feature vectors to determine fusion features includes:
[0032] Determine eye feature weights and motion feature weights;
[0033] Feature fusion is performed based on the eye feature weight, the motion feature weight, the motion feature vector and the eye feature vector to determine the fusion feature.
[0034] According to a second aspect of an embodiment of this specification, a multi-view data fusion driver behavior prediction device is provided, comprising:
[0035] A data acquisition module is configured to acquire image data; wherein the image data includes a first side image, a second side image and a front image;
[0036] A data sampling module is configured to perform frame sampling based on the image data to determine frame data;
[0037] A feature determination module, configured to determine a multi-view feature vector based on the frame data;
[0038] The feature fusion module is configured to perform feature fusion based on multi-view feature vectors to determine fusion features, and input the fusion features into a fully connected network layer to determine the driver behavior results.
[0039] According to a third aspect of an embodiment of this specification, a computing device is provided, including:
[0040] Memory and processor;
[0041] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned multi-view data fusion driver behavior prediction method are implemented.
[0042] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned multi-view data fusion driver behavior prediction method are implemented.
[0043] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multi-view data fusion driver behavior prediction method.
[0044] The embodiments of this specification provide a multi-view data fusion driver behavior prediction method and device, wherein the multi-view data fusion driver behavior prediction method includes: acquiring image data; wherein the image data includes a first side image, a second side image and a front image; performing frame sampling based on the image data to determine the frame data; determining a multi-view feature vector based on the frame data; performing feature fusion based on the multi-view feature vector to determine the fusion feature, and inputting the fusion feature into a fully connected network layer to determine the driver behavior result. By adopting three cameras and analyzing the driver's behavior from two aspects of overall movement and eye state, the occlusion problem can be effectively avoided and the prediction accuracy can be increased. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 is a flowchart of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0046] Figure 2 It is a schematic diagram of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0047] Figure 3 It is a scene diagram of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0048] Figure 4 This is a schematic diagram of action feature extraction of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0049] Figure 5 It is a schematic diagram of key points of a human body in a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0050] Figure 6 This is a feature extraction network architecture diagram of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0051] Figure 7 It is a schematic diagram of eye key point detection of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0052] Figure 8 It is a schematic diagram of the positions of key eye points of a multi-view data fusion driver behavior prediction method provided by an embodiment of this specification;
[0053] Fig. 9 It is a structural schematic diagram of a multi-view data fusion driver behavior prediction device provided by an embodiment of this specification;
[0054] Fig.10 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0055] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0056] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0057] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0058] In this specification, a multi-perspective data fusion driver behavior prediction method is provided. This specification also involves a multi-perspective data fusion driver behavior prediction device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0059] See also Figure 1 , Figure 1 A flowchart of a multi-view data fusion driver behavior prediction method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0060] Step 101: Acquire image data; wherein the image data includes a first side image, a second side image and a front image.
[0061] See also Figure 2 The embodiments of this specification take into account the particularity of the in-vehicle scene and the characteristics of the driver's abnormal behavior, use three cameras to collect data, and realize comprehensive detection of the driver's abnormal behavior based on multi-view data of the driver's limb movements and eye characteristics. Finally, the safety level of the driver's behavior is output through decision fusion, which can effectively reduce the problems caused by image occlusion, and comprehensively monitor abnormal behavior, and timely issue warnings to the driver's dangerous driving behavior, thereby ensuring travel safety.
[0062] In one possible implementation, acquiring image data includes: acquiring a first side image of the driver through a first camera; acquiring a second side image of the driver through a second camera; acquiring a front image of the driver through a third camera; and determining image data based on the first side image, the second side image, and the front image.
[0063] In practical applications, see Figure 3 First, multi-view data acquisition is carried out. Three cameras are used in the car to collect data. Cameras 1 and 2 are on the left and right sides of the driver, responsible for collecting motion information. Camera 3 faces the driver and is responsible for collecting face and eye information.
[0064] The embodiment of this specification uses three cameras and analyzes the driver's behavior from two aspects: overall movement and eye state, which can effectively avoid occlusion problems and increase prediction accuracy.
[0065] Step 102: Perform frame sampling based on the image data to determine frame data.
[0066] In one possible implementation, frame sampling is performed based on image data to determine frame data, including: determining a first interval and a second interval; performing frame sampling based on the first side image, the second side image, and the first interval to determine action frame data; and performing frame sampling based on the front image and the second interval to determine eye frame data.
[0067] In practical applications, this method uses different intervals of equally divided frame sampling methods for different cameras in response to different processing information. The video information collected by the camera is tentatively set at 30 frames per second. A video clip is a unit for extracting feature information. The time to capture actions such as drinking, smoking, eating, nodding, etc. is about 0.5 seconds to 3 seconds, and the time to capture fatigue states such as blinking is about 0.25 seconds. Therefore, the frame sampling module of this method uses different numbers of sampling clips in different processing branches. In action recognition, the sampling interval is 2 frames, and in eye feature recognition, the interval is 1 frame. The two branches use 36 frames and 18 frames as a sampling clip for sampling. The sampling order is from first to last, which ensures the timing of network sampling, while reducing the amount of sampling tasks and improving system efficiency.
[0068] In the embodiment of this specification, when sampling frames, the sampling strategy is to use an equally divided frame sampling method with different intervals for different cameras according to different processing information, which can improve system efficiency.
[0069] Step 103: Determine a multi-view feature vector based on the frame data.
[0070] In practical applications, see Figure 3 The data collected by the No. 1 and No. 2 cameras are frame sampled respectively and then merged into the action feature extraction module to obtain the intermediate features f1 and f2, and then the two perspective features are fused to obtain the action feature F1; the data collected by the No. 3 camera is frame sampled and then input into the eye key point positioning module to locate the key points, and then the EAR value is calculated to obtain the eye feature F2.
[0071] In a possible implementation, determining a multi-view feature vector based on frame data includes: performing motion analysis on the action frame data to determine the motion feature vector; and performing eye feature extraction on the eye frame data to determine the eye feature vector.
[0072] For details, see Figure 4 , performing motion analysis on the action frame data to determine the action feature vector, including: extracting skeleton sequence data based on the action frame data; determining angle data based on the skeleton sequence data, and inputting the angle data into a convolution model to determine initial motion features; performing feature fusion based on the initial motion features corresponding to the first side image and the second side image to determine the action feature vector.
[0073] For example, due to its high reliability, Openpose can be selected as the front-end network of the system to process continuous video streams. Since video data needs to be processed in real time, and considering that in driving scenarios, the recognition of driver actions mainly involves the skeleton sequence of the driver's upper body, in order to ensure accuracy while saving computing overhead as much as possible, 7 key points can be selected as the main skeleton of the human body, such as Figure 5 shown.
[0074] Table 1
[0075] Annotation Order Joint name 0 nose 1 left elbow 2 left wrist 3 right elbow 4 right wrist 5 neck 6 Base of the spine
[0076] Further, referring to Table 1, after the openpose model, a heat map of human key points can be obtained. NMS (non-maximum suppression) is applied to the heat map of human key points to obtain the scattered coordinates of human key points. Then, the affinity between key point pairs is calculated by 8-point discrete sampling line integrals combined with PAF (partial affinity field) information. Finally, the KM algorithm is applied to assemble the human skeleton sequence required by this method for use in subsequent modules.
[0077] After the human skeleton sequence is output, it will be used as the input of the action recognition network for detection. Because the recognition of action involves many factors, for example, during driving, a slow turn of the head may be the driver actively relaxing his shoulders and neck, but if the head position changes too quickly, it may be a short nod of sleepiness caused by driving fatigue. Therefore, this technology uses three branches of angle, angular velocity, and angular acceleration for action recognition, and then fuses the results after convolution. The structural diagram is shown in the figure below. Figure 6 shown.
[0078] The network input size is 6*36*3, which is composed of 3 independent data stacks, namely angle sequence, angular velocity sequence and angular acceleration sequence. They describe the motion information from 3 dimensions for subsequent network use. This method combines the posture characteristics of the driver during driving and defines a human skeleton consisting of 7 key points. The 7 key points can form 6 skeletons to minimize system expenses. The essence of action is the movement of the skeleton, so the skeleton sequence theoretically contains all the information needed to understand the action. For the driver, the change of human posture is mainly reflected in the change of the angle between each segment of the skeleton and the reference plane. Therefore, we first calculate the angle between each segment of the skeleton and the horizontal plane based on the skeleton information of each frame to form a 6-dimensional feature vector:
[0079] f=[θ 1 ,θ 2 ,θ 3 ,θ 4 ,θ 5 ,θ 6 ]
[0080] The angle of each bone segment is calculated by the inverse tangent formula:
[0081]
[0082] In actual operation, the direction of the bone vector should be determined first, and the angle value should be corrected according to the quadrant in which it is located, so that the angles of all bones are between [0, 2π). We manually extract the features of the skeleton of each frame to form a 6-dimensional feature vector. In this technology, we set the sliding window size for each recognition to 36 frames, that is, each time 6*36 data is sent to the first branch of the subsequent convolution module for calculation and output of intermediate features.
[0083]
[0084] We can obtain the angular velocity and displacement velocity sequence by performing the first-order difference on the obtained angle and center of gravity position sequence:
[0085] The bone angle range we defined in the previous article is [0, 2π), and the counterclockwise direction is the positive direction, so a positive angular velocity means that the bone moves counterclockwise, and a negative angular velocity means the opposite. Similarly, we continue to differentiate the angular velocity and displacement velocity sequences to obtain the acceleration sequence:
[0086]
[0087] The above operation decomposes the obtained skeleton sequence into three sequences of angle, angular velocity and angular acceleration required by the subsequent network, providing the network with relatively rich action semantic information from different dimensions. Then the data of the three branches are input into the convolution model, the convolution kernel size is 6*4, the step length is 4, and the convolution is performed along the frame length. Finally, the 6*9 feature data output by each branch is feature fused, the three channels are converted to a single channel, the extracted features are reduced in dimension, and then the features f1 and f2 of the two cameras are fused to obtain the driver action recognition feature F1 (18*1).
[0088] In the embodiment of this specification, when identifying actions, the design only selects 7 skeleton key points based on the driver's action characteristics, which can effectively reduce the calculation overhead while basically meeting the actual needs, and is suitable for application in real-time systems.
[0089] Specifically, eye feature extraction is performed on the eye frame data to determine the eye feature vector, including: extracting features based on the eye frame data to determine initial eye features; calculating attention based on the initial eye features to determine attention weights; performing feature fusion based on the attention weights and the original feature maps corresponding to the eye frame data to obtain target eye features; and evaluating the eye state based on the target eye features to determine the eye feature vector.
[0090] In practical applications, the self-attention mechanism is a technology that can dynamically adjust the attention distribution according to the internal relationship of the input data. In this mechanism, each input position is assigned a relationship weight with other positions, and then the model's attention to the input is adjusted according to these weights. The eye key point detection method based on the self-attention mechanism can dynamically adjust the model's attention to the eye key points according to the characteristics of the eye area in the input image, thereby improving the accuracy and robustness of eye key point detection.
[0091] See also Figure 7 First, the input driver face image is extracted through a convolutional neural network (CNN) to obtain a feature map. For each position in the feature map, it is mapped to a low-dimensional feature vector. For the feature vector of each position, a fully connected network is used to calculate the relationship weight with other positions. Then, according to the calculated attention weight, the feature vector of each position in the feature map is weighted and summed to obtain a weighted feature representation. The weighted feature representation is fused with the original feature map to obtain the final feature representation.
[0092] The self-attention network consists of three main parts: feature extraction module, attention calculation module and feature fusion module.
[0093] Feature extraction module: Convolutional neural network (CNN) is used to extract the features of the input image.
[0094] Attention calculation module: Use a fully connected network, the input of which is the feature vector of each position in the feature map, and the output is the attention weight of the corresponding position. Through this fully connected network, the model can learn the importance of each position in the entire input image and adjust the model's attention to the input based on these weights.
[0095] Feature fusion module: applies the calculated attention weights to the feature map and fuses the weighted features with the original feature map.
[0096] After obtaining the coordinates of the key points of the eye, the EAR value is calculated. The EAR value of the eye refers to the degree of eye closure (EyeAspect Ratio). It is an indicator used to evaluate the state of the eye.
[0097]
[0098] Among them, p1, p2, p3, p4, p5, and p6 represent the coordinates of the key points of the eyes, and their positions are as follows: Figure 8 As shown, the EAR values of the two eyes for 18 frames are calculated each time, and the EAR values of the two eyes are averaged to obtain the EAR vector F2 (18*1).
[0099] The embodiments of this specification propose a new eye key point detection method. The eye key point detection method based on the self-attention mechanism can dynamically adjust the model's attention to the eye key points according to the characteristics of the eye area in the input image, thereby improving the accuracy and robustness of eye key point detection.
[0100] Step 104: Perform feature fusion based on the multi-view feature vectors to determine fusion features, and input the fusion features into a fully connected network layer to determine the driver behavior results.
[0101] In practical applications, the action features and eye features are weighted and fused, and finally the danger level of the driver's behavior is output from 1 to 10 through a fully connected network.
[0102] In a possible implementation, feature fusion is performed based on multi-view feature vectors to determine fusion features, including: determining eye feature weights and motion feature weights; and performing feature fusion based on the eye feature weights, motion feature weights, motion feature vectors, and eye feature vectors to determine fusion features.
[0103] In practical applications, the action recognition features and eye fatigue features obtained in the above steps are weighted and fused, and different weights are assigned to the F1 feature and the F2 feature.
[0104] Finally, the fully connected network outputs the driver's action danger level from 0 to 10, where 0 represents normal driving. Then, a reward and punishment mechanism can be designed based on the action danger level to ensure the driver's driving safety.
[0105] The embodiments of this specification provide a multi-view data fusion driver behavior prediction method and device, wherein the multi-view data fusion driver behavior prediction method includes: acquiring image data; wherein the image data includes a first side image, a second side image and a front image; performing frame sampling based on the image data to determine the frame data; determining a multi-view feature vector based on the frame data; performing feature fusion based on the multi-view feature vector to determine the fusion feature, and inputting the fusion feature into a fully connected network layer to determine the driver behavior result. By adopting three cameras and analyzing the driver's behavior from two aspects of overall movement and eye state, the occlusion problem can be effectively avoided and the prediction accuracy can be increased.
[0106] Corresponding to the above method embodiment, this specification also provides a multi-view data fusion driver behavior prediction device embodiment, Fig. 9 FIG. 1 is a schematic diagram showing the structure of a multi-view data fusion driver behavior prediction device provided by an embodiment of this specification. Fig. 9 As shown, the device comprises:
[0107] The data acquisition module 901 is configured to acquire image data; wherein the image data includes a first side image, a second side image and a front image;
[0108] The data sampling module 902 is configured to perform frame sampling based on the image data to determine the frame data;
[0109] A feature determination module 903 is configured to determine a multi-view feature vector based on the frame data;
[0110] The feature fusion module 904 is configured to perform feature fusion based on the multi-view feature vectors to determine the fusion feature, and input the fusion feature into the fully connected network layer to determine the driver behavior result.
[0111] In a possible implementation, the data acquisition module 901 is further configured to:
[0112] Acquire a first side image of the driver through a first camera;
[0113] acquiring a second side image of the driver through a second camera;
[0114] Acquiring a frontal image of the driver through a third camera;
[0115] Image data is determined based on the first lateral image, the second lateral image, and the frontal image.
[0116] In a possible implementation, the data sampling module 902 is further configured to:
[0117] determining a first interval and a second interval;
[0118] Perform frame sampling based on the first side image, the second side image and the first interval to determine action frame data;
[0119] Frame sampling is performed based on the frontal image and the second interval to determine eye frame data.
[0120] In a possible implementation, the feature determination module 903 is further configured to:
[0121] Perform motion analysis on the motion frame data to determine the motion feature vector;
[0122] Eye feature extraction is performed on the eye frame data to determine the eye feature vector.
[0123] In a possible implementation, the feature determination module 903 is further configured to:
[0124] Extract skeleton sequence data based on action frame data;
[0125] Determine angle data based on skeleton sequence data, and input the angle data into a convolutional model to determine initial motion features;
[0126] Feature fusion is performed based on initial action features corresponding to the first side image and the second side image to determine an action feature vector.
[0127] In a possible implementation, the feature determination module 903 is further configured to:
[0128] Perform feature extraction based on eye frame data to determine initial eye features;
[0129] Perform attention calculation based on the initial eye features and determine the attention weight;
[0130] Perform feature fusion based on the attention weight and the original feature map corresponding to the eye frame data to obtain the target eye feature;
[0131] The eye state is evaluated based on the target eye features to determine the eye feature vector.
[0132] In a possible implementation, the feature fusion module 904 is further configured to:
[0133] Determine eye feature weights and motion feature weights;
[0134] Feature fusion is performed based on the eye feature weight, the motion feature weight, the motion feature vector and the eye feature vector to determine the fusion feature.
[0135] The embodiments of this specification provide a multi-view data fusion driver behavior prediction method and device, wherein the multi-view data fusion driver behavior prediction device includes: acquiring image data; wherein the image data includes a first side image, a second side image and a front image; performing frame sampling based on the image data to determine the frame data; determining a multi-view feature vector based on the frame data; performing feature fusion based on the multi-view feature vector to determine the fusion feature, and inputting the fusion feature into a fully connected network layer to determine the driver behavior result. By adopting three cameras and analyzing the driver's behavior from two aspects of overall movement and eye state, the occlusion problem can be effectively avoided and the prediction accuracy can be increased.
[0136] The above is a schematic scheme of a multi-perspective data fusion driver behavior prediction device of this embodiment. It should be noted that the technical scheme of the multi-perspective data fusion driver behavior prediction device and the technical scheme of the multi-perspective data fusion driver behavior prediction method mentioned above belong to the same concept. For details not described in detail in the technical scheme of the multi-perspective data fusion driver behavior prediction device, please refer to the description of the technical scheme of the multi-perspective data fusion driver behavior prediction method mentioned above.
[0137] Fig.10The block diagram of a computing device 1000 according to an embodiment of the present specification is shown. The components of the computing device 1000 include but are not limited to a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.
[0138] The computing device 1000 also includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0139] In one embodiment of the present specification, the above components of the computing device 1000 and Fig.10 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.10 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0140] The computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1000 may also be a mobile or stationary server.
[0141] Among them, the processor 1020 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned multi-perspective data fusion driver behavior prediction method. The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned multi-perspective data fusion driver behavior prediction method belong to the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned multi-perspective data fusion driver behavior prediction method.
[0142] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned multi-view data fusion driver behavior prediction method.
[0143] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multi-view data fusion driver behavior prediction method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the multi-view data fusion driver behavior prediction method mentioned above.
[0144] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multi-view data fusion driver behavior prediction method.
[0145] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the multi-view data fusion driver behavior prediction method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the multi-view data fusion driver behavior prediction method mentioned above.
[0146] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0147] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0148] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0149] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0150] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.
Claims
1. A multi-view data fusion driver behavior prediction method, characterized in that: include: Acquire image data; wherein the image data includes a first side image, a second side image and a front image; Perform frame sampling based on the image data to determine frame data; Determine a multi-view feature vector based on the frame data; Based on the multi-view feature vector, feature fusion is performed to determine fusion features, and the fusion features are input into a fully connected network layer to determine the driver behavior result.
2. The method according to claim 1, characterized in that The acquiring of image data comprises: Acquire the first side image of the driver by a first camera; acquiring the second side image of the driver by a second camera; acquiring the front image of the driver by a third camera; Image data is determined based on the first lateral image, the second lateral image, and the frontal image.
3. The method according to claim 1, characterized in that The performing frame sampling based on the image data to determine the frame data includes: determining a first interval and a second interval; Perform frame sampling based on the first side image, the second side image and the first interval to determine action frame data; Frame sampling is performed based on the frontal image and the second interval to determine eye frame data.
4. The method according to claim 3, characterized in that The determining of the multi-view feature vector based on the frame data comprises: Performing motion analysis on the motion frame data to determine a motion feature vector; Eye feature extraction is performed on the eye frame data to determine an eye feature vector.
5. The method according to claim 4, characterized in that The performing motion analysis on the motion frame data to determine the motion feature vector includes: Extracting skeleton sequence data based on the action frame data; Determine angle data based on the skeleton sequence data, and input the angle data into a convolution model to determine initial motion features; Feature fusion is performed based on the initial action features corresponding to the first side image and the second side image to determine an action feature vector.
6. The method according to claim 4, characterized in that The step of extracting eye features from the eye frame data to determine an eye feature vector includes: Performing feature extraction based on the eye frame data to determine initial eye features; Performing attention calculation based on the initial eye features to determine an attention weight; Performing feature fusion based on the attention weight and the original feature map corresponding to the eye frame data to obtain a target eye feature; An eye state assessment is performed based on the target eye feature to determine an eye feature vector.
7. The method according to claim 4, characterized in that The step of performing feature fusion based on the multi-view feature vectors to determine fusion features includes: Determine eye feature weights and motion feature weights; Feature fusion is performed based on the eye feature weight, the action feature weight, the action feature vector and the eye feature vector to determine a fusion feature.
8. A multi-view data fusion driver behavior prediction device, characterized in that: include: A data acquisition module, configured to acquire image data; wherein the image data includes a first side image, a second side image and a front image; A data sampling module is configured to perform frame sampling based on the image data to determine frame data; A feature determination module, configured to determine a multi-view feature vector based on the frame data; The feature fusion module is configured to perform feature fusion based on the multi-view feature vector to determine a fusion feature, and input the fusion feature into a fully connected network layer to determine a driver behavior result.
9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the multi-view data fusion driver behavior prediction method described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multi-view data fusion driver behavior prediction method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Child ADHD screening evaluation system based on a multi-modal deep learning technology
CN111528859A
Posture feature based driver fatigue state detection method and system
CN111616718A
Driver behavior recognition method and device based on posture supplementary features
CN117315633A
Driving behavior detection method based on multiple view angles
CN118015600A
Multi-modal image fusion and processing method based on multi-dimensional attention mechanism
CN118247621A
