Distraction behavior detection method based on multi-stream feature fusion and domain adaptation
By constructing the MSFDA-Net model and combining multi-flow feature fusion and domain adaptive network, the accuracy and generalization problems of distracted driving detection methods in complex backgrounds are solved, achieving higher detection accuracy and generalization ability, and improving road traffic safety.
Patent Information
- Application Number
- CN202411057486.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-12
AI Technical Summary
Existing methods for detecting distracted driving are inadequate in terms of detection accuracy and generalization ability, especially in terms of recognition rate and performance in complex driving scenarios.
We constructed the MSFDA-Net distraction behavior detection model by concatenating a pre-trained Openpose model and a pre-trained YOLOv5 model, combining a multi-stream feature fusion network and a domain adaptive network module, and adopting a cross-domain feature learning strategy to enhance the model's understanding and generalization ability of high-level semantic information.
It improves the accuracy of distracted driving behavior detection in complex scenarios, reduces false positives and false negatives, enhances the model's generalization ability across different drivers and datasets, and promotes the accuracy of road traffic safety detection.
Smart Images

Figure CN118942075A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of behavior detection, and in particular relates to a distraction behavior detection method based on multi-stream feature fusion and domain adaptation. Background Art
[0002] In the complex road traffic system of "people-vehicle-road", driver behavior is one of the important factors affecting traffic safety. The relevant survey results of the Chinese Academy of Social Sciences and an insurance agency showed that 34% of the respondents had distracted driving behaviors during driving. In addition, the report of the US Highway Safety Administration showed that 25% to 30% of traffic accidents in the United States were related to distracted driving behaviors.
[0003] At present, distracted driving detection methods are mainly divided into indirect detection methods and direct detection methods. The cost of indirect detection methods is relatively low and does not affect the driver's operating behavior, but the detection accuracy is easily affected by driving conditions and driving habits, and the research method and accuracy need to be improved. The direct monitoring method has a higher accuracy rate, but the cost of related equipment is high, and it may affect normal driving, and the actual application is difficult.
[0004] With the rapid development of computer vision and deep learning technology, the detection method of distracted driving behavior has made significant progress. The Chinese invention patent with application number 202111564360.0 discloses a driver behavior recognition method based on posture estimation feature fusion. Although it uses YOLO-V4 and OpenPose to jointly obtain the key features of driver behavior, the subsequent feature fusion and the low generalization degree of the combined model itself and the lack of modification methods to improve accuracy still have a lot of room for improvement in the recognition rate in actual applications and the performance in dealing with complex background driving scenes. Summary of the invention
[0005] Purpose of the invention: In order to solve the problems mentioned in the background technology, the present invention provides a distraction behavior detection method based on multi-stream feature fusion and domain adaptation. By constructing the MSFDA-Net distraction behavior detection model, the pre-trained Openpose model and the pre-trained yolov5 model are connected in series to obtain the model backbone, and the multi-stream feature fusion network and the domain adaptive network module are used to establish an end-to-end model. On the basis of the original function of the module, a cross-domain feature learning strategy is adopted to further enhance the model's understanding and generalization ability of high-level semantic information, improve the detection accuracy of distracted driving behavior in complex scenarios, and provide new directions and means for actual driving scenario applications.
[0006] Technical solution:
[0007] The present invention provides a distraction behavior detection method based on multi-stream feature fusion and domain adaptation, the method comprising the following steps:
[0008] S1 obtains driving data and preprocesses it to create a data set;
[0009] S2 builds the MSFDA-Net distraction behavior detection model:
[0010] S2.1 concatenates the pre-trained Openpose model and the pre-trained yolov5 model to perform posture estimation and target object detection on driving data, and calculate body joint features and semantic features representing human-object interactions;
[0011] S2.2 uses multi-stream feature fusion network and domain adaptive network module to establish an end-to-end model;
[0012] S2.2.1 build a multi-stream feature fusion network, process features using feature dimension reduction and feature abstraction, and input them into the domain adaptation network module;
[0013] S2.2.2 The domain adaptation network module adopts a cross-domain feature learning strategy to enhance the model's understanding and generalization capabilities of high-level semantic information;
[0014] S3 adjusts and trains the parameter configuration of the MSFDA-Net distraction behavior detection model to obtain the optimal results and complete distraction behavior detection.
[0015] Furthermore, the S1 implementation process is as follows:
[0016] The distraction behavior data are divided by driver and divided into training set and test set in a ratio of 8:2.
[0017] Furthermore, the S2.1 comprises the following steps:
[0018] The pre-trained Openpose model is used to detect and locate body joints; the pre-trained yolov5s model is used to detect target objects that are frequently used or interacted with during driving. Paired relational features are used to encode the configuration of a pair of joints. The configuration of body joints is modeled as high-level semantic features for recognition. There are N(N-1) / 2 pair combinations of N joints. For each pair, the relational feature vector f is calculated by considering the x and y positions of the joints on the image plane. Given a pair of joints j 1 and j 2 , and their respective positions (x 1 ,y 1 ) and x 2 ,y 2 ), their relationship is expressed using distance r and direction θ:
[0019]
[0020] The range of the angle θ is [-π, π]. It is mapped to the interval [0, 2π] by applying the modulus operator. Then, it is divided into h intervals. The size r weights the corresponding interval where θ is located. The length of the feature vector f is the number of intervals. The relationship feature vector f between a pair of joints is calculated only when both joints are detected. Otherwise, f will be set to zero. This process is performed for all (N(N-1) / 2) pairs, and the extracted paired feature vectors are merged into a single feature vector F = [f 1 ,f 2 ,…,f N(N-1) / 2 ], whose length is N(N-1) / 2×h;
[0021] Histogram of directional relationship features representing the relationship between body joints and objects is extracted in a similar way to the posture feature f. For N body joints and O objects, there are N×O possible pairwise combinations. For each pair, the feature is calculated. and stacked into a single feature vector Its size is N×O×h.
[0022] Furthermore, the S2.2.1 comprises the following steps:
[0023] A lightweight multi-stream feature fusion network consisting of fully connected layers and dropout layers is constructed. A dropout layer is introduced between FC layers with a dropout rate set to 0.5. The fully connected layer constitutes the core of the network and is responsible for learning complex nonlinear mappings in the input data. A sequence of FC layers with different capacities is designed, namely FC(2048), FC(1024) and FC(512), which gradually reduces the spatial dimension of the features and improves the abstraction level of the features.
[0024] Furthermore, the cross-domain feature learning strategy described in S2.2.2 adopts an adversarial method to learn feature representation, and the specific training steps are as follows:
[0025] Assume that x∈X represents the input distraction behavior sample;
[0026] After feature extractor E f (x;θ f )→f, each input sample x is mapped to a feature vector f in the feature space F;
[0027] Through the class label predictor G y (x;θ y )→y, the feature vector f is mapped to the parameter θ y The class label y of
[0028] Through the domain classifier G d (x;θd )→d, with parameter θ d , map the above feature vector f to the domain label d.
[0029] Furthermore, in the adversarial approach, a distraction detection class label predictor is designed to encourage the network to learn cross-domain feature representations;
[0030] The loss function corresponding to the distraction detection class label predictor is defined as:
[0031]
[0032] Among them, L ce is the cross entropy loss function; G y is the class label predictor, θ y is the class label predictor parameter, E f is the feature extractor, x i is the input sample, θ f are the feature extractor parameters; y i is the actual value;
[0033] The loss function for the domain label classifier is defined as:
[0034]
[0035] Among them, L ce is the cross entropy loss function; G d is the domain label classifier, θ d is the domain label classifier parameter; E f is the feature extractor, x i is the input sample, θ f are the feature extractor parameters; d i is the actual value;
[0036] The cross entropy loss function formula is as follows:
[0037]
[0038] Among them, x i Represents the i-th element of the true label, y i Represents the probability that the model predicts that x belongs to the i-th category;
[0039] The overall objective loss function is defined as:
[0040] L total (θ f ,θ y ,θ d )=L y (θ f ,θ y )-λL d (θf ,θ d )
[0041] Where λ represents L y and L d The antagonistic balance parameters;
[0042] To achieve classification accuracy and uniform alignment of the source and target domains, the feature extractor parameters θf and class label classifier parameters θy are minimized, and the domain classifier parameters θd are maximized;
[0043] (θ f ,θ y ) = arg min L total (θ f ,θ y ,θ d )
[0044] (θ f ,θ d ) = arg max L total (θ f ,θ y ,θ d )
[0045] Furthermore, the detailed steps of S3 are as follows:
[0046] The input image size is 224×224, the number of iterations is 100 rounds, the Batch_Size is 16, the bin number h of the pose feature is 12, the norm is L2, and MultiStepLR is used for learning rate decay. The learning rate is 0.01, the learning rate momentum is 0.9, the weight decay is 5×10-3, gamma=0.05, and decay=0.75.
[0047] Beneficial effects:
[0048] 1. The present invention uses the interactive information between the driver and the objects in the car to assist in detection, so that the model can understand the driver's behavior more accurately in the process of identifying and fusing features, thereby reducing the occurrence of misjudgments and missed judgments.
[0049] 2. The present invention uses a multi-stream feature fusion network and a domain adaptive network module to establish an end-to-end model. Based on the original module functions of the two, the end-to-end model is further enhanced in the MSFDA-Net distraction behavior detection model through feature dimension reduction and feature abstraction feature processing methods and domain adaptive network cross-domain feature learning strategy. The generalization ability of the end-to-end model for driving distraction behavior application scenarios is further enhanced, the detection accuracy in complex background influences is improved, and the driver's actions are more accurately identified, which plays a driving role in maintaining road traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a flow chart of the present invention;
[0051] Figure 2 It is the process of extracting features of body joints;
[0052] Figure 3 The process of extracting the feature histogram of the human-object directional relationship;
[0053] Figure 4 This is a diagram of the multi-stream feature fusion network structure;
[0054] Figure 5 This is the domain adaptation network structure diagram. DETAILED DESCRIPTION
[0055] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.
[0056] In order to better understand the present invention, the following is a more detailed description of a distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to the present invention in conjunction with the accompanying drawings in the examples of the present invention. Figure 1 As shown, the specific steps are as follows:
[0057] Step 1: Obtain the target data and preprocess it to complete the data set creation.
[0058] The present invention uses the public datasets StateFarm and AUC.
[0059] The StateFarm dataset includes 22,424 images of 25 drivers. The dataset is randomly divided into a training set and a test set according to 80% and 20% of the number of drivers. 18,051 images of 20 drivers are selected as the training set, and 4,373 images of the remaining 5 drivers are selected as the test set.
[0060] The AUC dataset includes 11,665 images of 31 drivers. The labels of the AUC dataset are the same as those of the StateFarm dataset. According to the suggestions of the dataset creator, 90% and 10% of the drivers are selected as the training set and test set, respectively. 10,542 images of 27 drivers are selected as the training set, and 1,123 images of the remaining 4 drivers are selected as the test set.
[0061] Step 2: Build the MSFDA-Net distraction behavior detection model, connect the pre-trained Openpose model and the pre-trained yolov5 model in series, perform posture estimation and target object detection on the driving data, calculate the body joint features and the semantic features representing the human-object interaction. The detailed steps are as follows:
[0062] Perform pose estimation and target object detection on driving data, and calculate semantic features representing human pose and human-object interaction.
[0063] In order to model the configuration of body joints as high-level semantic features for identifying driver behavior, it is necessary to detect and locate these key parts in a given image. At the same time, in order to simulate the interaction between the human body and objects, it is also necessary to detect target objects that are frequently used or interacted with during driving, such as mobile phones and water cups.
[0064] The present invention uses two pre-trained deep learning models. First, the OpenPose model is used for human posture estimation, which can accurately detect the key joints of the human body. Second, the YOLOv5s model is used for target object detection. It performs well in object detection and can quickly and accurately identify objects such as mobile phones and water cups in driving scenes.
[0065] First, the detected key joints are preprocessed: Due to the inevitable partial occlusion or lighting conditions caused by the driving environment, the detected joints are noisy. These noises are:
[0066] (1) Detected passenger joints; (2) Pedestrians on the roadside (visible through the car window); (3) Joints of one person are divided into multiple people; (4) Missing joints; (5) False detections. Noise is reduced by applying the following methods on the detected joints: (1) For each detected joint point, calculate a bounding box (BB) containing these joint points. If multiple bounding boxes are detected, calculate the IoU to find their overlap; (2) If there is significant overlap between two or more bounding boxes (IoU ≥ 5%), then check whether the detected joint point is assigned to multiple people, that is, multiple bounding boxes. This is done by simply looking at the correspondence of the joint points. Assume that there are two overlapping bounding boxes (B1 and B2), find the correspondence of each joint point in the overlapping bounding boxes. There are three possibilities: 1) The joint point appears in both B1 and B2, 2) It does not exist in either, and 3) It only appears in one of them. In the first case, consider the joint point with the higher detection score. In the second case, mark it as not present, and in the third case, consider the joint point. This process continues until all detected joint points have been visited; (3) When there is no or minimal overlap, the bounding box with the largest area and the largest number of detected joint points is considered to be the driver, because the driver occupies most of the image; (4) If one or more smaller bounding boxes appear within a large bounding box. In this case, the smaller bounding boxes are usually pedestrians on the roadside who are visible through the car window and appear in the image. The present invention simply considers the larger bounding box (i.e., the driver) because the driver is visible and occupies a larger image area.
[0067] Use pairwise relational features to encode the configuration of a pair of joints. The configuration of body joints is modeled as high-level semantic features for recognition. If there are N joints, then there are N(N-1) / 2 possible pairwise combinations. For each pair, the relational feature vector f is calculated by considering the x, y positions of the joints on the image plane. Given a pair of joints j 1 and j 2 , and their respective positions (x 1 ,y 1 ) and (x 2 ,y 2 ), their relationship is expressed using distance r and direction θ:
[0068]
[0069] The angle θ ranges from [-π, π] and is mapped to the interval [0, 2π] by applying the modulo operator. Then, it is divided into h intervals, and the size r weights the corresponding interval where θ is located. The length of the feature vector f is the number of intervals (i.e., h). The process of extracting features from 8 intervals (h = 8) is as follows Figure 2The relationship feature vector f between a pair of joints is calculated only when both joints are detected. Otherwise, f is set to zero. This process is repeated for all possible (N(N-1) / 2) pairs and the extracted pairs of feature vectors are merged into a single feature vector F = [f 1 ,f 2 ,…,f N(N-1) / 2 ], whose length is N(N-1) / 2×h.
[0070] Histogram of directional relationship features representing the relationship between body joints and objects is extracted in a similar way to the posture feature f, such as Figure 3 For N body joints and O objects, there are N×O possible pairwise combinations. For each pair, compute the feature and stacked into a single feature vector Its size is N×O×h (h is the number of direction intervals).
[0071] Step 3: Build a multi-stream feature fusion network to fuse multiple features for distraction behavior detection.
[0072] The present invention proposes a multi-stream feature fusion network for fusing body joint features and human-object interaction features, such as Figure 4 As shown in Figure 2, this deep neural network architecture consists of a fully connected layer (FC) and a dropout layer, which together achieve effective extraction and fusion of key features.
[0073] The fully connected layers form the core of the network and are responsible for learning complex nonlinear mappings in the input data. A sequence of FC layers with different capacities is designed, namely FC(2048), FC(1024), and FC(512). These layers not only gradually reduce the spatial dimension of the features, but also gradually increase the level of abstraction of the features. For example, the FC(2048) layer, with its powerful 2048 neurons, is able to handle high-dimensional feature representations and capture subtle correlations between input features. Subsequently, the FC(512) layer further refines these features to provide a more compact and abstract feature vector for the action recognition task.
[0074] In order to improve the generalization ability of the network and avoid overfitting, a dropout layer is introduced between the FC layers. The dropout rate is set to 0.5, which ensures that approximately half of the neurons are randomly disabled in each training iteration. This strategy forces the network to learn a more robust feature representation and encourages the network to develop a distributed feature learning mechanism.
[0075] In addition, the lightweight design of the multi-stream feature fusion network significantly reduces the demand for computing resources without compromising performance. The advantage of this design is that the network is not only suitable for high-performance servers, but also for resource-constrained devices such as mobile devices or embedded systems, thus broadening its application range.
[0076] The network's multi-stream input support capability enables it to process features from different sensors or data sources simultaneously. This multi-stream architecture not only integrates the dynamic changes of body joints, but also considers the complex interactions between the human body and objects, providing rich and accurate contextual information for behavior recognition.
[0077] Step 4: Introduce a domain adaptation network to improve the model's adaptability to differences in data sources.
[0078] In order to improve the cross-driver and cross-dataset generalization problem in distracted behavior detection, a domain adaptation network is introduced and combined with a multi-stream feature fusion network. The multi-stream feature fusion network effectively captures different levels of visual information by gradually reducing the spatial dimension of the features while improving the abstract level of the features. This process not only retains the detailed information in the low-level features, but also strengthens the generalization ability of the high-level features, providing rich input for the domain adaptation network.
[0079] The domain adaptation network uses these multi-level features to reduce the distribution difference between domains by learning the common features and specific features between the source domain and the target domain, thereby improving the generalization of the model. The output of the multi-stream feature fusion network is directly used as the input of the domain adaptation network, so that the model can further learn cross-domain feature representations through adversarial training methods while maintaining sensitivity to key visual details. This strategy can not only enhance the model's understanding of high-level semantic information, but also enhance its generalization ability between different datasets and drivers.
[0080] The detailed steps of the adversarial training method are as follows:
[0081] Assume x∈X represents the input distraction behavior sample. In the feature extractor E f (x;θ f )→f, each input sample x is mapped to a feature vector f on the feature space F. Then, the feature vector f is passed through the class label predictor G y (x;θ y )→y is mapped to the parameter θ y Finally, the feature vector f is passed through the domain label classifier G d (x;θ d )→d is mapped to the parameter θ d The domain label d.
[0082] The loss function corresponding to the distraction detection class label predictor is defined as:
[0083]
[0084] Among them, L ce is the cross entropy loss function; G y is the class label predictor, θ y is the class label predictor parameter; E f is the feature extractor, x i is the input sample, θ f are the feature extractor parameters; y i is the actual value.
[0085] The loss function for the domain label classifier is defined as:
[0086]
[0087] Among them, L ce is the cross entropy loss function; G d is the domain label classifier, θ d is the domain label classifier parameter; E f is the feature extractor, x i is the input sample, θ f are the feature extractor parameters; d i is the actual value.
[0088] The cross entropy loss function formula is as follows:
[0089]
[0090] Among them, x i Represents the i-th element of the true label, y i It represents the probability that the model predicts that x belongs to the i-th category.
[0091] The goal of this invention is to accurately classify the source domain dataset and minimize the classification error of the current distraction detection, as well as to improve the classification accuracy of cross-driver or cross-dataset experiments by aligning the data across drivers or across datasets. Therefore, the overall objective loss function is defined as:
[0092] L total (θ f ,θ y ,θ d )=L y (θ f ,θ y )-λL d (θ f ,θ d )
[0093] Where λ represents L y and L dThe antagonistic balance parameters.
[0094] To achieve classification accuracy and uniform alignment of source and target domains, the feature extractor parameters θf and class label classifier parameters θy are minimized, and the domain classifier parameters θd are maximized.
[0095] (θ f ,θ y ) = arg min L total (θ f ,θ y ,θ d )
[0096] (θ f ,θ d ) = arg max L total (θ f ,θ y ,θ d )
[0097] Step 5: Adjust and train the parameter configuration of MSFDA-Net to obtain the optimal results and complete distraction behavior detection.
[0098] The experiment of this invention is based on PyCharm 1.8.0 version under Windows 11 environment. The CPU is AMD Ryzen 76800H and the GPU is NVDIA GeForce RTX3090. The parameters of the model are configured as follows: the input image size is 224×224, the number of iterations is 100 rounds, the Batch_Size is 16, the bin number h of the posture feature is 12, the norm is L2, MultiStepLR is used for learning rate decay, the learning rate is 0.01, the learning rate momentum is 0.9, and the weight decay is 5×10 -3 , gamma=0.05, decay=0.75.
[0099] To verify the effectiveness of the distraction detection model, several representative models were selected for comparison with the same configuration and hyperparameters. These models have been benchmarked on the StateFarm and AUC datasets. The comparison results are shown in Table 1.
[0100] Table 1
[0101]
[0102]
[0103] In order to verify the generalization of the distraction behavior detection model, a cross-dataset experiment was conducted, using a mixture of StateFarm and AUC for cross-dataset training. One experiment used the training set of AUC and the test set of StateFarm, and the other experiment used the training set of StateFarm and the test set of AUC. The experimental results are detailed in Table 2.
[0104] Table 2
[0105]
[0106] It can be seen from Tables 1 and 2 that the detection accuracy and generalization ability of the proposed MSFDA-Net in cross-driver and cross-dataset scenarios are better than those of other tested models. The method proposed in this paper can effectively detect distracted driving behavior and complete the distracted behavior detection task with good generalization.
[0107] The above description does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiments shall still fall within the protection scope of the technical solution of the present invention.
Claims
1. A distraction behavior detection method based on multi-stream feature fusion and domain adaptation, characterized in that: The method comprises the following steps: S1 obtains driving data and preprocesses it to create a data set; S2 builds the MSFDA-Net distraction behavior detection model: S2.1 concatenates the pre-trained Openpose model and the pre-trained yolov5 model to perform posture estimation and target object detection on driving data, and calculate body joint features and semantic features representing human-object interactions; S2.2 uses multi-stream feature fusion network and domain adaptive network module to establish an end-to-end model; S2.2.1 build a multi-stream feature fusion network, process features using feature dimension reduction and feature abstraction, and input them into the domain adaptation network module; S2.2.2 The domain adaptation network module adopts a cross-domain feature learning strategy to enhance the model's understanding and generalization capabilities of high-level semantic information; S3 adjusts and trains the parameter configuration of the MSFDA-Net distraction behavior detection model to obtain the optimal results and complete distraction behavior detection.
2. The distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to claim 1, characterized in that: The S1 implementation process is as follows: The distraction behavior data are divided by driver and divided into training set and test set in a ratio of 8:
2.
3. The distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to claim 1, characterized in that: The S2.1 comprises the following steps: The pre-trained Openpose model is used to detect and locate body joints; the pre-trained yolov5s model is used to detect target objects that are frequently used or interacted with during driving. Paired relationship features are used to encode the configuration of a pair of joints. The configuration of body joints is modeled as high-level semantic features for recognition. There are N(N-1) / 2 pair combinations of N joints. For each pair, the relationship feature vector f is calculated by considering the x and y positions of the joints on the image plane. Given a pair of joints j1 and j2, and their respective positions (x1, y1) and (x2, y2), their relationship is represented by distance r and direction θ: The angle θ ranges from [-π,π] and is mapped to the interval [0,2π] by applying the modulus operator. Then, it is divided into h intervals, and the size r weights the corresponding interval where θ is located. The length of the feature vector f is the number of intervals. The relationship feature vector f between a pair of joints is calculated only when both joints are detected, otherwise, f will be set to zero; this process is performed for all (N(N-1) / 2) pairs, and the extracted paired feature vectors are merged into a single feature vector F = [f1,f2,…,f N(N-1) / 2 ], whose length is N(N-1) / 2×h; Histogram of directional relationship features representing the relationship between body joints and objects is extracted in a similar way to the posture feature f. For N body joints and O objects, there are N×O possible pairwise combinations. For each pair, the feature is calculated. and stacked into a single feature vector Its size is N×O×h.
4. The distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to claim 1, characterized in that: The S2.2.1 comprises the following steps: A lightweight multi-stream feature fusion network consisting of fully connected layers and dropout layers is constructed. A dropout layer is introduced between FC layers with a dropout rate set to 0.
5. The fully connected layer constitutes the core of the network and is responsible for learning complex nonlinear mappings in the input data. A sequence of FC layers with different capacities is designed, namely FC(2048), FC(1024) and FC(512), which gradually reduces the spatial dimension of the features and improves the abstraction level of the features.
5. The distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to claim 1 or 4, characterized in that: The cross-domain feature learning strategy described in S2.2.2 adopts an adversarial method to learn feature representation. The specific training steps are as follows: Assume that x∈X represents the input distraction behavior sample; After feature extractor E f (x;θ f )→f, each input sample x is mapped to a feature vector f in the feature space F; Through the class label predictor G y (x;θ y )→y, the feature vector f is mapped to the parameter θ y The class label y of Through the domain classifier G d (x;θ d )→d, with parameter θ d , map the above feature vector f to the domain label d.
6. The distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to claim 5, characterized in that: In adversarial methods, designing distraction detection class label predictors encourages the network to learn cross-domain feature representations; The loss function corresponding to the distraction detection class label predictor is defined as: Among them, L ce is the cross entropy loss function; G y is the class label predictor, θ y is the class label predictor parameter, E f is the feature extractor, x i is the input sample, θ f are the feature extractor parameters; y i is the actual value; The loss function for the domain label classifier is defined as: Among them, L ce is the cross entropy loss function; G d is the domain label classifier, θ d is the domain label classifier parameter; E f is the feature extractor, x i is the input sample, θ f are the feature extractor parameters; d i is the actual value; The cross entropy loss function formula is as follows: Among them, x i Represents the i-th element of the true label, y i Represents the probability that the model predicts that x belongs to the i-th category; The overall objective loss function is defined as: L total (i f ,the y ,the d )=L y (i f ,the y )-λL d (i f ,the d ) Where λ represents L y and L d The antagonistic balance parameters; In order to achieve classification accuracy and uniform alignment of source and target domains, the feature extractor parameters θ f and class label classifier parameters θ y Minimize the domain classifier parameter θ d maximize; (i f ,the y )=argminL total (i f ,the y ,i d ) (i f ,the d )=argmaxL total (i f ,the y ,the d ) 7. The distraction behavior detection method based on multi-stream feature fusion and domain adaptation according to claim 1, characterized in that: The detailed steps of S3 are as follows: The input image size is 224×224, the number of iterations is 100 rounds, the Batch_Size is 16, the bin number h of the pose feature is 12, the norm is L2, and MultiStepLR is used for learning rate decay. The learning rate is 0.01, the learning rate momentum is 0.9, the weight decay is 5×10-3, gamma=0.05, and decay=0.75.
Citation Information
Patent Citations
Driver behavior recognition method based on attitude estimation feature fusion
CN114241458A
Human body posture image intelligent identification method and system
CN115527269A
Distraction driving behavior identification method based on improved YoloV5 and Openpose
CN116403194A
Abnormal behavior recognition method based on space-time diagram convolutional neural network
CN116959099A
Attitude reconstruction interaction behavior understanding method based on skeleton and image features
CN117238026A
