Method and system for detecting train driver gestures and multitasking due to fatigue driving

Abstract

The application discloses a train driver gesture and fatigue driving multitask fusion detection method and system, and relates to the field of image processing. The method comprises the following steps: acquiring a target image; inputting the target image into a pre-trained multitask fusion detection model to obtain a gesture detection result and / or a fatigue state detection result output by the multitask fusion detection model; wherein the multitask fusion detection model is a model for jointly detecting gestures and fatigue states, which is obtained by training a preset neural network model by using a sample image dataset containing fatigue state labels and gesture labels, and can be used for jointly detecting fatigue driving and gestures.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a detection method and system for multi-task fusion of train driver gestures and fatigue driving. Background Technology

[0002] As the direct controllers of locomotive safety, train drivers are required to "use hand gestures, observe, and communicate verbally"—a clearly defined operational requirement in the railway driving field. However, in actual operation, some drivers simplify procedures and use non-standard hand gestures. Incorrect hand gestures can threaten train safety. Besides non-standard gestures, fatigue driving is also a significant hidden danger to train safety. Summary of the Invention

[0003] The purpose of this application is to provide a multi-task fusion detection method for train driver gestures and fatigue driving, which can jointly detect fatigue driving and gestures.

[0004] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for detecting train driver gestures and fatigue driving through a multi-task fusion approach, including: Acquire the target image; The target image is input into a pre-trained multi-task fusion detection model to obtain the gesture detection result and / or fatigue state detection result output by the multi-task fusion detection model. The multi-task fusion detection model is a model used for joint detection of gestures and fatigue states, obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations. The neural network model is trained as follows: image features of each sample image in the sample image dataset are extracted; fatigue state information and gesture information contained in the sample image are determined based on the image features; and the first sample image with gesture information in the sample image dataset is processed based on the mask branch to obtain the mask prediction result of the first sample image; the neural network model is trained based on the total loss generated by the fatigue state information, gesture information and the mask prediction result.

[0005] Secondly, this application provides a detection system that integrates train driver hand gestures and fatigue driving multi-task fusion, including: The acquisition module is used to acquire the target image; The detection module is used to input the target image into a pre-trained multi-task fusion detection model to obtain the gesture detection result and / or fatigue state detection result output by the multi-task fusion detection model. The multi-task fusion detection model is a model used for joint detection of gestures and fatigue states, obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations. The model building and training module is used to extract image features from each sample image in the sample image dataset, determine the fatigue state information and gesture information contained in the sample image based on the image features, process the first sample image with gesture information in the sample image dataset based on the mask branch, and obtain the mask prediction result of the first sample image; and train the neural network model based on the total loss generated by the fatigue state information, gesture information and the mask prediction result.

[0006] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the detection method for multi-task fusion of train driver gestures and fatigue driving as described above.

[0007] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the detection method for multi-task fusion of train driver gestures and fatigue driving as described above.

[0008] According to the specific embodiments provided in this application, the following technical effects are disclosed: According to the solution provided in the embodiments of the present invention, by training a multi-task fusion detection model, the model learns knowledge of gesture detection and fatigue state detection based on sample image data containing fatigue state annotations and gesture annotations. Thus, during the input of a target image, it can output possible gesture detection results and fatigue state detection results that conform to the learning process, thereby enabling joint detection of fatigue driving and gestures.

[0009] Furthermore, during the joint detection process, in addition to training the multi-task fusion detection model to output fatigue state information and gesture information, a mask branch is set up to specifically process the first sample image with gesture information. By performing mask prediction, a more accurate mask image region is obtained, and the loss value corresponding to the mask prediction result is added to the total loss. During model training, fatigue state data does not interfere with the data processing of the mask branch. In this way, the accuracy of gesture recognition can be improved, and the instance segmentation method through mask prediction can obtain more accurate gesture classification results, thereby improving the robustness and accuracy of multi-task detection of train driver gestures and fatigue driving.

[0010] In addition, regarding the feasibility of model deployment, the single-model multi-task integrated design avoids the separate deployment of two models. Instead, a single multi-task fusion detection model simultaneously completes fatigue detection and gesture segmentation, reducing model size and significantly lowering computing power consumption, thus adapting to the onboard computing environment of trains. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the first detection method for train driver hand gestures and fatigue driving fusion provided in this application; Figure 2 A flowchart illustrating the second method for detecting train driver hand gestures and fatigue driving fusion provided in this application; Figure 3 This is a schematic diagram of the structure of a multi-task detection system for train driver hand gestures and fatigue driving provided in this application.

[0013] Figure 4 This is a diagram illustrating an application environment according to one embodiment of this application.

[0014] Figure 5 This is a schematic diagram of a computer device structure according to one embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] In one embodiment of the present invention, see Figure 1 A detection method for train driver gestures and fatigue driving multi-task fusion is provided, including the following steps S101-S102.

[0018] S101: Acquire the target image.

[0019] S102: Input the target image into the pre-trained multi-task fusion detection model to obtain the gesture detection results and / or fatigue state detection results output by the multi-task fusion detection model.

[0020] The multi-task fusion detection model is a model used for joint detection of gestures and fatigue states, obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations. The neural network model is trained as follows: extract image features from each sample image in the sample image dataset, determine the fatigue state information and gesture information contained in the sample image based on the image features, and process the first sample image with gesture information in the sample image dataset based on the mask branch to obtain the mask prediction result of the first sample image; train the neural network model based on the total loss generated by the fatigue state information, gesture information and mask prediction result.

[0021] The aforementioned target images can be obtained by using cameras, mobile phones, or other imaging devices to capture images of the train driving environment.

[0022] The above detection results include at least one of the following information: if the output is a gesture detection result, the multi-task fusion detection model outputs the gesture category and can output the image region where the gesture was detected; if the output is a fatigue state detection result, the multi-task fusion detection model outputs the state category indicating whether a fatigue state exists and can output the image region where a fatigue state was detected.

[0023] To obtain such detection results, in the above embodiments, the fatigue state information in the annotation information used by the model during training may include: a state category indicating whether a fatigue state exists. For example, ID1, 2, and 3 are set in the annotation to represent three different degrees of fatigue state, which can specifically correspond to "awake", "fatigued", and "no state" respectively.

[0024] In the sample image data, sample images labeled with the above-mentioned state categories are designated as second sample images. Furthermore, the fatigue state information may also include bounding boxes of the detected state categories as regression information.

[0025] Gesture labels include gesture categories. To distinguish them from status categories, gesture categories can be assigned different ID values, such as 4, 5, and 6, to represent different categories of gestures. Specifically, they can correspond to the three categories of "Run", "Ready", and "Stop".

[0026] Gesture annotations can also include bounding boxes of detected gesture categories as regression information.

[0027] The set of second sample image data, i.e., the fatigue detection dataset, is defined as follows: ; in, Use the input RGB image as the second sample image. The image's height and width; The bounding box annotation set for regression information. For the first The target number of images; Label it as a fatigue category.

[0028] Furthermore, in order to train the mask branch for mask prediction, mask annotations are only added to the gesture annotations, specifically set as the true mask probability matrix, that is, the true mask region is annotated by a binarized matrix.

[0029] The dataset of the third sample images, i.e., the gesture segmentation dataset, is defined as: ; in, The mask is labeled at the pixel level, where 1 represents the gesture pixel and 0 represents the background. The data consists of gesture category labels. To achieve unified training for both tasks, the source dataset needs to be labeled and formatted: offset mapping is performed on the category labels of the gesture dataset. This makes the merged label space into a ,in Corresponding to fatigue category, The corresponding gesture class avoids cross-task tag ID conflicts. It is the bounding box annotation set of the gesture segmentation dataset. This indicates that the input RGB image is used as the third sample image.

[0030] The fused sample image dataset is represented as ,in This is a gesture segmentation dataset that has undergone label offsetting and format normalization. The dataset contains... Images, total number of annotations: This provides a unified data foundation for training multi-task models. The number of fatigue detection datasets, It is the number of gesture segmentation datasets. It represents the number of annotations for the i-th image in the fatigue detection dataset. It represents the number of annotations for the j-th image in the gesture segmentation dataset.

[0031] Specifically, the sample image dataset contains multiple batches of subsets, and in each subset, the number of second sample images containing fatigue state information and the number of third sample images containing gesture information are equal. To alleviate the problem of imbalanced sample distribution between the two types of tasks, the model adopts a balanced sampling strategy. Let the fatigue detection dataset be... The gesture segmentation dataset is The sampling probability satisfies Each batch of samples Alternate selection from two sets: in By setting the batch size, this strategy ensures that each batch contains an equal number of fatigue and gesture samples, effectively mitigating model bias caused by uneven sample distribution.

[0032] The default neural network model can be an improved Mask RCNN (Regions with CNN features) model.

[0033] The fatigue state information includes the state category output by the multi-task joint detection model and the corresponding regression box; the gesture information includes the gesture category output by the model and the corresponding regression box; and the mask prediction result is the pixel-by-pixel raw prediction score output by the mask branch, which is used to perform pixel-level prediction of the train driver's gesture area and characterize the confidence level of each pixel belonging to the gesture category.

[0034] By comparing the differences between the model's output and the labeled gesture and fatigue state information, the model parameters can be adjusted to ensure that the output of the multi-task fusion detection model is consistent with the label, thereby learning the recognition capabilities of gesture and fatigue state detection.

[0035] For details of the specific data processing procedure, please refer to the following examples.

[0036] According to the solution provided in the embodiments of the present invention, by training a multi-task fusion detection model, the model learns knowledge of gesture detection and fatigue state detection based on sample image data containing fatigue state annotations and gesture annotations. Thus, during the input of a target image, it can output possible gesture detection results and fatigue state detection results that conform to the learning process, thereby enabling joint detection of fatigue driving and gestures.

[0037] Furthermore, during the joint detection process, in addition to training the multi-task fusion detection model to output fatigue state information and gesture information, a mask branch is set up to specifically process the first sample image with gesture information. By performing mask prediction, a more accurate mask image region is obtained, and the loss value corresponding to the mask prediction result is added to the total loss. During model training, fatigue state data does not interfere with the data processing of the mask branch. In this way, the accuracy of gesture recognition can be improved, and the instance segmentation method through mask prediction can obtain more accurate gesture classification results, thereby improving the robustness and accuracy of multi-task detection of train driver gestures and fatigue driving.

[0038] In addition, regarding the feasibility of model deployment, the single-model multi-task integrated design avoids the separate deployment of two models. Instead, a single multi-task fusion detection model simultaneously completes fatigue detection and gesture segmentation, reducing model size and significantly lowering computing power consumption, thus adapting to the onboard computing environment of trains.

[0039] In one embodiment, image enhancement processing can be performed when acquiring image features to improve the quality of image feature extraction, as detailed in the following embodiments.

[0040] For each sample image, obtain the geometric feature map of that sample image; A feature pyramid network is used to enhance the geometric feature map, resulting in an enhanced feature map corresponding to the geometric feature map. The region proposal module generates candidate regions corresponding to the enhanced feature map, and extracts the regional features of the candidate regions as the image features of the sample image.

[0041] Specifically, the aforementioned neural network model may include the following modules: a ResNet50 module, used to extract features from the first sample image and obtain the image features of the sample image; a region proposal module, used to obtain candidate regions; and a ROI-align (Region of Interest Alignment) module, used to extract features from the candidate regions, obtain region features, and set a task-specific ROI header to obtain fatigue state information, gesture information, and mask prediction results.

[0042] In one embodiment, for each sample image, the geometric feature map of the sample image can be obtained in the following manner: The geometric feature map of the sample image is obtained using a ResNet50 network. The ResNet50 network has deformable convolutional layers, which obtain geometric feature maps containing irregular geometric features by adjusting the sampling points.

[0043] For a standard 3×3 convolution For each location mapped on the geometric feature map output by ResNet50 Equation (1) represents: ; in It corresponds to the convolution result of the output geometric feature map. These are the convolution kernel weights. These are the integer coordinate feature values ​​of the input feature map of the sample image input to ResNet50.

[0044] in, Enumerated The position within. In deformation convolution, the deformation offset is... ,in Equation (1) becomes equation (2): ; Thus far, the sampling has taken place at irregular and offset locations. Above. Due to Since the values ​​are usually decimals, equation (2) cannot directly calculate the eigenvalues ​​of the input feature map at non-integer coordinates. Equation (2) can be achieved through bilinear interpolation, see equation (3) below: ; in, This represents the eigenvalue at position P; This represents the arbitrary decimal coordinate position to be sampled in deformation convolution, corresponding to formula (2). , It enumerates all global spatial locations in the feature map. It is the kernel of bilinear interpolation. It is two-dimensional, and it is divided into two one-dimensional kernels: ; in, Equation (3) can be calculated quickly because Only for some It is non-zero. , Let x represent the x-components of q and p. , Let represent the y-components of q and p, respectively.

[0045] Gestures are irregular and varied in shape. Using the original 3×3 convolution sampling introduces too much complex background, making it difficult to extract image features. Therefore, deformable convolution is used to replace the ordinary 3×3 convolution in the ResNet50 feature extraction network. Deformable convolution is an extension of standard convolution. For planar graphics, deformable convolution adds the learned offset to the network sampling positions in the standard convolution, deforming the sampling positions of the standard convolution and concentrating them more in the Region of Interest (ROI). This allows the sampling points of the convolution kernel to change position according to the shape of the input image, forming an adaptive ability and exhibiting better geometric invariance.

[0046] Therefore, by replacing the original standard convolutional layers in the ResNet50 module with deformable convolutional layers, the sampling effect and feature representation ability can be improved.

[0047] In one embodiment, image enhancement of geometric feature maps is performed using a feature pyramid network, including: The geometric feature maps are processed through residual modules, and each downsampling layer in the feature pyramid network is connected to the output of each residual module to enhance the geometric feature maps of different scales extracted by each residual module.

[0048] Furthermore, during residual connection, the first downsampling layer in the feature pyramid network can be connected to the output of the last residual module of the ResNet50 module to enhance the smallest-scale image geometric feature map output by the ResNet50 module.

[0049] Its effect is as follows: 1. Greater semantic depth: The geometric feature map of driving behavior requires the "deepest level of abstraction".

[0050] The last residual module: After complete residual learning, it outputs the highest level semantic features, which have filtered out shallow textures and noise, and only retain high-level geometric structures (facial proportions, gesture skeletons, and pose relationships).

[0051] In contrast, other residual modules have lower semantic depth, still containing a large amount of mid-level textures and local details, and their ability to express the overall geometric pattern of driving behavior is weaker than that of the deepest layer.

[0052] 2. A wider range of driving scenarios: Driving scenarios need to cover the entire structure.

[0053] The first downsampling layer of FPN has the smallest scale and the largest receptive field in the feature pyramid network, and can completely cover the global spatial layout of the driver's face and gestures.

[0054] In contrast, other downsampling layers have a larger scale and a smaller receptive field, and can only capture local areas, making it impossible to model the overall geometric relationship of driving behavior.

[0055] 3. Higher purity of geometric features: The minimum scale is the target of geometric enhancement. Only the last residual module can output the minimum scale feature; only the first downsampling layer can be strictly aligned with its scale in the FPN.

[0056] The feature pyramid projects the extracted multi-scale volume geometric feature maps C2, C3, C4, and C5 and performs top-down fusion. The fusion method can be addition or splicing, so that the semantic features of the high-level layer and the detailed features of the low-level layer are preserved, resulting in the corresponding multi-level fused feature enhanced feature maps P2, P3, P4, and P5.

[0057] The Region Proposal Network (RPN) generates candidate bounding boxes (proposals) at each pyramid level in the feature pyramid network, which are the candidate regions corresponding to each augmentation map. Thus, the task-specific Region of Interest (ROI) head is responsible for mapping these proposals to the appropriate feature layers and performing classification, regression, and mask prediction.

[0058] The selection of the hierarchy from FPN (Feature Pyramid Networks) to ROI typically follows a canonical scale mapping rule: if the area of ​​the proposal is... Then it is mapped to the corresponding level. Approximately according to the formula ; After mapping to different levels, enhanced feature maps P2, P3, P4, and P5 are formed. Here... The baseline layer index is obtained by setting hyperparameters. This mapping ensures that proposals at different scales are pooled features from scale-adapted pooling layers.

[0059] The task-specific region of interest header includes a detection branch and a mask branch. The detection branch is used for classification and regression to obtain fatigue state information and gesture information. The classification process obtains category information, and the regression process obtains bounding boxes. The mask branch is used for instance segmentation.

[0060] The detection branch uses the ROIAlign operation to crop and align each candidate region from the FPN feature map into a fixed 7×7 feature map. Then, it performs feature transformation through two fully connected network layers (1024 dimensions per layer), and finally outputs the classification scores of gesture category and state category, as well as the regression offsets of gesture category and state category, respectively.

[0061] The masked branch is handled as follows: The image features of the first sample image are pooled to output the pixel prediction tensor of the first sample image as the mask prediction result; wherein, the pixel prediction tensor is used to represent the probability information of the first sample image belonging to each gesture category.

[0062] Specifically, the masking branch processes the region features, consisting of 6 layers of 512-channel convolutions (CONV_DIM=512, NUM_CONV=6). This is followed by upsampling (deconv or transpose conv) to amplify the output, which is then pooled using 1×1 convolutions to generate a preset number of channels (num_classes) for the region features—this is the pixel prediction tensor. The number of channels matches the total number of classes, with each channel corresponding to a target class. The probability information is a numerical value at each spatial location representing the unnormalized confidence that the pixel belongs to the corresponding class.

[0063] The convolutional feature transformation process of the masked branch is described as follows: The first 3×3 convolution transforms the input features... =256 Projected to intermediate features =512, the subsequent five 3×3 layers maintain the 512-channel mapping, and finally Deconv maps the 512 channels to 256 channels and upsamples by 2 times. The final 1×1 convolutional layer outputs num_classes channels to obtain the pooling result. The number of parameters in a single convolutional layer is k represents the level.

[0064] The input features are those obtained by the pooling layers in the mask branch during the feature extraction stage. Specifically, the corresponding regions of the enhanced feature maps P2, P3, P4, and P5 are pooled into feature maps of a fixed size. During this process, the pooling resolution is set to 16, resulting in an initial spatial resolution of 16 for each candidate region. .

[0065] Furthermore, strong constraints are imposed on the pooling layer to achieve the transformation from general instance segmentation to task-driven fine segmentation, that is, the mask branch is controlled to process only the first sample image data with gesture information.

[0066] Specifically, in the multi-task fusion scenario of train driver fatigue detection and gesture segmentation, this refers to the application of hard, exclusive, and task-specific structural constraints on the feature input source of the mask branch, the structure of the pooling layer in the mask branch, and the calculation range of the mask loss. The aim is to transform general, open instance segmentation into task-driven segmentation specifically for fine-grained train driver gesture segmentation, preventing interference from irrelevant tasks (i.e., fatigue state detection) on the mask branch from both structural and computational perspectives. This is achieved through a three-layer constraint mechanism: 1. Strong constraint on pooling layer input features: specifying that the input can only be features from the four layers P2, P3, P4, and P5 of the FPN; 2. Strong constraint on the mask calculation range (task isolation): distinguishing by labeled category IDs, only calculating the segmentation loss for gesture-type instances (IDs 4, 5, and 6), as detailed in the following example; fatigue data (IDs 1, 2, and 3) are completely filtered out, never entering the mask branch and not participating in any mask-related calculations. 3: Parameter stability constraints when there are no gesture samples. If there are no gesture instances in the current batch, L2 regularization terms are forcibly applied to constrain the branch parameters to prevent parameter degradation and branch failure.

[0067] During forward propagation, the model extracts features P2–P5 sequentially from the FPN output and feeds them into the pooling layer. The pooling layer uses the ROIAlignV2 operation to crop and resample each candidate region into a fixed 16×16 feature map.

[0068] During the training phase, the pixel prediction tensor is used as the input to the mask cross-entropy loss to supervise pixel-level prediction of the gesture region. During the image detection phase of acquiring the target image, the pixel prediction tensor is normalized to a probability map by sigmoid and then used to generate a binary segmentation mask for the train driver's gesture by thresholding, thus achieving accurate segmentation of the gesture instance.

[0069] Specifically, during the training phase, after completing 6 layers of convolutional feature transformation, a deconvolutional layer upsamples the feature map by a factor of 2 to recover higher resolution spatial information. Then, a 1×1 convolution is used to generate the final pixel prediction tensor. If the number of classes is (K), the output tensor dimension of the mask branch is , where (N) is the number of candidate instances, and (H) and (W) are the spatial resolutions. During training, the model selects the mask prediction result from the corresponding channel based on the true class index of each instance and performs pixel-by-pixel supervision with the true mask.

[0070] In one embodiment, the total loss is generated as follows: ; in, Indicates the total loss; Indicates detection loss, Indicates the partition loss. This represents the preset weighting coefficients used to balance the segmentation loss and detection loss; The segmentation loss is obtained based on the cross-entropy loss generated from the mask prediction results, while the detection loss is obtained based on the classification loss of fatigue state information and the first bounding box regression loss, the classification loss of gesture information and the second bounding box regression loss. The detection loss covers the entire task, while the segmentation loss only constrains the gesture class, thereby achieving task decoupling and gradient balancing to avoid mutual interference between tasks.

[0071] Detection loss Determined in the following manner: ; in, Indicates the number of candidate regions. Represents the cross-entropy classification loss. , This represents the predicted class probability of the r-th candidate region, for each of the gesture and state classes. Output the probability of being classified into different ID categories; This represents the true label of the r-th candidate region. The true label may include the state category of fatigue state annotation and / or the gesture category of gesture annotation. This represents the preset second weighting coefficient, used to balance the bounding box loss and cross-entropy loss; This represents the first bounding box loss and the second bounding box loss. This represents the offset of the predicted bounding box output by the neural network model. This represents the ground truth bounding box offset; the predicted bounding box offset is the offset between the model-predicted r-th candidate region and the labeled bounding box; the ground truth bounding box offset is the actual offset between the r-th candidate region and the labeled bounding box. Specifically, , It is a smoothed L1 loss function.

[0072] Segmentation loss Determined in the following manner: ; in, This represents the number of gesture class candidate regions contained in the first sample image. Indicates the number of candidate regions. This represents a set of candidate regions for gestures. The mask cross-entropy loss represents the result of the mask prediction. This represents the prediction mask probability matrix. Represents the true mask probability matrix. The learnable parameters of the mask branch, L2 regularization terms. This is used to prevent segmentation branch parameter degradation and gradient vanishing when there is no first sample image. The prediction mask probability matrix is ​​a binarized matrix of the representation mask predicted by the model.

[0073] The gesture category candidate region refers to the candidate region in the first sample image that is labeled with the gesture category.

[0074] Masked cross-entropy loss The specific calculation method is as follows: ; in, This represents the height of the mask feature map corresponding to a single gesture ROI (Region of Interest). This represents the width of the mask feature map corresponding to a single gesture ROI. This represents the value in the i-th row and j-th column of the true mask probability matrix. This represents the value in the i-th row and j-th column of the prediction mask probability matrix.

[0075] Based on the above, regarding the multi-task fusion, a task-aware composite loss function is designed. The detection loss covers all tasks, while the segmentation loss only constrains the gesture class, achieving task decoupling and gradient balancing to avoid mutual interference between tasks. A balanced sampling strategy is adopted.

[0076] In terms of scenario robustness, strong task constraints and balanced sampling are used to eliminate the interference of invalid samples and invalid losses on the model, resulting in more stable training and stronger generalization ability. Using dual datasets for training covers more driving postures, scenarios, and lighting conditions, improving data diversity and scenario adaptability.

[0077] The following example illustrates a feasible experimental environment for implementing the solution provided in this application.

[0078] The configuration used in the experiment included: Linux operating system, deep learning framework: pyTorch, programming tools: Python 3.8, and CUDA 11.4. This paper uses and configures the environment as Linux + PyTorch 1.12.1 + CUDA 11.4, and uses a V100-SXM2-32GB graphics card to complete all experiments.

[0079] Evaluation indicators: Four commonly used evaluation metrics were used to evaluate the model's performance: accuracy (P), recall (R), and F1 score. Based on the true class and model prediction of the test samples in the dataset, samples were categorized into true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN).

[0080] Through experiments, the performance advantages of the model provided in this application and the original models in related technologies can be compared in various indicators, thereby demonstrating that the above-mentioned technical solution of this application significantly enhances fatigue detection performance while maintaining gesture segmentation capabilities, and achieves the best balance of performance, stability and practicality in multi-task scenarios.

[0081] The following is passed Figure 2 The embodiments shown illustrate the overall process of the present invention.

[0082] Step 1: Replace the standard convolutional layers of the ResNet50 module in the original Mask RCNN network with deformable convolutional layers; add a feature pyramid network to the output of the ResNet50 module to construct an improved ResNet50 module; design TaskSpecificMaskROIHeads (task-specific region of interest heads), and achieve the transformation from general instance segmentation to task-driven fine segmentation by reconfiguring the mask branch pooler (pooling layer) and using strong input constraints to the pooling layer; design a composite loss function to calculate the total loss, effectively avoiding the problems of mask branch parameter degradation and gradient vanishing; ensure the stability of multi-task training through balanced sampling; finally, construct a multi-task fusion detection model for train driver gestures and fatigue driving, which includes the improved ResNet50 module, the original region proposal module and ROI Align module, and TaskSpecificMaskROIHeads connected in sequence. Step 2: Collect images of train drivers' hand gestures and fatigue driving to construct a sample image dataset. Use the sample image dataset to train the improved network and obtain a multi-task fusion detection model. Step 3: Input the train driver's gestures and fatigue driving images (i.e., target images) into the multi-task fusion detection model. By improving the deformable convolutional layer in the ResNet50 module, the sampling points of the convolutional kernel are adaptively adjusted to capture the irregular geometric features of the image, obtaining a geometric feature map. By improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the ResNet50 module, the geometric feature map is enhanced at different scales to obtain multi-scale fatigue and gesture enhanced feature maps. The region proposal module detects the enhanced feature map and generates several candidate regions for gestures and fatigue postures. The ROI-Align module extracts the feature vectors corresponding to each candidate region from the multi-scale driving behavior enhanced feature map. Through the TaskSpecificMaskROIHeads structure, task-driven fine segmentation is achieved by relying on the reconfigured mask branch pooler. Combined with the composite loss function and balanced sampling strategy, the problems of mask branch parameter degradation and gradient vanishing are avoided, and the stability of multi-task training is ensured. Finally, the detection results containing driver gesture category, fatigue state and corresponding bounding box, and segmentation mask are output.

[0083] Corresponding to the above method embodiments, in one embodiment of the present invention, a detection system for multi-task fusion of train driver gestures and fatigue driving is also provided. The detection system for multi-task fusion of train driver gestures and fatigue driving includes: The acquisition module is used to acquire the target image; The detection module is used to input the target image into a pre-trained multi-task fusion detection model to obtain the gesture detection result and / or fatigue state detection result output by the multi-task fusion detection model. The multi-task fusion detection model is a model used for joint detection of gestures and fatigue states, obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations. The model building and training module is used to extract image features from each sample image in the sample image dataset, determine the fatigue state information and gesture information contained in the sample image based on the image features, process the first sample image with gesture information in the sample image dataset based on the mask branch, and obtain the mask prediction result of the first sample image; and train the neural network model based on the total loss generated by the fatigue state information, gesture information and the mask prediction result.

[0084] The embodiments of this application also provide a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the detection method for multi-task fusion of train driver gestures and fatigue driving as described in any of the above embodiments.

[0085] The embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the detection method for multi-task fusion of train driver gestures and fatigue driving as described in any of the above embodiments.

[0086] The embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of detecting train driver gestures and fatigue driving multi-task fusion as described in any of the above embodiments.

[0087] The detection method for train driver gestures and fatigue driving multi-task fusion provided in any of the above embodiments of this application can be applied to, for example... Figure 4The application environment shown is illustrated. Terminal 401 communicates with server 402 via a network. A data storage system can store the data that server 402 needs to process. The data storage system can be set up independently, integrated into server 402, or placed in the cloud or on other servers. Terminal 401 can send a target image to server 402, and server 402 can acquire the target image; input the target image into a pre-trained multi-task fusion detection model to obtain the gesture detection result and / or fatigue state detection result output by the multi-task fusion detection model; wherein, the multi-task fusion detection model is: a model for joint detection of gestures and fatigue states obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations; the neural network model is trained in the following manner: extracting image features of each sample image in the sample image dataset, determining the fatigue state information and gesture information contained in the sample image based on the image features, and processing the first sample image with gesture information in the sample image dataset based on the mask branch to obtain the mask prediction result of the first sample image; training the neural network model based on the total loss generated by the fatigue state information, gesture information and the mask prediction result.

[0088] Among them, terminal 401 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 402 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0089] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the multi-task fusion detection method for train driver gestures and fatigue driving according to any of the above embodiments.

[0090] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0091] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0092] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0093] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0094] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting train driver hand gestures and fatigue driving through multi-task fusion, characterized in that, The detection method for train driver gestures and fatigue driving multi-task fusion includes: Acquire the target image; The target image is input into a pre-trained multi-task fusion detection model to obtain the gesture detection result and / or fatigue state detection result output by the multi-task fusion detection model. The multi-task fusion detection model is a model used for joint detection of gestures and fatigue states, obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations. The neural network model is trained as follows: image features of each sample image in the sample image dataset are extracted; fatigue state information and gesture information contained in the sample image are determined based on the image features; and the first sample image with gesture information in the sample image dataset is processed based on the mask branch to obtain the mask prediction result of the first sample image; the neural network model is trained based on the total loss generated by the fatigue state information, gesture information and the mask prediction result.

2. The detection method for train driver gestures and fatigue driving multi-task fusion according to claim 1, characterized in that, The extraction of image features from each sample image in the sample image dataset includes: For each sample image, obtain the geometric feature map of that sample image; The geometric feature map is enhanced using a feature pyramid network to obtain the enhanced feature map corresponding to the geometric feature map. The region proposal module generates candidate regions corresponding to the enhanced feature map, and the regional features of the candidate regions are extracted as image features of the sample image.

3. The detection method for train driver gestures and fatigue driving multi-task fusion according to claim 2, characterized in that, The process of obtaining the geometric feature map of the sample image includes: The geometric feature map of the sample image is obtained using a ResNet50 network, wherein the ResNet50 network has deformable convolutional layers, and the deformable convolutional layers obtain geometric feature maps containing irregular geometric features by adjusting the sampling points.

4. The detection method for train driver gestures and fatigue driving multi-task fusion according to claim 2, characterized in that, The image enhancement of the geometric feature map using a feature pyramid network includes: The geometric feature maps are processed by the residual module, and each downsampling layer in the feature pyramid network is connected to the output of each residual module to enhance the image geometric feature maps of different scales extracted by each residual module.

5. The detection method for train driver hand gestures and fatigue driving multi-task fusion according to claim 2, characterized in that, The total loss is generated in the following manner: ; in, This represents the total loss; Indicates detection loss, Indicates the partition loss. This indicates the preset weighting coefficients; The segmentation loss is obtained based on the cross-entropy loss generated from the mask prediction result, and the detection loss is obtained based on the classification loss and first bounding box regression loss of the fatigue state information, the classification loss and second bounding box regression loss of the gesture information. The detection loss Determined in the following manner: ; in, This indicates the number of candidate regions. express Cross-entropy classification loss, This represents the predicted class probability of the r-th candidate region. This represents the true label of the r-th candidate region. This represents the preset second weighting coefficient; This represents the first bounding box loss and the second bounding box loss. This represents the offset of the predicted bounding box output by the neural network model. Indicates the offset of the true bounding box; The segmentation loss Determined in the following manner: ; in, This represents the number of gesture class candidate regions contained in the first sample image. This indicates the number of candidate regions. This represents the set of candidate regions for the gesture class. The mask cross-entropy loss represents the mask prediction result. This represents the probability matrix of the mask. Represents the true mask probability matrix. This represents the learnable parameters of the mask branch.

6. The detection method for train driver gestures and fatigue driving multi-task fusion according to claim 1, characterized in that, The step of obtaining the mask prediction result of the first sample image includes: The image features of the first sample image are pooled to output the pixel prediction tensor of the first sample image as the mask prediction result; wherein, the pixel prediction tensor is used to represent the probability information of the first sample image belonging to each gesture category.

7. The detection method for train driver hand gestures and fatigue driving multi-task fusion according to claim 1, characterized in that, The sample image dataset contains multiple batches of sub-datasets, and in each sub-dataset, the number of second sample images with fatigue state information and the number of third sample images with gesture information are equal.

8. A detection system integrating train driver hand gestures and fatigue driving multi-task, characterized in that, The train driver hand gesture and fatigue driving multi-task fusion detection system includes: The acquisition module is used to acquire the target image; The detection module is used to input the target image into a pre-trained multi-task fusion detection model to obtain the gesture detection result and / or fatigue state detection result output by the multi-task fusion detection model. The multi-task fusion detection model is a model used for joint detection of gestures and fatigue states, obtained by training a preset neural network model with a sample image dataset containing fatigue state annotations and gesture annotations. The model building and training module is used to extract image features from each sample image in the sample image dataset, determine the fatigue state information and gesture information contained in the sample image based on the image features, process the first sample image with gesture information in the sample image dataset based on the mask branch, and obtain the mask prediction result of the first sample image; and train the neural network model based on the total loss generated by the fatigue state information, gesture information and the mask prediction result.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the detection method for multi-task fusion of train driver gestures and fatigue driving as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the detection method for multi-task fusion of train driver gestures and fatigue driving as described in any one of claims 1-7.

Images