An edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series

Abstract

This invention relates to the field of computer vision detection technology and provides an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series. The method includes the following steps: acquiring portrait image frame data; analyzing the portrait image frame data according to a position estimation model to obtain the image to be detected; storing the image to be detected according to a dynamic candidate sequence; processing the image to be detected according to a fatigue distraction detection model to obtain the detection result; and uploading the detection result to the cloud. This invention, based on an edge-cloud collaborative method, enables real-time detection of fatigue driving behavior on low-computing-power edge devices. It uses a multi-task model to reduce the consumption of computing resources and adopts an edge-cloud collaborative computing framework to reduce the false detection rate of edge-side recognition results. Ultimately, it can accurately detect fatigue driving behavior in real time even with limited computing power.

Description

Technical Field

[0001] This invention relates to the field of computer vision detection technology, specifically to an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series. Background Technology

[0002] Real-time monitoring and early warning of driver fatigue are crucial for improving road traffic safety. Traditional fatigue detection methods primarily rely on analyzing the driver's physiological signals, such as electroencephalograms (EEGs) and heart rate. However, these methods require physical contact with the driver, making implementation conditions quite demanding. In recent years, computer vision-based fatigue detection technology has developed rapidly. These methods mainly determine driver fatigue by analyzing facial features, such as the state of the eyes and mouth. However, these methods often depend on high-performance computing equipment for real-time analysis, which not only consumes significant computational resources but also carries the risk of high false alarm rates.

[0003] With the rise of cloud computing and edge computing, edge-cloud collaborative computing has received increasing attention. Edge-cloud collaborative computing aims to rationally distribute computing tasks between the cloud and edge devices, thereby fully utilizing the computing power of both to achieve efficient computing results. However, in practical applications, combining edge-cloud collaboration with fatigue driving detection while ensuring real-time performance and accuracy remains a challenging problem.

[0004] Furthermore, existing edge-cloud collaborative fatigue driving detection methods often require multiple models to work together, which not only increases computational resource consumption but may also reduce model inference speed. Therefore, how to reduce computational resource consumption and improve model inference speed while ensuring detection accuracy is also a challenge currently facing the technology.

[0005] In summary, developing a method that can perform real-time fatigue driving detection on low-computing-power edge devices while reducing false alarms and improving detection accuracy is a current research hotspot and challenge. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series, so as to solve the problems existing in the above-mentioned background technology.

[0007] This invention is implemented as follows: a cloud-edge collaborative fatigue driving detection method based on dynamic candidate time series, the method comprising the following steps:

[0008] Acquire human portrait image frame data;

[0009] The image to be detected is obtained by analyzing the human image frame data based on the position estimation model, so that the driver's position can be determined and the driver's head image can be cropped out.

[0010] The images to be detected are stored according to a dynamic candidate sequence;

[0011] The detection result is obtained by processing the image to be detected according to the fatigue distraction detection model. The detection result is the behavior of driver fatigue driving. Both the position estimation model and the fatigue distraction detection model are trained by convolutional neural networks.

[0012] The detection results are uploaded to the cloud, allowing the cloud to calibrate the results based on a high-precision model. Abnormal results are removed, and the fatigue distraction detection model is updated iteratively.

[0013] As a further aspect of the present invention: the step of analyzing portrait image frame data based on a position estimation model to obtain the image to be detected specifically includes:

[0014] The portrait image frame data is processed according to the position estimation model, and the decoding output of the position estimation model is the bounding box regression parameters and the depth position regression parameters.

[0015] The image to be detected about the driver is obtained based on the boundary regression parameters and depth regression parameters.

[0016] As a further aspect of the present invention: the convolutional neural network includes a backbone network, an enhanced feature extraction network, and a depth detection head. The enhanced feature extraction network is used to sample features in the backbone network at multiple different scales, and the depth detection head is used to distinguish between people in the front and rear seats of the vehicle.

[0017] As a further aspect of the present invention: the fatigue distraction detection model is used to detect the driver's behaviors of opening and closing eyes, opening and closing mouth, facing the front and turning the head. The convolutional neural network includes a target detection head and a posture estimation head. The target detection head is used to detect the features of the driver's eyes and mouth, and the posture estimation head is used to detect the driver's head rotation angle.

[0018] As a further aspect of the present invention: the step of storing the image to be detected according to a dynamic candidate sequence specifically includes:

[0019] The image to be detected is tiled according to a dynamic candidate sequence, wherein the image to be detected consists of multiple image frames;

[0020] Reverse detection of image frames is performed based on the dynamic candidate sequence;

[0021] When no abnormal behavior is observed in any of the image frames, a video is generated based on the image frames in the dynamic candidate sequence and uploaded to the cloud.

[0022] Delete the image frames in the dynamic candidate sequence.

[0023] As a further aspect of the present invention: the step of performing reverse detection of image frames based on the dynamic candidate sequence further includes:

[0024] When an image frame exhibits non-abnormal behavior, delete the image frame and all preceding image frames.

[0025] Based on the image to be detected, the missing image frames in the dynamic candidate sequence are supplemented and reversed detection is performed, so that all image frames in the dynamic candidate sequence are abnormal behaviors.

[0026] As a further aspect of the present invention, the steps of storing the image to be detected according to the dynamic candidate sequence and processing the image to be detected according to the fatigue distraction detection model to obtain the detection result are performed asynchronously in a multi-threaded manner.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] This invention acquires human image frame data through a camera, and the edge device can process the human image frame data to extract the head information of the driver in the car in the image. The edge device processes the data to obtain detection results, which can determine some fatigue driving behaviors of the driver. Moreover, the detection results can be uploaded to the cloud, and the powerful computing capabilities of the cloud can be used to verify the detection results. In summary, this invention uses an edge-cloud collaborative method. The edge device uses a lightweight model to detect fatigue driving behavior and uploads the detection results to the cloud. The cloud device uses a high-precision model to re-identify the original video to remove false alarms and improve accuracy. The cooperation between edge and cloud devices can greatly improve the effect. Attached Figure Description

[0029] Figure 1 This is an application environment diagram for an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series.

[0030] Figure 2 This is a flowchart of an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series.

[0031] Figure 3 This is a flowchart illustrating how a cloud-edge collaborative fatigue driving detection method based on dynamic candidate time series analyzes human image frame data using a position estimation model to obtain the image to be detected.

[0032] Figure 4 This is a flowchart illustrating how images to be detected are stored according to dynamic candidate sequences in an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0035] Figure 1 This is an application environment diagram for an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series, such as... Figure 1 As shown, this application environment includes end-side devices and cloud servers.

[0036] End-side equipment includes camera equipment for collecting information and some processing equipment that goes with the camera equipment. Of course, for fatigue driving, it is generally on highways, so the camera equipment can be highway speed measuring cameras, photo cameras, and ETC shooting equipment, etc.

[0037] A cloud server can be a standalone physical server or terminal, or a server cluster consisting of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud storage, and CDN.

[0038] The edge devices are mainly used to collect information and verify the information according to the fatigue distraction detection model, while the cloud server is used to verify the verification results of the edge devices and collect the information uploaded by the edge devices to update the original fatigue distraction detection model of the edge devices.

[0039] like Figure 2 As shown in the figure, this embodiment of the invention provides an edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series, the method comprising the following steps:

[0040] S100, acquire portrait image frame data;

[0041] S200: The image to be detected is obtained by analyzing the human image frame data according to the position estimation model, so that the driver's position is determined and the driver's head image is cropped out.

[0042] S300, stores the image to be detected according to the dynamic candidate sequence;

[0043] S400, the image to be detected is processed according to the fatigue distraction detection model to obtain the detection result, the detection result being the driver's fatigued driving behavior, and both the position estimation model and the fatigue distraction detection model are trained by a convolutional neural network;

[0044] The S500 uploads the detection results to the cloud, allowing the cloud to calibrate the results based on a high-precision model. Abnormal results are removed, and the fatigue distraction detection model is updated iteratively.

[0045] It should be noted that the human image frame data includes all the people in the vehicles on the road. Therefore, within a unit of time, information on multiple vehicles and drivers may be recorded. Thus, the images to be detected are diverse. They can be classified according to facial recognition technology, or they can be reclassified based on vehicle information for identity recognition. Steps S100 to S400 are all run on low-computing-power edge devices, while step S500 is performed on cloud testing with sufficient computing power. By inferring the video stream through low-computing-power edge devices, the real-time nature of the results is ensured, and the results are reviewed through the high-precision model of cloud testing to reduce false alarms and ensure the accuracy of the results.

[0046] In this embodiment of the invention, a camera is used to acquire human image frame data. The edge device can process the human image frame data, extract the head information of the driver in the car in the image, and obtain detection results through processing. It can determine some fatigue driving behaviors of the driver and upload the detection results to the cloud. The powerful computing capabilities of the cloud are used to verify the detection results. In summary, this invention uses an edge-cloud collaborative method. The edge device uses a lightweight model to detect fatigue driving behavior and uploads the detection results to the cloud. The cloud device uses a high-precision model to re-identify the original video to remove false alarms and improve accuracy. The cooperation between edge and cloud devices can greatly improve the effect.

[0047] like Figure 3 As shown, in a preferred embodiment of the present invention, the step of analyzing portrait image frame data according to the position estimation model to obtain the image to be detected specifically includes:

[0048] S201, The portrait image frame data is processed according to the position estimation model, and the decoding output of the position estimation model is the bounding box regression parameters and the depth position regression parameters;

[0049] S202, obtain the image to be detected about the driver based on the boundary regression parameters and depth regression parameters.

[0050] In this embodiment of the invention, the convolutional neural network includes a backbone network, an enhanced feature extraction network, and a depth detection head. The enhanced feature extraction network is used to sample features in the backbone network at various scales. The depth detection head is used to distinguish between people in the front and rear seats of the vehicle. The fatigue distraction detection model is used to detect the driver's behaviors of opening and closing eyes, opening and closing mouth, facing the front, and turning the head. The convolutional neural network includes a target detection head and a pose estimation head. The target detection head is used to detect the features of the driver's eyes and mouth. The pose estimation head is used to detect the driver's head rotation angle. Video data is collected by a camera and manually labeled to form a fatigue driving dataset. The convolutional neural network is trained using the dataset to obtain a position estimation model and a fatigue distraction detection model. For the position estimation model, the dataset labeling requires category information, four bounding box information, and depth information within [0-1], specifically two categories: people and mobile phones. For the fatigue distraction detection model, the final detection categories are divided into opening eyes, closing eyes, opening mouth, closing mouth, facing the front, and turning the head.

[0051] The location estimation model consists of a backbone network, an enhanced feature extraction network, and a depth detection head. The enhanced feature extraction network samples features from the backbone network at three different scales, resulting in enhanced features at three scales, thus improving the accuracy of the location estimation model. The depth detection head adds a depth information regression branch to the classification regression branch and bounding box regression branch to distinguish between front and rear passengers in the vehicle, avoiding misidentification. Specifically, the output of the location estimation model can ultimately be decoded into bounding box regression parameters and depth location regression parameters, where the bounding box regression parameters are obtained from the bounding box regression branch, and the depth location regression parameters are obtained from the depth information regression branch.

[0052] The aforementioned fatigue distraction detection model includes a backbone network, an enhanced feature extraction network, a target detection head, and a pose estimation head. The backbone network and the enhanced feature extraction network are the same as those in the aforementioned position estimation model. Specifically, the target detection head outputs mouth and eye detection boxes, and the output of the pose estimation head can be decoded into a decimal within [0,1], representing the head rotation angle. In summary, a lightweight network structure is used, a dynamic weight reparameter module is used to improve the accuracy of the deployed model, and a multi-task regression branch is used to improve the model inference speed.

[0053] like Figure 4 As shown in the preferred embodiment of the present invention, the step of storing the image to be detected according to the dynamic candidate sequence specifically includes:

[0054] S301, the image to be detected is tiled according to a dynamic candidate sequence, wherein the image to be detected consists of multiple image frames;

[0055] S302, sequentially detect image frames according to the dynamic candidate sequence;

[0056] S303, when no abnormal behavior is observed in any image frames, a video is generated based on the image frames in the dynamic candidate sequence and uploaded to the cloud;

[0057] S3031, when an image frame exhibits non-abnormal behavior, delete the image frame and all previous image frames;

[0058] S3032, based on the image to be detected, the missing image frames in the dynamic candidate sequence are supplemented and reverse detection is performed, so that all image frames in the dynamic candidate sequence are abnormal behaviors, and a video is generated and uploaded to the cloud;

[0059] S304, delete the image frames in the dynamic candidate sequence.

[0060] In this embodiment of the invention, the initial length of the dynamic candidate sequence is set to k, and the image frame detection method is a sliding window recognition method with a sliding window size of n. Images to be detected are extracted from the dynamic candidate sequence and fed into the fatigue distraction detection model for processing. First, the first k image frames are extracted from the real-time video stream to fill the dynamic candidate sequence, and each frame is detected in reverse order to check for abnormal states until the first image frame without abnormal behavior is detected. Then, the frame and all previous frames in the sequence are deleted, and frames are extracted from the video stream to fill the sequence again. Next, the above two steps are repeated until the dynamic candidate sequence is full and consists entirely of abnormal frames. At this point, the category of abnormal behavior in the sequence is determined. Finally, the abnormal behavior frames in the dynamic candidate sequence are encapsulated into a video and uploaded to the cloud for verification. The cloud saves the verification results and the abnormal video, expanding the dataset for model iteration and updates. A lightweight network structure is used, utilizing a dynamic weight reparameter module to improve the accuracy of the deployed model, and a multi-task regression branch to improve the model inference speed. All processing steps and the video push-pull stream are executed in two threads, and all detection and verification results can be viewed in the cloud.

[0061] The above description only details the preferred embodiments of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0062] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0063] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0064] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the disclosure in the specification and embodiments. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A cloud-edge collaborative fatigue driving detection method based on dynamic candidate time series, characterized in that, The method includes the following steps: Acquire human portrait image frame data; The image to be detected is obtained by analyzing the human image frame data based on the position estimation model, so that the driver's position can be determined and the driver's head image can be cropped out. The process of storing the images to be detected according to a dynamic candidate sequence includes: tiling the images to be detected according to the dynamic candidate sequence, wherein the images to be detected consist of multiple image frames; performing reverse detection on the image frames according to the dynamic candidate sequence; when no abnormal behavior is found in any of the image frames, generating a video based on the image frames in the dynamic candidate sequence and uploading it to the cloud; and deleting the image frames in the dynamic candidate sequence. The fatigue distraction detection model processes the image to be detected to obtain the detection result, which is the behavior of a driver driving while fatigued. Both the position estimation model and the fatigue distraction detection model are trained by a convolutional neural network. The convolutional neural network includes a backbone network, an enhanced feature extraction network, and a depth detection head. The enhanced feature extraction network is used to sample the features in the backbone network at multiple different scales. The depth detection head is used to distinguish between the people in the front and rear seats of the vehicle. The fatigue distraction detection model is used to detect the driver's behaviors of opening and closing eyes, opening and closing mouth, facing the front, and turning the head. The convolutional neural network includes a target detection head and a pose estimation head. The target detection head is used to detect the features of the driver's eyes and mouth, and the pose estimation head is used to detect the rotation angle of the driver's head. The detection results are uploaded to the cloud, allowing the cloud to calibrate the results based on a high-precision model. Abnormal results are removed, and the fatigue distraction detection model is updated iteratively.

2. The edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series according to claim 1, characterized in that, The step of analyzing portrait image frame data based on the position estimation model to obtain the image to be detected specifically includes: The portrait image frame data is processed according to the position estimation model, and the decoding output of the position estimation model is the bounding box regression parameters and the depth position regression parameters. The image to be detected about the driver is obtained based on the bounding box regression parameters and depth regression parameters.

3. The edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series according to claim 1, characterized in that, The step of performing reverse detection of image frames based on the dynamic candidate sequence further includes: When an image frame exhibits non-abnormal behavior, delete the image frame and all preceding image frames. Based on the image to be detected, the missing image frames in the dynamic candidate sequence are supplemented and reversed detection is performed, so that all image frames in the dynamic candidate sequence are abnormal behaviors.

4. The edge-cloud collaborative fatigue driving detection method based on dynamic candidate time series according to claim 1, characterized in that, The steps of storing the image to be detected according to the dynamic candidate sequence and processing the image to be detected according to the fatigue distraction detection model to obtain the detection result are performed asynchronously in a multi-threaded manner.

Images