A cloud-edge collaboration based driving safety detection method

Abstract

The application discloses a driving safety detection method based on cloud-edge cooperation, which comprises the following steps: collecting driver video data and preprocessing; establishing a driving safety detection model and respectively deploying it on an edge device and a cloud server; introducing an intermediate classifier between at least one group of adjacent network layers of each sub-model and training; adopting a time slice rotation scheduling algorithm for multi-thread deployment, realizing time slice allocation of each sub-model, inputting preprocessed data into each trained sub-model and performing inference in parallel; dynamically acquiring a split point of each sub-model to split the corresponding sub-model into an edge network and a cloud network, realizing data compression transmission and recording a starting inference position of the cloud network; the cloud server starts inference according to the acquired compressed data and the starting inference position of the corresponding cloud network, and obtains an inference result; and the inference result is displayed by using a management end and a warning is sent to the driver. The method can accurately and quickly detect abnormal behaviors of the driver and improve driving safety.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of safe driving, and particularly relates to a driving safety detection method based on cloud-edge cooperation. BACKGROUND

[0002] With the rise of artificial intelligence, a large number of detection algorithms are applied to the field of safe driving, and play an important role in detecting bad driving behaviors and preventing traffic accidents. However, due to the excessive computing requirements and increasingly complex network deployment of the deep neural networks on which each detection model is based, it has been difficult to maintain high-performance inference on mobile terminals. The more practical alternative solution is to deploy the inference of the neural network on a cloud server, and the edge device transmits the collected data to the cloud server through network communication, and the inference result is transmitted back after being processed by the cloud server.

[0003] However, the existing point cloud inference is highly dependent on the cloud server, and the inference speed is often limited by the network transmission. For a large batch of data, the network load is often too heavy, causing a bottleneck in the overall performance, and the real-time performance of the system cannot be guaranteed. Moreover, the cost of a high-performance cloud server is often high, which is not conducive to popularization. SUMMARY

[0004] The purpose of the present application is to solve the above problems, and a driving safety detection method based on cloud-edge cooperation is proposed, which can detect abnormal behaviors of drivers more accurately and quickly through real-time task processing by cloud-edge cooperation, and improve the safety during driving.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] The driving safety detection method based on cloud-edge cooperation proposed by the present application comprises the following steps:

[0007] S1, collecting driver video data by using a camera and performing preprocessing;

[0008] S2, establishing a driving safety detection model and deploying it on an edge device and a cloud server respectively, the driving safety detection model comprising three sub-models of a parallel object detection model, a posture recognition model and a fatigue detection model, each sub-model comprising a plurality of network layers and a plurality of Relu layers;

[0009] S3, introducing an intermediate classifier between at least one group of adjacent network layers of each sub-model, and training each sub-model, and when alpha is greater than threshold, the training process is exited, and a trained sub-model is obtained, wherein alpha is the prediction confidence of any class of the current intermediate classifier, and threshold is a preset control threshold variable of the current intermediate classifier;

[0010] S4, a time slice round-robin scheduling algorithm is used for multi-thread deployment, time slice allocation of each sub-model is realized, and preprocessed driver video data is input into each trained sub-model for concurrent inference;

[0011] S5, dynamically obtaining the split point of each sub-model to split the corresponding sub-model into an edge network and a cloud network, realizing data compression transmission and recording the starting inference position of the cloud network, all Relu layers are potential split points, and the dynamic acquisition process of the split point is as follows:

[0012] S51, determining the initial split point of the corresponding sub-model;

[0013] S52, creating a priority queue Q, the priority queue Q is used to store data that has reached the split point, the priority of each batch of data of each sub-model is the same at the beginning, FIFO algorithm is used for data with the same priority, and whether the condition X+G0<=W*T' is met is judged, if yes, the data is transmitted to the cloud server after compression, and the position of the Relu layer for transmission is recorded as the starting inference position of the cloud network, that is, the split point position, otherwise, the data is promoted by one priority, and waits to join the priority queue Q when passing through the next Relu layer, wherein W is the network transmission bandwidth, T' is the network transmission delay, G0 is the existing network load, and X is the head data of the priority queue Q;

[0014] S53, when the data exceeds the preset time and has not joined the priority queue Q, the privilege is obtained according to the first probability P, the privilege is to wait for the network to be idle to load the data, wherein the first probability P=|threshold1-α1| / tolerance, tolerance represents the tolerance error, α1 is the prediction confidence of the first intermediate classifier of the corresponding sub-model, threshold1 is the control threshold variable preset for the first intermediate classifier of the corresponding sub-model, and when the privilege is not obtained, the privilege is immediately given to the second intermediate classifier;

[0015] S54, comparing the task amount processed by each sub-model, if the ratio of the maximum task amount to the minimum task amount exceeds the first preset proportion n1, it is considered that the task is unbalanced, the priority of all transmission tasks of the sub-model corresponding to the minimum task amount is increased, otherwise, the priority remains unchanged;

[0016] S6, the cloud server starts inference according to the obtained compressed data and the starting inference position of the corresponding cloud network, and obtains the inference result of each sub-model;

[0017] S7, using the management end to display the inference result and issuing a warning to the driver according to the inference result.

[0018] Preferably, the object detection model adopts a YOLO series model or a DETR series target detection model, the gesture recognition model adopts an openpose model, and the fatigue detection model includes a Dlib model and a PERCLOS model, and the Dlib model is used to detect the situations of closing eyes, yawning, and head tilting and input to the PERCLOS model.

[0019] Preferably, the PERCLOS model performs the following operations:

[0020] The frame numbers of the situations of closing eyes, yawning, and head tilting in the first preset frame number are calculated respectively, and whether the frame numbers of the three situations exceed the second preset frame number is determined respectively, if yes, different weight values are given to each situation, and the model score Y is calculated s , otherwise, the frame numbers are accumulated again, the situation in the next first preset frame number is calculated, and the cycle is repeated, and the model score Y s is calculated.

[0021] Y s = 0.25x + 0.145y + 0.605z

[0022] In the formula, x represents the frame number of closing eyes, y represents the frame number of yawning, and z represents the frame number of head tilting.

[0023] Whether the model score Y s exceeds the first preset threshold value is determined, if yes, it is considered that the driver is in a fatigue state, otherwise, it is considered that the driver is in a non-fatigue state.

[0024] Preferably, the intermediate classifier is a softmax layer, and is introduced between 15%, 30%, 50%, 70%, 80%, and 90% network layers of the corresponding sub-model respectively.

[0025] Preferably, the time slices of each sub-model are allocated as follows:

[0026] T i = t i / (t1+t2+…+t n )*T

[0027] Where t i is the average inference time of the sub-model i, i = 1, 2, 3, and T is the preset time slice.

[0028] Preferably, after multi-thread deployment by using the time slice round-robin scheduling algorithm, the analyzer is also used to optimize the multi-thread scheduling, and the analyzer performs the following operations:

[0029] Comparing the task amount processed by each sub-model, if the ratio of the maximum task amount to the minimum task amount exceeds a second preset proportion n2, it is considered that the task is unbalanced, and the time slice of the thread of the sub-model corresponding to the minimum task amount is preempted by the thread of the sub-model corresponding to the maximum task amount for (n2-1)*T A , T A is the time slice of the sub-model A corresponding to the maximum task amount, otherwise, the original thread time slice of each sub-model is kept unchanged.

[0030] Preferably, the data compression adopts one or more of precision compression, sparse data compression, sparse matrix compression and down-sampling compression.

[0031] Preferably, the precision compression is to compress 32-bit double-precision data into 8-bit integer data, and the down-sampling compression is two-dimensional maximum pooling down-sampling or two-dimensional average pooling down-sampling.

[0032] Preferably, the preprocessing is image enhancement processing.

[0033] Preferably, the initial segmentation point is the Relu layer corresponding to the minimum inference average time, and when the inference average time is the same, the initial segmentation point is the Relu layer corresponding to the minimum network throughput.

[0034] Compared with the prior art, the beneficial effects of the present application are:

[0035] The method is deployed on edge devices and cloud servers by establishing a driving safety detection model, fuses multiple inference models such as object detection models, posture recognition models and fatigue detection models, and uses multi-threading to complete cloud-edge collaborative processing of real-time inference tasks, meets the needs of users for real-time and low-cost equipment prices, greatly improves the inference speed, and introduces intermediate classifiers and threshold controls between the network layers of each sub-model, which helps to improve the real-time performance of model inference and reduce the complexity of model training, and can adapt to the needs of dynamic adjustment and scheduling and meet the needs of user-defined services, so that the abnormal behavior of the driver can be detected more accurately and quickly, and the safety during driving is improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Fig. 1 is a flowchart of the driving safety detection method based on cloud-edge collaboration of the present application;

[0037] Fig. 2 is a structure schematic diagram of the cloud-edge collaborative safety driving monitoring system of the present application;

[0038] Fig. 3 is an operation flowchart of the PERCLOS model of the present application. DETAILED DESCRIPTION

[0039] With reference to the drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0040] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0041] The method can construct a cloud-edge collaborative safe driving monitoring system according to specific application scenarios to adapt to the needs of dynamic adjustment and scheduling and meet the needs of user-defined services.

[0042] As shown in Figs. 1-3 A driving safety detection method based on cloud-edge collaboration includes the following steps:

[0043] S1, collecting driver video data by using a camera and performing preprocessing.

[0044] In an embodiment, the preprocessing is image enhancement processing. Driver video data may have problems such as deformation, damage, abnormality, and blur when collected. Through image enhancement processing such as transformation, filtering, and denoising, the subsequent model can learn better.

[0045] Specifically, the constructed cloud-edge collaborative safe driving monitoring system includes a camera, an edge device, a cloud server, and a management end. The camera has high real-time performance and is used to capture real-time driver video data. The camera type and specifications can be selected according to actual application. The camera is connected to the edge device through an interface to ensure correct data transmission. The edge device uses Jetson Nano, which is an embedded development board based on GPU processor developed by NVIDIA. The ARM end of Jetson Nano is transplanted with Ubuntu 18.04 LTS system and supports multiple AI frameworks and algorithms, which can perform GPU processing operations. Jetson Nano supports two CSI video input interfaces and USB camera input, and can simultaneously capture two CSI videos and USB camera data and display and process them in real time. The cloud server needs to have high performance and high storage and needs to support AI platforms and GPU development. The management end is used for display output, such as a display.

[0046] S2, a driving safety detection model is established and deployed on an edge device and a cloud server respectively, the driving safety detection model includes three sub-models of an object detection model, a gesture recognition model and a fatigue detection model in parallel, each sub-model includes a plurality of network layers and a plurality of Relu layers.

[0047] In an embodiment, the object detection model adopts a YOLO series model or a DETR series target detection model, the gesture recognition model adopts an openpose model, and the fatigue detection model includes a Dlib model and a PERCLOS model, the Dlib model is used to detect the situations of closing eyes, yawning and head tilting and input to the PERCLOS model.

[0048] In an embodiment, the PERCLOS model performs the following operations:

[0049] The frame numbers of the situations of closing eyes, yawning and head tilting in a first preset frame number are calculated respectively, and whether the frame numbers of the three situations exceed a second preset frame number is judged respectively, if yes, different weight values are given to each situation, and a model score Y is calculated s , otherwise, the frame numbers are accumulated again, the situation in a next first preset frame number is calculated, and the cycle is repeated, and a model score Y s is calculated.

[0050] Y s = 0.25x + 0.145y + 0.605z

[0051] In the formula, x represents the frame number of closing eyes, y represents the frame number of yawning, and z represents the frame number of head tilting.

[0052] Whether the model score Y s exceeds a first preset threshold value is judged, if yes, it is considered that the driver is in a fatigue state, otherwise, it is considered that the driver is in a non-fatigue state.

[0053] The object detection model adopts a YOLO series model or a DETR series target detection model, mainly aiming at driving behaviors prone to distraction such as making a phone call, playing a mobile phone, drinking, drinking water and smoking. The detection targets are mobile phones, wine bottles, water cups and cigarettes. The task of object target detection is to timely identify the above objects and combine the gesture recognition model to detect whether the driver appears distracted driving behavior.

[0054] The gesture recognition model uses an openpose model to recognize gestures, extracts the skeleton of the driver, and determines whether the driver has a distraction or an undesirable driving behavior. The model mainly recognizes the following gestures: making a phone call, playing a mobile phone, drinking water, drinking alcohol, smoking, both hands leaving the steering wheel, whether the arms are away from the car window, etc. The specific recognition method can be determined according to specific occasions and tests. For example, drinking alcohol, drinking water, and smoking can determine the distance of the object from the face, and whether both hands are away from the steering wheel can determine the distance of both hands from the steering wheel.

[0055] Fatigue detection can be achieved in various ways. In this embodiment, the fatigue detection model includes a Dlib model and a PERCLOS model. The Dlib model is used to detect facial feature points first, and then the PERCLOS model is used to analyze the feature points to detect whether there is a fatigue behavior. The Dlib model can describe the face by determining the key points of the face. During detection, the closed eyes, yawning, and head tilt expressions are analyzed.

[0056] The improved PERCLOS model is used to analyze the fatigue of the driver. Within a certain number of frames (first preset number of frames), the number of frames with closed eyes, the number of frames with yawning, and the number of frames with head tilt are analyzed. For example, if the current frame is determined to have closed eyes, the closed eye frame count is incremented by 1, otherwise the closed eye frame count is incremented by 0. The same applies to yawning and head tilt. By assigning different weight values to each case, a threshold value (first preset threshold value) is set for the model score. When the model score exceeds the threshold value, fatigue is detected. The process is shown in Fig. 3 Combining multiple factors can improve the accuracy of detection.

[0057] S3, introduce an intermediate classifier between at least one group of adjacent network layers of each sub-model, and train each sub-model. When a satisfies threshold, the training process is exited, and a trained sub-model is obtained, where a is the prediction confidence of any class of the current intermediate classifier, and threshold is the preset control threshold variable of the current intermediate classifier.

[0058] In an embodiment, the intermediate classifier is a softmax layer, and is introduced between 15%, 30%, 50%, 70%, 80%, and 90% of the network layers of the corresponding sub-model, respectively.

[0059] During training of each sub-model, in order to improve real-time performance of model inference and reduce complexity of model training, an intermediate classifier and a threshold control are introduced, so that the network can be exited early when the accuracy reaches a certain satisfactory level without further propagation. The result of the intermediate classifier network is similar to the last softmax layer of the sub-model. Since part of the recognition may be relatively simple, a relatively accurate result can be obtained after passing through the shallow network. If the inference result continues to move to the deep layer, the accuracy of the inference result improves little or even decreases. At this time, the early exit strategy can make the neural network more flexible and greatly improve the speed of network training and inference.

[0060] The threshold control is introduced in the intermediate classifier layer (that is, the control threshold variable threshold is introduced). After passing through the softmax layer, if the prediction confidence a of a certain category is greater than the given threshold, the network can be exited early without further propagation to improve the classification accuracy. The given threshold strategy makes the real-time performance and accuracy of the network controllable. When a low threshold is set, the network may exit at the shallow network, the model inference real-time performance is good, but more accuracy is lost. When a high threshold is set, the network needs to propagate and infer for a long time, at this time the real-time performance is poor, but the accuracy is higher than the former. The balance can be made according to the actual demand.

[0061] S4, a time slice round-robin scheduling algorithm is used for multi-thread deployment to realize time slice allocation of each sub-model, and the preprocessed driver video data is input into each trained sub-model for concurrent inference.

[0062] In an embodiment, the time slice allocation of each sub-model is as follows:

[0063] T i =t i / (t1+t2+…+t n )*T

[0064] Where t i is the average inference time of sub-model i, i = 1, 2, 3, and T is the preset time slice.

[0065] In an embodiment, after multi-thread deployment using the time slice round-robin scheduling algorithm, the analyzer is also used to optimize multi-thread scheduling. The analyzer performs the following operations:

[0066] Compare the task amounts processed by each sub-model. If the ratio of the maximum task amount to the minimum task amount exceeds the second preset proportion n2, it is considered that the task is unbalanced, and the time slice of the thread of the sub-model corresponding to the minimum task amount is preempted by the thread of the sub-model corresponding to the maximum task amount for (n2-1)*T A , T A is the time slice of the sub-model A corresponding to the maximum task amount, otherwise, the time slices of the original threads of each sub-model remain unchanged.

[0067] Reasonable thread optimization scheduling is important because the real-time performance of the whole system is limited by the model with the slowest inference speed. The multi-threading adopts a time slice round-robin scheduling algorithm, and the time slice is given according to the average inference time of each sub-model. The sub-models are allocated time slices according to the proportion of the average inference time, to realize the initial allocation. Because there is subsequent data network transmission, the inconsistency of the transmission data size of each sub-model will also affect the synchronization of the sub-model inference. For the different synchronization of the model, an analyzer is also needed to optimize the scheduling of the multi-threading.

[0068] S5, dynamically obtaining the split point of each sub-model to split the corresponding sub-model into an edge network and a cloud network, realizing data compression transmission and recording the starting inference position of the cloud network, all Relu layers are potential split points, and the dynamic obtaining process of the split point is as follows:

[0069] S51, determining the initial split point of the corresponding sub-model;

[0070] S52, creating a priority queue Q, the priority queue Q is used to store the data that has reached the split point, the priority of each batch of data of each sub-model is the same at the beginning, FIFO algorithm is adopted for the data with the same priority, and whether the condition X+G0<=W*T' is met is judged, if yes, the data is transmitted to the cloud server after compression, and the position of the Relu layer performing the transmission is recorded as the starting inference position of the cloud network, that is, the split point position, otherwise, the data is promoted to a higher priority, and waits to be loaded into the data when the next Relu layer is reached. The priority queue Q, wherein W is the network transmission bandwidth, T' is the network transmission delay, G0 is the existing network load, and X is the head data of the priority queue Q;

[0071] S53, when the data exceeds the preset time and has not been added to the priority queue Q, the privilege is obtained according to the first probability P, the privilege is to wait for the network to be idle to load the data, wherein the first probability P = |threshold1-α1| / tolerance, tolerance represents the tolerance error, α1 is the prediction confidence of the first intermediate classifier of the corresponding sub-model, threshold1 is the control threshold variable preset for the first intermediate classifier of the corresponding sub-model, and when the privilege is not obtained, the privilege is immediately given to the second intermediate classifier;

[0072] S54, comparing the task amount processed by each sub-model, if the ratio of the maximum task amount to the minimum task amount exceeds the first preset proportion n1, it is considered that the task is unbalanced, the priority of all transmission tasks of the sub-model corresponding to the minimum task amount is increased, otherwise, the priority remains unchanged.

[0073] In an embodiment, the data compression adopts one or more of precision compression, sparse data compression, sparse matrix compression and down-sampling compression.

[0074] In an embodiment, the precision compression is to compress 32-bit double-precision data into 8-bit integer data, and the down-sampling compression is two-dimensional maximum pooling down-sampling or two-dimensional average pooling down-sampling.

[0075] The data compression adopts one or more of precision compression, sparse data compression, sparse matrix compression, and down-sampling compression. Generally, one compression mode is used, and if real-time requirement is high, multiple modes can be combined.

[0076] Among them, for high real-time tasks, precision compression is adopted, which can lose numerical accuracy, such as changing 32-bit double-precision to 8-bit integer, compressing data, and improving data transmission efficiency. In addition, because the segmentation point is the Relu layer, the negative part of the original data becomes 0 after the activation function. Because the data volume is large, the proportion of 0 values in the data will also increase as the network deepens. Sparse array compression or sparse matrix compression can be used for compression. Or it can also be compressed by down-sampling. For example, data can be compressed by two-dimensional maximum pooling down-sampling, average pooling down-sampling, etc.

[0077] In an embodiment, the initial segmentation point is the Relu layer corresponding to the minimum inference average time, and when the inference average time is the same, the initial segmentation point is the Relu layer corresponding to the minimum network throughput.

[0078] Each sub-model is divided into an edge network and a cloud network by selecting a certain Relu layer. Due to the complex network structure of each sub-model, some networks will make the transmission data volume large, so the Relu is taken as the segmentation point to limit the negative values activated by the Relu to 0, which supports data compression. In order to consider the synchronization of each sub-model inference, the following dynamic selection method is adopted:

[0079] For a single sub-model, multiple key Relu layers can be selected for testing to obtain average inference time, network throughput, etc. as indicators, and an ideal segmentation point is selected. Generally, the shorter the average inference time, the more ideal the segmentation point. Note that the segmentation point cannot be too deep into the network to prevent the model from being inferred on the edge device for a long time and unable to be transmitted to the cloud server.

[0080] A priority queue Q is created for storing a batch of data that has arrived at a split point, at the beginning, the priority of each batch of data of each sub-model is the same, for the same priority, FIFO (first in first out) strategy is adopted. For whether to perform network transmission, assuming that the network transmission bandwidth is W, the network transmission delay is T', and the existing network load is G0, for the head data X of the priority queue Q, if X+G0<=W*T', it can enter the network transmission to the cloud server, otherwise, it is promoted to a priority, and when it passes through the next Relu layer, it is added to the queue. The Relu layer performing transmission needs to record the position of the Relu layer, so that the cloud server can identify the starting inference position of the cloud network. Because the data also needs to be compressed, it will not cause excessive network congestion.

[0081] Generally, because some data is too large, it cannot be easily added to the network, and has been inferred for a long time in the edge device. The solution is to judge the prediction confidence a1 in the corresponding sub-model 15% intermediate classifier, and set the first probability P according to the difference with the control threshold variable, the formula is as follows: P=|threshold1-a1| / tolerance, (tolerance represents the tolerance error, generally 5%). Make it get the privilege under this first probability P, the privilege refers to when the data cannot be added to the network, instead of continuing to infer to the next Relu layer, but waiting for the network to be idle to load the data, this strategy can ensure the transmission of large batch data. And when no privilege is obtained, privilege should be given immediately at the 30% layer.

[0082] For the imbalance of each sub-model inference, it is usually caused by the inconsistency of the split point position of each sub-model and the inference speed of the sub-model. Similarly, the priority can be controlled to determine the order of entering the network transmission. Set the first preset ratio n1, for sub-model A and B, if the sub-task processed by A is n times of B, such as n1=1.3-1.7, it is considered that the task is relatively unbalanced, and the priority of all transmission tasks of B needs to be increased to speed up the network transmission process of B. The task amount processed by sub-model A corresponds to the maximum task amount, and the task amount processed by sub-model B corresponds to the minimum task amount.

[0083] S6, the cloud server starts inference according to the obtained compressed data and the starting inference position of the corresponding cloud network, and obtains the inference results of each sub-model. After the compressed data is transmitted to the cloud server, the starting inference position of the cloud network is obtained by obtaining the Relu position in the data, and the inference is continued at this position.

[0084] S7, using the management end to display the inference result and issuing a warning to the driver according to the inference result.

[0085] The method is deployed on an edge device and a cloud server by establishing a driving safety detection model, fuses multiple inference models such as an object detection model, a posture recognition model and a fatigue detection model, and uses multi-threading to complete cloud-edge collaborative processing of real-time inference tasks, meets the demand of users for real-time performance and low equipment price, greatly improves the inference speed, introduces an intermediate classifier and a threshold control between network layers of each sub-model, helps to improve the real-time performance of model inference and reduce the complexity of model training, and can adapt to the needs of dynamic adjustment and scheduling and meet the demand of users for customized services according to specific scene applications, so that the abnormal behavior of the driver can be detected more accurately and quickly, and the safety in the driving process is improved.

[0086] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above-described embodiments are not described, but as long as the combinations of the technical features do not contradict each other, they should be considered within the scope of the present disclosure.

[0087] The above-described embodiments only express the specific and detailed embodiments described in the present application, but should not be construed as limiting the scope of the patent application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A driving safety detection method based on cloud-edge collaboration, characterized in that: The cloud-edge collaborative driving safety detection method includes the following steps: S1. Collect driver video data using a camera and preprocess it; S2. Establish a driving safety detection model and deploy it on edge devices and cloud servers respectively. The driving safety detection model includes three sub-models: a parallel object detection model, a posture recognition model, and a fatigue detection model. Each sub-model includes multiple network layers and multiple ReLU layers. S3. Introduce intermediate classifiers between at least one set of adjacent network layers of each sub-model and train each sub-model. When α > threshold, exit the training process and obtain the trained sub-model. Here, α is the prediction confidence of any class of the current intermediate classifier, and threshold is the preset control threshold variable of the current intermediate classifier. S4. Use a time-slice round-robin scheduling algorithm for multi-threaded deployment to allocate time slices for each sub-model, and input the pre-processed driver video data into each trained sub-model for concurrent inference. S5. Dynamically obtain the segmentation points of each sub-model. The corresponding sub-model is segmented into an edge network and a cloud network. Data compression and transmission are implemented, and the starting inference position of the cloud network is recorded. All ReLU layers are potential segmentation points. The dynamic acquisition process of the segmentation points is as follows: S51. Determine the initial segmentation points of the corresponding sub-model; S52. Create a priority queue Q. The priority queue Q is used to store data that has reached the split point. Initially, the data of each batch of each sub-model has the same priority. For data with the same priority, the FIFO algorithm is used, and it is determined whether the condition X + G0 <= W * T′ is satisfied. If so, the data is compressed and transmitted to the cloud server. The position of the ReLU layer during transmission is recorded as the starting inference position of the cloud network, i.e., the split point position. Otherwise, the data is promoted by one priority and waited to be added to the priority queue Q when it passes through the next ReLU layer. Where W is the network transmission bandwidth, T′ is the network transmission delay, G0 is the current network load, and X is the head data of the priority queue Q. S53. When the data has not been added to the priority queue Q after a preset time, privilege is obtained according to the first probability P. The privilege is to wait for the network to be idle enough to load the data. The first probability P = |threshold1-α1| / tolerance, where tolerance represents the tolerance error, α1 is the prediction confidence of the first intermediate classifier of the corresponding sub-model, and threshold1 is the preset control threshold variable of the first intermediate classifier of the corresponding sub-model. If the privilege is not obtained, the privilege is immediately granted in the second intermediate classifier. S54. Compare the workload of each sub-model. If the ratio of the maximum workload to the minimum workload exceeds the first preset ratio n1, the workload is considered unbalanced. The priority of all transmission tasks of the sub-model corresponding to the minimum workload is increased. Otherwise, the priority remains unchanged. S6. The cloud server starts inference based on the acquired compressed data and the corresponding starting inference position of the cloud network, and obtains the inference results of each sub-model. S7. Display the reasoning results using the management terminal and issue warnings to the driver based on the reasoning results.

2. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: The object detection model adopts the YOLO series model or the DETR series target detection model, the pose recognition model adopts the OpenPose model, and the fatigue detection model includes the Dlib model and the PERCLOS model. The Dlib model is used to detect closed eyes, yawning, and head tilting and inputs them into the PERCLOS model.

3. The driving safety detection method based on cloud-edge collaboration as described in claim 2, characterized in that: The PERCLOS model performs the following operations: Calculate the number of frames within a first preset frame count that exhibit the three scenarios of closed eyes, yawning, and head tilting. Then, determine whether the number of frames for each scenario exceeds a second preset frame count. If so, assign different weight values ​​to each scenario and calculate the model score Y. s Otherwise, the frame count is recalculated, and the situation within the next first preset frame count is calculated. This process is repeated, and the model score Y is obtained. s The formula is as follows: AND s =0.25x+0.145y+0.605z In the formula, x represents the number of frames with eyes closed, y represents the number of frames with yawning, and z represents the number of frames with head tilted. Judge the model score Y s If the threshold is exceeded, the driver is considered to be fatigued; otherwise, the driver is considered to be not fatigued.

4. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: The intermediate classifier is a softmax layer, which is introduced between the 15%, 30%, 50%, 70%, 80%, and 90% network layers of the corresponding sub-models.

5. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: The time slice allocation for each sub-model is as follows: T i =t i / (t1+t2+…+t n )*T Among them, t i Let T be the average inference time for sub-model i, where i = 1, 2, 3, and T is the preset time slice.

6. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: After employing the time-slice round-robin scheduling algorithm for multi-threaded deployment, a analyzer is used to optimize multi-threaded scheduling. The analyzer performs the following operations: Compare the workload of each sub-model. If the ratio of the maximum workload to the minimum workload exceeds the second preset ratio n², the workload is considered unbalanced. The thread of the sub-model with the minimum workload will then preempt the time slice of the thread of the sub-model with the maximum workload by (n²-1)*T. A T A The time slice for sub-model A corresponds to the largest task volume; otherwise, the time slices of the original threads of each sub-model remain unchanged.

7. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: The data compression employs one or more of the following: precision compression, sparse data compression, sparse matrix compression, and downsampling compression.

8. The driving safety detection method based on cloud-edge collaboration as described in claim 7, characterized in that: The precision compression is to compress 32-bit double-precision data into 8-bit integer data, and the downsampling compression is two-dimensional max pooling downsampling or two-dimensional average pooling downsampling.

9. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: The preprocessing is image enhancement processing.

10. The driving safety detection method based on cloud-edge collaboration as described in claim 1, characterized in that: The initial split point is the ReLU layer corresponding to the minimum average inference time, and when the average inference time is the same, the initial split point is the ReLU layer corresponding to the minimum network throughput.

Images