Non-driving posture recognition and safety assessment method based on lightweight architecture
By employing a lightweight architecture for non-driving posture recognition and safety assessment, and utilizing lightweight neural networks and key point localization technology, the accuracy and timeliness issues of driver posture assessment in Level 3 autonomous driving are addressed, achieving efficient safety assessment and resource conservation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH
- Filing Date
- 2024-04-11
- Publication Date
- 2026-07-21
Smart Images
Figure CN118262339B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of traffic safety technology, specifically to a non-driving posture recognition and safety assessment method based on a lightweight architecture. Background Technology
[0002] Autonomous driving technology is rapidly advancing the intelligentization of new energy vehicles and greatly enhancing the user driving experience, allowing drivers to engage in non-driving tasks. However, when the vehicle's driving status and the driver's level of control are severely mismatched, it can lead to enormous economic losses and catastrophic casualties. Therefore, the recognition and safety assessment of drivers' non-driving postures are of paramount practical importance.
[0003] Current research primarily focuses on identifying driver distraction and fatigue in non-autonomous driving scenarios. However, with the rapid development of Level 3 autonomous driving technology, driver posture has changed significantly. Therefore, it is necessary to develop non-driving posture recognition for Level 3 autonomous vehicles to effectively improve the accuracy and timeliness of recognition methods, thereby facilitating safe takeover of autonomous vehicles. Summary of the Invention
[0004] The purpose of this invention is to address the issues of accuracy and timeliness in safety assessment of non-driving postures using existing recognition methods, and to propose a non-driving posture recognition and safety assessment method based on a lightweight architecture.
[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0006] A non-driving posture recognition and safety assessment method based on a lightweight architecture includes the following steps:
[0007] Step 1: Acquire in-vehicle monitoring video data, which includes manual driving posture and non-driving posture during autonomous driving;
[0008] Step 2: Decompose the in-vehicle monitoring video data into multiple video segments;
[0009] Step 3: For each video segment, label the manual driving posture and the non-driving posture during autonomous driving in the image frame to build a sample set, and use the sample set to train the neural network.
[0010] Step 4: Use the trained neural network to identify each frame of the surveillance video. When the identification result is a manual driving posture, locate the key points of the current frame and use the location result as a reference key point.
[0011] When the recognition result is a non-driving posture during autonomous driving, the key points of the current frame image are located, and a safety assessment is performed using the current key point location result and the reference key point closest to the current frame.
[0012] Furthermore, the specific steps of step four are as follows:
[0013] Step 4: Use the trained neural network to identify each frame of the surveillance video. When the identification result is manual driving posture, locate the key points of the current frame image to obtain the driver's activity area S and the key point position of the driver's skeleton. Then, retain the pixels of the driver's activity area S and clear the remaining pixels to zero. Finally, use the key point position of the driver's skeleton as the reference key point.
[0014] When the recognition result is a non-driving posture during autonomous driving, the key point localization is performed on the current frame image to obtain the driver's activity area S and the driver's skeletal key point position. Then, the pixels of the driver's activity area S are retained and the remaining pixels are cleared to zero. Finally, the safety assessment is performed using the current driver's skeletal key point position and the reference key point closest to the current frame.
[0015] Furthermore, the key points of the driver's skeleton include: the key point of the nose, the key point of the right shoulder, the key point of the right elbow, the key point of the right palm, the key point of the right hip, and the key point of the right knee.
[0016] Furthermore, the specific steps of the security assessment are as follows:
[0017] Step 41: Based on the reference key points, obtain the distance d from the nose key point to the right shoulder key point. Then, with the nose key point as the center, divide the area into two concentric circles with radii of d / 2 and d respectively. Divide the area into zone 1, zone 2 and zone 3 with the inner circle, the middle ring and the outer area of the ring.
[0018] Step 42: Based on the reference key points, obtain the key point positions of the right hip and the right knee. Then, vertically upward from the right hip key point to the image edge point, and form a triangular region based on the right hip key point, the right knee key point, and the image edge point. Obtain the circumcircle of the triangular region. The region within the triangular region is designated as Region I, the region between the triangular region and the circumcircle is designated as Region II, and the region outside the circumcircle is designated as Region III.
[0019] Step 43: Based on the current skeletal key point positions of the driver, obtain the key point positions of the nose and the right palm;
[0020] Step 44: Based on the areas divided in Step 41, determine whether the key nose point in Step 43 is located in Zone 1, Zone 2, or Zone 3;
[0021] Steps 4 and 5: Based on the areas divided in Step 4 and 2, determine whether the key point of the right palm in Step 4 and 3 is in Zone I, Zone II, or Zone III;
[0022] Step 46: Conduct a safety assessment based on the results of Steps 44 and 45.
[0023] Furthermore, the specific steps of steps four and six are as follows:
[0024] Step 461: Assign scores to different non-driving postures during autonomous driving;
[0025] Step 462: Assign scores to Zones 1, 2, and 3 as defined in Step 41, and to Zones I, II, or III as defined in Step 42.
[0026] Step 463: Based on Step 462, obtain the scores for the nose key point position and the right palm key point position corresponding to the non-driving posture during autonomous driving.
[0027] Step 464: Conduct a security assessment based on the total score from Steps 461 and 463.
[0028] Furthermore, the different non-driving postures during autonomous driving include waiting to drive, turning to the side to pick up an item, operating the central screen, reading, looking at a mobile phone, using a laptop, making a phone call, and eating.
[0029] Furthermore, the specific steps of step two are as follows:
[0030] The in-vehicle monitoring video data is broken down into multiple video segments every 2 seconds;
[0031] The specific steps for constructing the sample set are as follows:
[0032] Under each category, each video segment is used as an independent sample. Frames of a 2-second continuous video are extracted, resized to 128×171 and saved, and converted into images with coherent poses. Each independent sample contains at least 16 images, thus constructing a sample set.
[0033] Furthermore, the keypoint localization is performed using a lightweight YOLOv8 keypoint localization algorithm.
[0034] Furthermore, the neural network is a lightweight three-dimensional convolutional neural network.
[0035] Furthermore, the loss function of the lightweight 3D convolutional neural network is expressed as:
[0036]
[0037] Where p and q are probability distributions, p(x) is the expected output, and q(x) is the actual output.
[0038] The beneficial effects of this invention are:
[0039] This application first identifies the non-driving state, and then performs a safety assessment by using the location information of the driver's skeletal key points. The technical solution of this application ensures the accuracy of the safety assessment, with an accuracy rate of up to 96.75%. Attached Figure Description
[0040] Figure 1 This is a framework diagram of a specific implementation method of this application;
[0041] Figure 2 This is a schematic diagram of the preprocessing process;
[0042] Figure 3 This is a schematic diagram illustrating the process of separating 3D convolution from convolution depth.
[0043] Figure 4 Diagram showing the partition boundaries for key points. Figure 1 ;
[0044] Figure 5 Diagram showing the partition boundaries for key points. Figure 2 . Detailed Implementation
[0045] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.
[0046] Specific Implementation Method 1: This implementation method, based on a lightweight architecture, for non-driving posture recognition and safety assessment, is characterized by the following steps:
[0047] Step 1: Acquire in-vehicle monitoring video data, which includes manual driving posture and non-driving posture during autonomous driving;
[0048] Step 2: Decompose the in-vehicle monitoring video data into multiple video segments;
[0049] Step 3: For each video segment, label the manual driving posture and the non-driving posture during autonomous driving in the image frame to build a sample set, and use the sample set to train the neural network.
[0050] Step 4: Use the trained neural network to identify each frame of the surveillance video. When the identification result is a manual driving posture, locate the key points of the current frame and use the location result as a reference key point.
[0051] When the recognition result is a non-driving posture during autonomous driving, the key points of the current frame image are located, and a safety assessment is performed using the current key point location result and the reference key point closest to the current frame.
[0052] Specific Implementation Method Two: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that the specific steps of step four are as follows:
[0053] Step 4: Use the trained neural network to identify each frame of the surveillance video. When the identification result is manual driving posture, locate the key points of the current frame image to obtain the driver's activity area S and the key point position of the driver's skeleton. Then, retain the pixels of the driver's activity area S and clear the remaining pixels to zero. Finally, use the key point position of the driver's skeleton as the reference key point.
[0054] When the recognition result is a non-driving posture during autonomous driving, the key point localization is performed on the current frame image to obtain the driver's activity area S and the driver's skeletal key point position. Then, the pixels of the driver's activity area S are retained and the remaining pixels are cleared to zero. Finally, the safety assessment is performed using the current driver's skeletal key point position and the reference key point closest to the current frame.
[0055] Specific Implementation Method 3: This implementation method is a further explanation of Specific Implementation Method 2. The difference between this implementation method and Specific Implementation Method 2 is that the key points of the driver's skeleton include: the key point of the nose, the key point of the right shoulder, the key point of the right elbow, the key point of the right palm, the key point of the right hip, and the key point of the right knee.
[0056] Specific Implementation Method Four: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that the specific steps of the security assessment are as follows:
[0057] Step 41: Based on the reference key points, obtain the distance d from the nose key point to the right shoulder key point. Then, with the nose key point as the center, divide the area into two concentric circles with radii of d / 2 and d respectively. Divide the area into zone 1, zone 2 and zone 3 with the inner circle, the middle ring and the outer area of the ring.
[0058] Step 42: Based on the reference key points, obtain the key point positions of the right hip and the right knee. Then, vertically upward from the right hip key point to the image edge point, and form a triangular region based on the right hip key point, the right knee key point, and the image edge point. Obtain the circumcircle of the triangular region. The region within the triangular region is designated as Region I, the region between the triangular region and the circumcircle is designated as Region II, and the region outside the circumcircle is designated as Region III.
[0059] Step 43: Based on the current skeletal key point positions of the driver, obtain the key point positions of the nose and the right palm;
[0060] Step 44: Based on the areas divided in Step 41, determine whether the key nose point in Step 43 is located in Zone 1, Zone 2, or Zone 3;
[0061] Steps 4 and 5: Based on the areas divided in Step 4 and 2, determine whether the key point of the right palm in Step 4 and 3 is in Zone I, Zone II, or Zone III;
[0062] Step 46: Conduct a safety assessment based on the results of Steps 44 and 45.
[0063] Specific Implementation Method Five: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that the specific steps of steps four and six are as follows:
[0064] Step 461: Assign scores to different non-driving postures during autonomous driving;
[0065] Step 462: Assign scores to Zones 1, 2, and 3 as defined in Step 41, and to Zones I, II, or III as defined in Step 42.
[0066] Step 463: Based on Step 462, obtain the scores for the nose key point position and the right palm key point position corresponding to the non-driving posture during autonomous driving.
[0067] Step 464: Conduct a security assessment based on the total score from Steps 461 and 463.
[0068] Specific Implementation Method Six: This implementation method is a further explanation of Specific Implementation Method Five. The difference between this implementation method and Specific Implementation Method Five is that the different non-driving postures during the autonomous driving process include waiting to drive, turning to the side to pick up items, operating the central screen, reading, looking at a mobile phone, using a laptop, making a phone call, and eating.
[0069] Specific Implementation Method Seven: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that the specific steps of step two are as follows:
[0070] The in-vehicle monitoring video data is broken down into multiple video segments every 2 seconds;
[0071] The specific steps for constructing the sample set are as follows:
[0072] Under each category, each video segment is used as an independent sample. Frames of a 2-second continuous video are extracted, resized to 128×171 and saved, and converted into images with coherent poses. Each independent sample contains at least 16 images, thus constructing a sample set.
[0073] Specific Implementation Method Eight: This implementation method is a further explanation of Specific Implementation Method Seven. The difference between this implementation method and Specific Implementation Method Seven is that the key point localization is performed using a lightweight YOLOv8 key point localization algorithm.
[0074] Specific Implementation Method Nine: This implementation method is a further explanation of Specific Implementation Method Eight. The difference between this implementation method and Specific Implementation Method Eight is that the neural network is a lightweight three-dimensional convolutional neural network.
[0075] Specific Implementation Method Ten: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that the loss function of the lightweight three-dimensional convolutional neural network is expressed as follows:
[0076]
[0077] Where p and q are probability distributions, p(x) is the expected output, and q(x) is the actual output.
[0078] This application first preprocesses the collected non-driving posture video data to obtain the driver's activity range and the location information of the driver's skeletal key points. Then, based on the driver's activity range, the pixels of the remaining images are zeroed out, and a sample set is constructed based on the correlation between the samples. Next, a lightweight three-dimensional convolutional neural network model is constructed to address the structural characteristics of spatiotemporally coherent non-driving postures. This three-dimensional neural network model is trained based on non-driving posture classification data, and the optimal model is selected by changing the hyperparameters of the network structure. Then, the driver's skeletal key point location information obtained from the data preprocessing is compared with a preset spatial range, and the driver's skeletal key points are scored in conjunction with the takeover time. A time-based statistical warning task is simultaneously set to prevent the driver from going too far into non-driving tasks. Finally, the driver's non-driving posture is classified and scored, and a safety assessment of the driver's non-driving posture is completed by combining the two scores.
[0079] Example:
[0080] Step 1: Video Data Preprocessing
[0081] (1) The data collection process was completed by 15 people, each in two sessions, for a total of 30 experiments. During the data collection, the camera was set up on the front of the passenger seat facing the driver. The dataset contained 12 hours and more than 9.6 million frames of the driver's manual driving posture and non-driving posture during autonomous driving, including but not limited to waiting to drive, turning to pick up items, operating the central screen, reading, looking at the mobile phone, using the laptop, making a phone call, eating, etc.
[0082] (2) When using data for model training:
[0083] The 30 long videos were broken down into 18,086 video segments, each lasting 2 seconds.
[0084] A lightweight YOLOv8 keypoint localization algorithm is used to perform keypoint localization operations on image frames of all video segments and output the driver's activity area S.
[0085] (3) When the data is used for real-time video or camera detection:
[0086] A lightweight YOLOv8 keypoint localization algorithm is used to perform keypoint localization operations on the image frames input by the system, outputting the driver's active area S and the sequence of keypoint position information V = {v0, v1, v2, v3, v4, v5} of the driver's skeleton. To reduce the amount of computation, only six points need to be located: the driver's nose position v0, the right shoulder position v1, the right elbow position v2, the right palm position v3, the right hip position v4, and the right knee position v5. The specific steps of (2) and (3) are as follows: Figure 2 As shown.
[0087] (4) Using the location information of the driver's activity area S, retain the pixel information of the driver's activity area S, and clear the pixels of the remaining areas to zero. After the clearing operation, the average height (D) of each frame is... H ) and width (D W All were reduced to their original size. Image resolution Reduced to Less than half the original amount, saving computational resources. At the same time, due to the removal of irrelevant background, the subsequent model training process also improved convergence speed and model accuracy. Experimental tests showed that this step improved the model's accuracy by nearly two percentage points.
[0088] Step 2: Construct a non-driving posture sample set
[0089] (1) Manually classify the non-driving postures in 18086 video segments (as described in step one) to balance the relative number of samples in each class.
[0090] (2) Under each category, each video segment is used as an independent sample. The frames of the 2-second continuous video are extracted, resized to 128×171 and saved. They are then converted into images with coherent poses to ensure that each independent sample contains at least 16 images. The images and labeled categories complete the construction of the sample set.
[0091] Step 3: Training and Evaluation of a Lightweight 3D Convolutional Neural Network Model
[0092] (1) This application employs a depthwise separable convolutional 3D convolutional neural network (e.g., Figure 3As shown in the diagram, 16 images are selected as input from each independent sample, and the spatiotemporal information of the data is extracted simultaneously. When the images are input into the model, they are first unfolded by channel, and then depthwise convolution is performed. After that, all channels are re-fused through pointwise convolution.
[0093] (2) Divide the non-driving posture sample set into training set, validation set and test set according to the proportion. Input the training set into the model for model training. Obtain the optimal model by changing the hyperparameters of the network structure. Set the learning rate to 0.03 and the number of iterations to 80. Finally, input the test set into the trained model to obtain the non-driving posture category of the test samples.
[0094] (3) When using the trained model for video recognition and real-time camera recognition, the system reads in 16 frames of images in real time. Step 1 directly preprocesses this input data and only performs operations (3) and (4).
[0095] (4) This application uses the expected cross-entropy of the classification output as the loss function:
[0096]
[0097] In the formula, probability distribution p represents the desired output, and probability distribution q represents the actual output.
[0098] The loss value calculated by the loss function and the accuracy of fault diagnosis calculated according to the following formula:
[0099]
[0100] In the formula n correct n is the number of all correctly classified samples. total This represents the total number of samples.
[0101] Training results show that the loss value is 0.0740 and the accuracy is 96.75%, which is at the leading level in the industry.
[0102] Step 4: Comprehensive Information Fusion and Security Assessment
[0103] The model designed and trained in this application has good recognition performance for non-driving postures in L3 autonomous driving scenarios. However, the process of quantifying indicators is still needed from the recognition results to the safety assessment. This step will provide the process of quantifying indicators and assessing safety through information fusion.
[0104] (1) For the various non-driving postures included in the data sample, under the current posture, each posture is scored according to the time from the driver receiving the information to the start of the action reaction. The full score is 60 points. Among them, the reaction time of "waiting to take over driving" is the shortest among all non-driving postures, and it is set as the full score as a reference.
[0105] (2) For the same non-driving posture, the driver's relative position inside the vehicle may differ, and the time from reacting to actually taking over driving control also varies depending on the position. Based on the average takeover time statistically analyzed in experiments, and combined with the lightweight principle of "avoiding repeated use of core computing resources," different levels of positional areas are defined in the visual image. Specifically, using the seat headrest as the reference position, two concentric circles are divided, and the inner circle, the middle ring, and other areas are used to divide areas into regions I, II, and III (e.g., ...). Figure 4 As shown), specifically, using the driver's skeletal key point position information obtained in step one, when the driver is waiting to drive or manually driving, the distance d from the driver's nose to the right shoulder is calculated based on the seat headrest. Two concentric circles are then formed with the nose position as the center and radii of d / 2 and d, respectively. The region division is updated only when the driver is waiting to drive or manually driving, dividing the area into regions I, II, and III based on the inner circle, the middle ring, and other regions (e.g., ...). Figure 5 As shown, using the steering wheel inside the car as the reference position, a triangular area is formed by the driver's right hip point, extending vertically upwards from the right hip point to the edge of the image, and the driver's right knee point.
[0106] Using the driver's skeletal key point position information obtained in step one, with the steering wheel inside the car as the reference position, a triangular area is formed by the driver's right hip point, the vertical line from the right hip point to the edge of the screen, and the driver's right knee point. This area is set as area I. The area outside area I of the circumscribed ellipse is set as area II, and the other areas are set as area III.
[0107] The location of the driver's nose indicates the time it takes for the driver to receive information and react; this area is defined relative to the position of the seat backrest. The location of the driver's hands indicates the time the driver needs to grip the steering wheel; this area is defined relative to the position of the steering wheel. When the driver's nose or hands are located in Zone I, a full score of 20 points is awarded respectively; the scores for Zones II and III decrease sequentially.
[0108] (3) Further, a total score of 100 points is set, with 60 points allocated to non-driving posture and 40 points to key skeletal points, of which the nose and palm positions are equally scored. Based on the output of the system, each item is scored according to the quantitative evaluation indicators, and the total score is finally calculated to conduct a safety assessment of the driver's non-driving posture. Table 1 will show the correspondence between the weighted total score and the safety assessment.
[0109] Table 1. Correspondence between weighted total score and safety assessment
[0110] 85-100 Safety 75-85 (excluding 85) generally 60-75 (excluding 75) There are risks involved; further assessment is needed over time. 60 years old and below Danger
[0111] When the weighted total score is in the range of [60, 75], further judgment needs to be made in conjunction with (4).
[0112] (4) Further, based on (2), a time-based early warning task is set up. The impact of the driver maintaining the same type of non-driving posture for different durations on the driver's takeover ability and level is different. When the driver's nose and palm positions deviate from the optimal area for receiving driving information and the optimal area for taking over the steering wheel for a long time (the specific duration needs to be set according to the actual road conditions), the system will issue a warning sound to remind the driver, so as to prevent the driver from completely losing attention from the main driving task due to maintaining a poor non-driving posture for a long time. If the driver still maintains the current poor non-driving posture, the system will forcibly lower the driver's safety score and issue a command for the vehicle to slowly decelerate and move to the slow lane. Specifically, as follows: Figure 1 As shown.
[0113] (1) This application uses a lightweight YOLOv8 human skeleton key point localization model to preprocess the input data, delineate the effective activity area of the driver, and clear the remaining pixels, so that the subsequent model training and recognition computation is reduced to about 45% of the original, less than half.
[0114] (2) This application innovatively applies a three-dimensional convolutional neural network model with depth separable convolution to the recognition of driver non-driving postures. By introducing a time series dimension into the two-dimensional plane, it can better extract non-driving posture features with temporal continuity, greatly improve the recognition accuracy, and at the same time, the computational load is only 3.7% of the original, showing an order-of-magnitude decrease.
[0115] (3) This application innovatively proposes a system process of "one-time identification and information integration" and proposes a lightweight idea of "not repeatedly calling core computing resources". The entire identification and safety assessment process only calls the network model with the largest amount of computation in the entire system once. By combining the non-driving posture classification results and the key point position information of the skeleton with the driver's reaction time and takeover time, a safety assessment quantitative index is innovatively given, and the level at which the driver is currently suitable for takeover is comprehensively obtained.
[0116] (4) This application innovatively sets a time statistics warning function in the system, which effectively avoids the problem that the driver's thinking and attention are too deeply involved in non-driving tasks during the autonomous driving process, resulting in a rapid decline in the driver's takeover ability and level.
[0117] (5) This application adopts a lightweight architecture throughout the entire process, which is intended to be deployed on vehicle-mounted assisted driving edge computing devices with limited computing power to identify and assess the non-driving posture of the driver under L3 autonomous driving conditions, ensuring the real-time performance, accuracy and privacy of the system. Using the method of this application, the accuracy of identification can reach 96.75%, which is at the leading level in the industry.
[0118] To demonstrate the effectiveness of this application, the following steps can be implemented in practical application:
[0119] The driver's current posture range S in the video data is located, and the driver's skeletal keypoint position sequence V is output. The area of S is kept at an average of half of the entire frame, and pixels outside S are cleared to zero. Taking a certain frame as an example, the total frame size is 128×171, and the width of the area where the driver is currently located is half of the total frame size. Width is the total screen area The resolution of the preprocessed input data can be calculated through step one:
[0120] Original: 128 × 171 = 21888
[0121] After processing:
[0122] In this example, the processed resolution is only 22% of the original, and the hardware computation is about 20% of the original, greatly reducing the demand on hardware performance.
[0123] The system automatically invokes the trained non-driving posture recognition model, inputs the data processed in the previous step into a 3D convolutional neural network based on depthwise separable convolution, and outputs classification results. In this step, the computational complexity formula for depthwise separable convolution is:
[0124] Mult-Adds=D K ×D K ×D K ×M×D W ×D H +M×N×D W ×D H (3)
[0125] The computational complexity formula for ordinary convolution is:
[0126] Mult-Adds=D K ×D K ×D K ×M×N×D W ×D H (4)
[0127] Therefore, the ratio of the computational cost to the original cost is:
[0128]
[0129] Typically, N is between 128 and 1024. When N is in the denominator, this term can be ignored. Therefore, in this example, this application greatly reduces the computational load to 2.47% of the original, demonstrating a significant lightweighting effect.
[0130] Since this example is only used for demonstration and is not a test set for training, the video input duration and driver posture richness are limited. In this example, the output classification accuracy exceeds 99%, which is higher than the overall accuracy of 96.75%, and the recognition effect is good.
[0131] This example includes four typical categories of non-driving postures: eating, talking on the phone, using a laptop, and turning sideways to retrieve an item. According to step five, these four postures in the video data are scored (divided into three parts, with maximum scores of 60, 20, and 20 respectively) and evaluated. The detailed calculation and evaluation process is as follows:
[0132] Diet: This category is scored 52.5 based on experimental data. In this example, the nose is located in Zone I, scoring 20, and the right palm is located in Zone II, scoring 16. Therefore, the total score is 52.5 + 20 + 16 = 88.5, which is considered safe in this case.
[0133] Making a phone call: This category is scored out of 45 based on experimental data. In this example, the nose is in zone I, scoring 20, and the right palm is in zone III, scoring 12. Therefore, the total score is 45 + 20 + 12 = 77, which is rated as average in this example.
[0134] Using a notebook: This category is scored out of 36 based on experimental data. In this example, the nose is located in Zone II, scoring 16, and the right palm is located in Zone II, also scoring 16. Therefore, the total score is 36 + 16 + 16 = 68. In this case, it is judged as a safety risk, and attention should be paid to the driver's duration of maintaining this posture.
[0135] Sideways flipping to retrieve items: This category is scored 22.5 based on experimental data. In this example, the nose is located in Zone III, scoring 12, and the right palm is located in Zone III, also scoring 12. Therefore, the total score is 22.5 + 12 + 12 = 46.5, which is judged as dangerous in this case. The driver needs to be reminded to correct their current posture and the vehicle should be slowed down appropriately.
[0136] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.
Claims
1. A non-driving posture recognition and safety assessment method based on a lightweight architecture, characterized in that... Includes the following steps: Step 1: Acquire in-vehicle monitoring video data, which includes manual driving posture and non-driving posture during autonomous driving; Step 2: Decompose the in-vehicle monitoring video data into multiple video segments; Step 3: For each video segment, label the manual driving posture and the non-driving posture during autonomous driving in the image frame to build a sample set, and use the sample set to train the neural network. Step 4: Use the trained neural network to identify each frame of the surveillance video. When the identification result is a manual driving posture, locate the key points of the current frame and use the location result as a reference key point. When the recognition result is a non-driving posture during autonomous driving, the key points of the current frame image are located, and a safety assessment is performed using the current key point location result and the reference key point closest to the current frame. The specific steps of the security assessment are as follows: Step 41: Based on the reference key points, obtain the distance d from the nose key point to the right shoulder key point. Then, with the nose key point as the center, divide the area into two concentric circles with radii of d / 2 and d respectively. Divide the area into zone 1, zone 2 and zone 3 with the inner circle, the middle ring and the outer area of the ring. Step 42: Based on the reference key points, obtain the key point positions of the right hip and the right knee. Then, vertically upward from the right hip key point to the image edge point, and form a triangular region based on the right hip key point, the right knee key point, and the image edge point. Obtain the circumcircle of the triangular region. The region within the triangular region is designated as Region I, the region between the triangular region and the circumcircle is designated as Region II, and the region outside the circumcircle is designated as Region III. Step 43: Based on the current skeletal key point positions of the driver, obtain the key point positions of the nose and the right palm; Step 44: Based on the areas divided in Step 41, determine whether the key nose point in Step 43 is located in Zone 1, Zone 2, or Zone 3; Steps 4 and 5: Based on the areas divided in Step 4 and 2, determine whether the key point of the right palm in Step 4 and 3 is in Zone I, Zone II, or Zone III; Step 46: Conduct a security assessment based on the results of Steps 44 and 45; The specific steps of steps four and six are as follows: Step 461: Assign scores to different non-driving postures during autonomous driving; Step 462: Assign scores to Zones 1, 2, and 3 as defined in Step 41, and to Zones I, II, or III as defined in Step 42. Step 463: Based on Step 462, obtain the scores for the nose key point position and the right palm key point position corresponding to the non-driving posture during autonomous driving. Step 464: Conduct a security assessment based on the total score from Steps 461 and 463.
2. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 1, characterized in that... The specific steps of step four are as follows: Step 4: Use the trained neural network to identify each frame of the surveillance video. When the identification result is manual driving posture, locate the key points of the current frame image to obtain the driver's activity area S and the key point position of the driver's skeleton. Then, retain the pixels of the driver's activity area S and clear the remaining pixels to zero. Finally, use the key point position of the driver's skeleton as the reference key point. When the recognition result is a non-driving posture during autonomous driving, the key point localization is performed on the current frame image to obtain the driver's activity area S and the driver's skeletal key point position. Then, the pixels of the driver's activity area S are retained and the remaining pixels are cleared to zero. Finally, the safety assessment is performed using the current driver's skeletal key point position and the reference key point closest to the current frame.
3. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 2, characterized in that... The key points of the driver's skeleton include: the key point of the nose, the key point of the right shoulder, the key point of the right elbow, the key point of the right palm, the key point of the right hip, and the key point of the right knee.
4. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 1, characterized in that... The different non-driving postures during autonomous driving include waiting to drive, turning to the side to pick up an item, operating the central screen, reading, looking at a mobile phone, using a laptop, making a phone call, and eating.
5. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 1, characterized in that... The specific steps of step two are as follows: The in-vehicle monitoring video data is broken down into multiple video segments every 2 seconds; The specific steps for constructing the sample set are as follows: Under each category, each video segment is used as an independent sample. Frames of a 2-second continuous video are extracted, resized to 128×171 and saved, and converted into images with coherent poses. Each independent sample contains at least 16 images, thus constructing a sample set.
6. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 1, characterized in that... The keypoint localization is performed using a lightweight YOLOv8 keypoint localization algorithm.
7. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 1, characterized in that... The neural network is a lightweight three-dimensional convolutional neural network.
8. The non-driving posture recognition and safety assessment method based on a lightweight architecture according to claim 7, characterized in that... The loss function of the lightweight 3D convolutional neural network is expressed as: Where p and q are probability distributions, For the desired output, This is the actual output.