Vehicle-mounted intelligent monitoring method and system based on multimode communication and behavior recognition

Abstract

The invention discloses a vehicle-mounted intelligent monitoring method and system based on multi-mode communication and behavior recognition, and relates to the related field of vehicle-mounted electronics and computer vision, the method adopts multi-mode communication to upload the acquired network signal intensity, vehicle driving state and cab real-time image and video data to the system, and the system is used for monitoring the behavior of the vehicle. Network signal strength changes are predicted according to historical data, and a multi-factor decision strategy is made to automatically switch communication networks; a pre-training target detection model adaptive to a night light environment is applied to position a face area of a driver, face key points are obtained through key point detection, dynamic fatigue characteristics of the driver are tracked in a cross-frame mode in combination with a target tracking algorithm, the eye state, the head posture and the yawning behavior of the driver are analyzed, and the fatigue index of the driver is calculated. And the vehicle driving safety is guaranteed through layered early warning. The problems that an existing vehicle-mounted intelligent monitoring system is single in communication mode, and a fatigue driving detection model embedded in the system cannot accurately position the face area of a driver are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle-mounted electronics and computer vision, in particular to a vehicle-mounted intelligent monitoring method and system based on multi-mode communication and behavior recognition. BACKGROUND

[0002] Vehicle-mounted intelligent monitoring refers to a system that uses devices and technologies installed on vehicles to monitor and analyze the driving state of the vehicle, the behavior of the driver, and the environment around the vehicle in real time. By monitoring the behavior of the driver and the environment around the vehicle in real time, potential risks are detected in a timely manner and warnings are issued to effectively prevent traffic accidents; at the same time, the monitoring system records video data during the driving process of the vehicle to provide accurate evidence for accident investigation and facilitate the quick handling of accident disputes.

[0003] The existing vehicle-mounted monitoring devices have a single communication mode, and when the vehicle travels to tunnel areas and remote areas, signal interruption often occurs, affecting real-time data transmission. In addition, the traditional fatigue driving detection method used in vehicle-mounted intelligent monitoring systems mainly includes methods based on vehicle driving state, driver physiological characteristics, and computer vision. The method based on vehicle driving state requires a large number of professional equipment to collect various vehicle driving information, which is costly and relies on subjective factors, affecting the accuracy of detection. The method based on physiological characteristics directly measures the brain, heart, and muscle signals of the driver, which can easily interfere with normal driving and pose a risk of accidents.

[0004] Fatigue driving detection based on computer vision is a commonly used method at present, which extracts the behavior characteristics of the driver through a neural network to identify the fatigue state. However, the existing models cannot adapt to night lighting conditions and complex driving environments, the accuracy of driver face positioning is insufficient, and the identification of fatigue state lacks comprehensive feature analysis, making it impossible to achieve real-time tracking detection. SUMMARY

[0005] To solve the technical problems of the above-mentioned prior art, the present application provides a vehicle-mounted intelligent monitoring method and system based on multi-mode communication and behavior recognition, which realizes efficient data transmission through multi-mode communication and automatically switches communication modes by integrating multiple factors. Through an improved target detection model, the driver's face area is accurately positioned, dynamic fatigue features are identified in combination with a target tracking algorithm, the driver's eye state, head posture, and yawning behavior are analyzed, and a fatigue index is calculated to make a layered warning.

[0006] The present application provides a vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition, comprising: (1) acquiring network signal strength data, vehicle driving state data, and real-time image and video data of the cab through vehicle-mounted data acquisition equipment inside the vehicle; (2) Using multi-mode communication, the data collected by the vehicle data acquisition device is transmitted to various modules of the vehicle intelligent monitoring system, and the network signal strength is predicted based on historical data. Multi-factor decision-making strategies are formulated to automatically switch the communication network and realize real-time data transmission. (3) Apply the pre-trained target detection model adapted to nighttime lighting environment to analyze and process the real-time image of the cab, locate the driver's facial area, and obtain facial key points through key point detection. (4) Based on the detection results of the driver's facial key points, the driver's cab video data is processed by Deepsort target tracking algorithm, the driver's dynamic fatigue characteristics are tracked across frames, the driver's eye state, head posture and yawning behavior are analyzed, and the driver's fatigue index is calculated in a comprehensive manner. (5) Based on the calculated driver fatigue index, establish a tiered early warning mechanism to promptly remind drivers to adjust their driving behavior and coordinate with the vehicle system and monitoring center to take emergency measures.

[0007] This application also provides a vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition, including: Data acquisition module: used to acquire network signal strength data, vehicle driving status data, and real-time images and video data from the driver's cab through onboard data acquisition equipment inside the vehicle; Multi-mode communication module: It adopts multi-mode communication to transmit the data collected by the vehicle data acquisition device to various modules in the vehicle intelligent monitoring system, and automatically switches the communication network based on the prediction of network signal strength according to historical data and formulate multi-factor decision-making strategies to achieve real-time data transmission. Driver face localization and key point extraction module: Apply a pre-trained target detection model adapted to nighttime lighting environment to analyze and process real-time images of the cab, locate the driver's facial area, and obtain facial key points through key point detection; Driver fatigue analysis module: Based on the detection results of the driver's facial key points, combined with the Deepsort target tracking algorithm, the module processes the cab video data, tracks the driver's dynamic fatigue characteristics across frames, analyzes the driver's eye state, head posture and yawning behavior, and comprehensively calculates the driver's fatigue index. Early warning module: Based on the calculated driver fatigue index, it establishes a tiered early warning mechanism to promptly remind drivers to adjust their driving behavior and coordinate with the vehicle system and monitoring center to take emergency measures.

[0008] The present invention discloses the following technical effects: This invention proposes an in-vehicle intelligent monitoring method and system based on multi-mode communication and behavior recognition. It monitors abnormal driving status of vehicles, realizes intelligent detection of driver fatigue through behavior recognition algorithm, and improves the real-time performance of data transmission by automatically switching communication modes through multi-mode communication.

[0009] When vehicles drive through tunnels and remote mountainous areas, the signal strength is weak, and a single communication method cannot meet the data transmission requirements. This invention adopts a multi-mode communication method and uses time series analysis to predict the changing trend of network signal strength. Based on the real-time data of the monitored vehicle driving status, it analyzes various factors affecting signal strength to make decisions and automatically switches communication methods to ensure that data is transmitted to various modules of the vehicle intelligent monitoring system in a timely manner, providing real-time data support for traffic monitoring and improving the effectiveness of data transmission. To reduce the probability of traffic accidents caused by driver fatigue, this invention employs an intelligent algorithm to identify driver fatigue characteristics and uses a pre-trained target detection model to accurately locate the driver's facial region. To improve the recognition accuracy in nighttime driving environments, a differentiable image processing module is added to the target detection model. This module analyzes the visual information of the driver's cab image through a neural network, adaptively learns the parameters required for the image processing method, and enhances image quality. Based on the driver's facial region localization results, key points are obtained through key point detection. The driver's eye state, head posture, and yawning behavior are analyzed, and video data is analyzed through a target tracking algorithm to obtain the driver's dynamic fatigue characteristics. This addresses the error problem caused by existing fatigue detection methods analyzing fatigue probability from limited static features, as well as the problem of unclear facial features due to excessively fast movement speed. This invention comprehensively calculates the driver's fatigue index, identifies abnormal data in vehicle driving status, and sends alerts to the driver and monitoring center through a hierarchical early warning mechanism. It fully realizes intelligent monitoring of vehicle driving and driver status, effectively prevents the probability of traffic accidents, and improves the real-time performance and accuracy of data monitoring and analysis of the vehicle monitoring system. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0011] Figure 1 This is a flowchart illustrating the vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition provided in an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of the structure of an adaptive target detection model for driver face region localization provided in an embodiment of this application.

[0013] Figure 3 This is a schematic diagram of the structure of an in-vehicle intelligent monitoring system based on multi-mode communication and behavior recognition, provided in an embodiment of this application. Detailed Implementation

[0014] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application.

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In the following description, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0017] Example 1: This application provides a vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition, such as... Figure 1 As shown, the method includes: Step S10: Obtain network signal strength data, vehicle driving status data, and real-time images and video data from the driver's cab through the vehicle's onboard data acquisition equipment.

[0018] In this embodiment, network signal strength data includes received signal strength and a reference signal reception quality index. The Received Signal Strength Index (RSSI) represents the received signal strength, measured in decibels per milliwatt (dBm); higher signal strength indicates better communication quality. The Reference Signal Quality Index (RSRQ) is a reference signal reception quality index, calculated by comprehensively analyzing signal strength and interference levels. The formula is as follows: , RSSI and RSRP are the number of frequency domain resources allocated to users within the measurement bandwidth, and RSRP is the reference signal received power. The network parameters RSSI and RSRP are collected by the network testing equipment of the vehicle terminal to evaluate the network signal strength and assign timestamps and location information to the data.

[0019] Vehicle driving status data is acquired through sensors and a positioning module, including the vehicle's real-time position and speed data; and real-time data such as engine speed, accelerator pedal position, brake status, and steering wheel angle are acquired through the vehicle bus system; the positioning module has an embedded GNSS receiver to acquire, analyze, and process the radio signals transmitted by GNSS to calculate the vehicle's real-time three-dimensional position.

[0020] A driver detection camera (DMS) is installed in the driver's cab, with an embedded AI chip containing facial region localization, key point detection, and target tracking algorithm models for driver fatigue detection. The camera's resolution and frame rate are set to collect images and video data from the driver's cab in real time, and the data is transmitted to the camera's internal memory. The monitoring and detection results are then transmitted to the cloud monitoring center via network.

[0021] Step S20: Using multi-mode communication, the data collected by the vehicle data acquisition device is transmitted to various modules in the vehicle intelligent monitoring system. Based on historical data, the network signal strength is predicted, and a multi-factor decision-making strategy is formulated to automatically switch the communication network to achieve real-time data transmission.

[0022] In this embodiment, the vehicle-mounted intelligent monitoring system integrates multiple communication methods and supports multiple communication protocols. When a system component malfunctions or a certain communication signal is not received, it predicts signal strength changes in advance and automatically switches communication networks, ensuring normal data transmission and communication. The steps for predicting future network signal strength changes using a time series analysis algorithm based on Autoregressive Integral Moving Average (ARIMA) are as follows: Based on the collected historical network signal strength data, the data is organized into a time series format with a time interval of 1 second, and the RSSI, RSRQ and RSRP values ​​of the network signal are recorded. The time series data of historical network signal strength is preprocessed to remove outliers and missing values, and the data is differentially processed to obtain stationary data, and then a time series plot is drawn. Constructing an autoregressive moving average model, combining autoregressive, differencing, and moving average algorithms, the ARIMA model is expressed in the following form:

[0023] in, Is to carry out Predicted network signal strength data after second difference It is the first To the Historical network signal strength data at any given time; ... These are the coefficients of the autoregressive term; ... It is the coefficient of the moving average term. It is a random error term; This represents the minimum difference order that makes the sequence stationary. This indicates the number of lag terms in the partial autocorrelation function graph. This represents the lag term parameter in the autocorrelation function graph; Based on the autocorrelation function and partial autocorrelation function plots of historical network signal strength time series data, determine the parameters in the ARIMA model. , and The ARIMA model was fitted using historical network signal strength data, and the coefficients of the autoregressive term and the average term in the model were estimated using the least squares method. The ARIMA model is subjected to the Ljung-Box test to check whether the model residuals have autocorrelation, and the model parameters are dynamically adjusted. After passing the test, the final ARIMA model is used to predict future network signal strength data.

[0024] The multi-factor decision-making strategy analyzes the relationship between vehicle speed, location, and signal strength, selects a suitable network based on vehicle speed and location, avoids frequent switching, selects a network with better signal coverage, and defines the rules for automatic switching of communication networks: when the network signal strength is predicted to weaken, the comprehensive score of each network switching scheme is evaluated based on the current vehicle speed and the distance of the vehicle to the nearest base station, and the communication network switching scheme with the highest score is selected.

[0025] Step S30: Apply a pre-trained target detection model adapted to nighttime lighting environment to analyze and process real-time images of the cab, locate the driver's facial region, and obtain facial key points through key point detection.

[0026] In this embodiment, driver facial region localization is a key step in realizing driver fatigue detection. Considering that driver fatigue often occurs during nighttime driving, the image quality of cab images collected under nighttime lighting conditions is limited, affecting the accuracy of facial region localization. Therefore, an adaptive image processing module is used to improve image quality during the process of localizing the facial region through the target detection model.

[0027] The detailed steps for obtaining a pre-trained target detection model adapted to nighttime lighting environments are as follows: We collected official datasets for face recognition tasks and used data augmentation methods to process the image data in the dataset to simulate scenarios in real-world driving environments where the driver's face is obscured or the facial image is unclear due to driver head movement. Rotate and translate the image to generate face images from different perspectives, simulating the driver's head rotation and movement; crop the image to generate face images of different sizes covering the main facial area, simulating the driver's different distances from the camera; use a masking matrix to randomly select a fixed-size area of ​​the face image and set the pixel values ​​within the area to zero, simulating the driver's face being obscured.

[0028] Additive noise is added to the face images in the official dataset to reduce image brightness and contrast, generating image data under nighttime lighting conditions. This data, along with the original images and the image data generated by data augmentation, serves as the dataset for training the object detection model. Rectangular boxes are labeled on all images to outline the facial regions as labels. An adaptive target detection model for nighttime lighting environments was constructed, consisting of differentiable image blocks, a backbone network, a neck network, a hypergraph generation module, and a detection network. The parameters of each part of the model were initialized, and the parameters of the image processing method in the differentiable image blocks were set empirically. The model was trained using stochastic gradient descent, and the classification and regression losses were calculated. The learnable parameters in the model were updated by backpropagation based on the loss function values. The Adam optimizer was applied, and the initial learning rate was set to 0.0001. The model performance was verified while training, and the change curve of the loss function was recorded. When the set model training period is reached, observe the convergence of the loss function, adjust the training period and model hyperparameters, and retrain until the loss function converges. Save the best-performing model parameters as a pre-trained target detection model for adaptive nighttime lighting environments, and embed it into the camera AI chip to achieve local data processing.

[0029] The camera's AI chip embeds a MediaPipe facial landmark detector. Based on the driver's facial region localization results output by the pre-trained object detection model, MediaPipe processes the driver's facial region to detect key points such as the eyes and mouth, and draws the contours of the eyes and mouth. Step S40: Based on the detection results of the driver's facial key points, the driver's cab video data is processed in conjunction with the Deepsort target tracking algorithm. The driver's dynamic fatigue characteristics are tracked across frames, and the driver's eye state, head posture and yawning behavior are analyzed to comprehensively calculate the driver's fatigue index.

[0030] In this embodiment, the driver's eye state analysis is achieved by calculating the eye aspect ratio between the height and width of the eyes. The key points of each eye include the inner corner point, outer corner point, two points on the upper eyelid, and two points on the lower eyelid. The position coordinates of each key point in the image are recorded, and the eye aspect ratio is calculated. It can be calculated using the following formula:

[0031] in, and These represent the coordinates of the outer corner and inner corner of the eye, respectively. and This represents the coordinates of two key points on the upper eyelid. and This represents the coordinates of two key points on the lower eyelid; This indicates the calculation of the Euclidean distance between two points; calculate the left eye in the driver's facial region image. The aspect ratio of the right eye The final aspect ratio of the driver's eyes was obtained. : When three consecutive frames of driver facial images At that time, it was determined that the driver was in a closed-eye state.

[0032] When performing driver head pose analysis, based on the keypoint detection results, the following keypoints of the driver's face are first obtained from the image: the tip of the nose, the chin, the inner and outer corners of the eyes, and the left and right corners of the mouth. The tip of the nose is used as the center point of the head, the chin keypoint is used to determine the vertical direction of the head, and the eye and mouth keypoints are used to determine the horizontal direction and pose of the head, respectively. Next, a standard 3D model of the head is obtained, and the 3D coordinates of the facial keypoints are extracted from the model. The keypoints of the standard 3D model and the facial keypoints detected in the image are input into the PnP algorithm to calculate the camera extrinsic parameters and obtain the rotation matrix. Finally, the Euler angles of the head are extracted from the rotation matrix. The calculation formula is as follows:

[0033] in, Represents the arctangent function. It is a rotation matrix The Line number Column elements; when the driver's face is detected in five consecutive frames. When this occurs, it is determined to be an abnormal state.

[0034] Driver yawning behavior detection is achieved by calculating the mouth's aspect ratio. Based on six key points of the mouth, the aspect ratio is calculated. The formula is as follows:

[0035] in, and These represent the coordinates of the left and right corners of the mouth, respectively. and This represents the coordinates of two key points on the upper lip. and This represents the coordinates of two key points on the lower lip; This indicates the calculation of the Euclidean distance between two points; when the driver's facial image is detected for 2 consecutive seconds... At that moment, it was determined that the driver was yawning and was showing signs of fatigue.

[0036] Step S50: Based on the calculated driver fatigue index, establish a tiered early warning mechanism to promptly remind drivers to adjust their driving behavior and coordinate with the vehicle system and monitoring center to take emergency measures.

[0037] In this embodiment, a weighted voting mechanism is used to calculate the driver fatigue index. The formula is as follows:

[0038] in, This represents the percentage of time spent with eyes closed; it is the ratio of the number of frames in the closed-eye state to the number of frames during the detection time. This indicates the frequency of yawning, counting the number of yawns within the detection period. This indicates steering wheel vibration deviation.

[0039] Based on the driver fatigue index calculated in real time, a tiered early warning mechanism is adopted: Initial warning: The system uses flashing icons on the vehicle's dashboard to alert the driver. Intermediate warning: The buzzer inside the car sounded intermittently for 30 seconds. Emergency Alarm: The system will automatically reduce the vehicle's speed and transmit the vehicle's location data to the cloud monitoring center to report the situation to the traffic police department.

[0040] Example 2: This invention provides a detailed structure of an adaptive target detection model for nighttime lighting environments used for driver face localization, as follows: Figure 2 As shown: The model adopts the YOLO basic structure, which includes a backbone network, a neck network, and a detection network. It introduces a differentiable image processing block to generate enhanced real-time images of the cab. The image processing parameters that need to be optimized in this module are obtained by the parameter prediction module through adaptive learning of the brightness, color, hue, and nighttime lighting environment-specific information of the real-time images of the cab. The parameter prediction module consists of convolutional layers and fully connected layers, making full use of the feature extraction capabilities of convolutional operations.

[0041] The differentiable image processing block consists of six differentiable filters, enabling image dehazing, white balance adjustment, gamma filtering, tone adjustment, contrast adjustment, and sharpening. The image processing is represented as multiple pixel-level filters, with the pixel mapping relationship of each filter represented by a differentiable function. The parameters to be optimized in the function include the dehazing parameters. Sharpening parameters White balance parameters of RGB channels , and Gamma filter parameters Contrast adjustment parameters and hue parameters , For parameter index.

[0042] The backbone network uses the enhanced real-time image of the driver's cab as input features. It improves the model's feature extraction capability through convolutional modules, DW_C3K2 modules, and A2K2f modules. All three modules adopt a bottleneck structure. The DW_C3K2 module uses depthwise separable convolution to reduce the number of model parameters. The A2K2f module introduces an efficient region attention mechanism to adaptively learn global features.

[0043] The neck network, located between the backbone and detection networks, employs a dynamic upsampling module, the DW_C3K2 module, and channel stitching operations to achieve feature fusion and enhancement. The dynamic upsampling module treats the upsampling process as point sampling, interpolating input features into a continuous space to generate a content-aware, resampled continuous map of sampled points. This improves the accuracy of driver facial region localization and reduces the loss of important information points caused by downsampling. First, bilinear interpolation is used to interpolate the input features into a continuous space to obtain the interpolated feature map. Then, a content-aware offset O is generated by a sampling point generator using a bilinear layer and pixel rearrangement operation. This offset, along with the original sampling grid G, forms a sampling point set S. Finally, the processed feature map is resampled using a grid sampling function based on the generated sampling point set S to obtain the final upsampled feature map, which preserves the complete facial information of the driver.

[0044] The detection network is the decision-making part of the model, responsible for generating the final detection results. It receives multi-scale feature maps from the neck network and processes bounding box regression and class prediction separately through a decoupled detection head. For each location on each feature map, the detection head generates bounding box coordinates, class probability, and confidence score. Redundant bounding boxes are removed through non-maximum suppression, and the final detection result is output.

[0045] Furthermore, a hypergraph generation module for real-time cab images is introduced outside the basic structure. This module utilizes the high-order interactions between a hyperedge and multiple vertices in the hypergraph features to capture high-order visual associations in the image, adapting to the dynamic movement of the driver in cab images under nighttime lighting conditions. The hypergraph generation module for real-time cab images employs an adaptive hyperedge generation module to adaptively learn the participation degree of each vertex and hyperedge in the hypergraph, and generates high-order association enhancement features through hypergraph convolution. Finally, a hypergraph information fusion approach is used to collect and process multi-scale features from the backbone network. , and The generated higher-order association enhancement features , and Distributed via a dedicated channel to the connection points between the backbone network and the neck network, within the neck network, and at the connection points between the neck network and the detection head.

[0046] The hypergraph generation module will adapt to the hypergraph. Defined as ,in Represents a set of vertices. The adaptive hyperedge set is represented by flattening the input features into vertex features and generating a global vertex context using global average pooling and max pooling. A global offset is generated from the vertex context through a mapping layer. The global offset is then added to a learnable global prototype to obtain a dynamic hyperedge prototype, which represents the potential visual relevance in the cockpit scene. To calculate the participation of each vertex, a linear mapping is used from vertex features. Generate vertex query vector , Linear layer weight matrix; to further enhance feature diversity, a multi-head mechanism is used to divide the query vector and hypervariable prototype into multiple subspaces along the feature dimension, at the first... In the nth subspace, calculate the nth Vertex query vector and the A prototype Similarity between :

[0047] in, This represents the feature dimension of the query vector and prototype in each subspace; therefore, the overall similarity... Defined as the average of all subspaces, the similarity between the vertex query vector and the prototype is taken as the vertex's contribution to the hyperedge, and normalized along the vertex dimension to obtain the continuous participation matrix. :

[0048] in, The feature dimension of a vertex. The index is the vertex feature dimension; the continuous participation matrix is ​​the output of the adaptive hyperedge generation module.

[0049] After generating adaptive hyperedges, hypergraph convolution is performed to achieve feature aggregation and enhancement. Specifically, in hypergraph convolution, each hyperedge first collects features from all vertices. The hyperedge features are then formed through linear projection, and subsequently backpropagated to the vertices to update their representations, as expressed by the following formula:

[0050] in, Indicates the hyperedge feature. and These represent the number of vertices and the number of hyperedges, respectively. and Represents the weights of the linear layer. This represents the activation function. and These are the vertex and hyperedge indices, respectively. The hypergraph feature representation, consisting of hyperedges and vertices, constitutes a higher-order association. , For indexing.

[0051] Example 3: The vehicle-mounted intelligent monitoring system based on multi-mode communication and behavior recognition provided in this embodiment of the invention can execute the vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. The system framework is as follows: Figure 3 As shown, it includes the following modules: Data acquisition module: used to acquire network signal strength data, vehicle driving status data, and real-time images and video data from the driver's cab through onboard data acquisition equipment inside the vehicle; Multi-mode communication module: It adopts multi-mode communication to transmit the data collected by the vehicle data acquisition device to various modules in the vehicle intelligent monitoring system, and automatically switches the communication network based on the prediction of network signal strength according to historical data and formulate multi-factor decision-making strategies to achieve real-time data transmission. Driver face localization and key point extraction module: Apply a pre-trained target detection model adapted to nighttime lighting environment to analyze and process real-time images of the cab, locate the driver's facial area, and obtain facial key points through key point detection; Driver fatigue analysis module: Based on the detection results of the driver's facial key points, combined with the Deepsort target tracking algorithm, the module processes the cab video data, tracks the driver's dynamic fatigue characteristics across frames, analyzes the driver's eye state, head posture and yawning behavior, and comprehensively calculates the driver's fatigue index. Early warning module: Based on the calculated driver fatigue index, it establishes a tiered early warning mechanism to promptly remind drivers to adjust their driving behavior and coordinate with the vehicle system and monitoring center to take emergency measures.

[0052] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0053] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition, characterized in that, The method includes: (1) Obtain network signal strength data, vehicle driving status data, and real-time images and video data of the driver’s cab through the vehicle’s on-board data acquisition equipment; (2) Using multi-mode communication, the data collected by the vehicle data acquisition device is transmitted to various modules of the vehicle intelligent monitoring system, and the network signal strength is predicted based on historical data. Multi-factor decision-making strategies are formulated to automatically switch the communication network and realize real-time data transmission. (3) Apply the pre-trained target detection model adapted to nighttime lighting environment to analyze and process the real-time image of the cab, locate the driver's facial area, and obtain facial key points through key point detection. (4) Based on the detection results of the driver's facial key points, the driver's cab video data is processed by Deepsort target tracking algorithm, the driver's dynamic fatigue characteristics are tracked across frames, the driver's eye state, head posture and yawning behavior are analyzed, and the driver's fatigue index is calculated in a comprehensive manner. (5) Based on the calculated driver fatigue index, establish a tiered early warning mechanism to remind drivers to adjust their driving behavior and coordinate with the vehicle system and monitoring center to take emergency measures.

2. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (1), the network signal strength data includes Received Signal Strength Index (RSSI) and Reference Signal Received Quality Index (RSRQ); the RSRQ calculation formula is expressed as: , It measures the amount of frequency domain resources allocated to users within the measured bandwidth, and RSRP is the reference signal received power. The network test equipment of the vehicle terminal collects RSSI and RSRP network parameters, evaluates the network signal strength, and assigns timestamps and location information to the data. The vehicle driving status data is acquired through sensors and a positioning module, including the vehicle's real-time position and speed data; and real-time data on the vehicle's engine speed, accelerator pedal position, brake status, and steering wheel angle are acquired through the vehicle bus system. The positioning module has an embedded GNSS receiver to acquire, analyze, and process the radio signals transmitted by the GNSS, and calculate the vehicle's real-time three-dimensional position.

3. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (2), the time series prediction algorithm based on autoregressive integral moving average (ARIMA) is used to predict the changes in future network signal strength data. The steps are as follows: Based on the collected historical network signal strength data, the data is organized into a time series format with a time interval of 1 second, and the RSSI, RSRQ and RSRP values ​​of the network signal are recorded. The time series data of historical network signal strength is preprocessed to remove outliers and missing values, and the data is differentially processed to obtain stationary data, and then a time series plot is drawn. ARIMA model is constructed, and the model is fitted based on the autocorrelation function and partial autocorrelation function graphs of historical network signal intensity time series data. The parameters in the model are estimated and determined by the least squares method. The ARIMA model is subjected to the Ljung-Box test to check whether the model residuals have autocorrelation, and the model parameters are dynamically adjusted. After passing the test, the final ARIMA model is used to predict future network signal strength data.

4. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (3), the pre-trained target detection model adapted to nighttime lighting environment is obtained through the following steps: We collected official datasets for face recognition tasks and used data augmentation methods to process the image data in the dataset to simulate scenarios in a real driver's cab environment where the driver's face is obscured or the facial image is unclear due to driver head movement. Rotate and translate the image to generate face images from different perspectives, simulating the driver's head rotation and movement; crop the image to generate face images of different sizes covering the main facial area, simulating the driver's different distances from the camera; use a masking matrix to randomly select a fixed-size area of ​​the face image and set the pixel values ​​within the area to zero, simulating the driver's face being obscured. Image data under nighttime lighting conditions is generated by reducing image brightness and contrast. The original images and image data generated by data augmentation are integrated as the dataset for training the model, and all images are labeled. Build an adaptive target detection model for nighttime lighting environments and initialize parameters; train the model and calculate the loss function; calculate gradient backpropagation based on the loss function value to update the learnable parameters in the model; when the loss function converges, save the best-performing model parameters as the pre-trained target detection model for adaptive nighttime lighting environments.

5. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 4, characterized in that, The target detection model for adaptive nighttime lighting environments includes a backbone network, a neck network, and a detection network, and generates an enhanced real-time image of the driver's cab as the final input feature through a differentiable image processing block. The image processing parameters that need to be optimized in the differentiable image processing block are obtained by the parameter prediction module through adaptive learning of the brightness, color, hue and nighttime lighting environment-specific information of the real-time image of the cab. The backbone network uses a hypergraph generation module for real-time images of the cab to capture high-order visual associations and adapt to the dynamic movement of the driver in the cab images under nighttime lighting conditions. The hypergraph generation module for real-time images of the cab utilizes an adaptive hyperedge generation module to adaptively learn the participation of each vertex and each hyperedge in the hypergraph, and generates high-order associations to enhance features through hypergraph convolution. By employing a hypergraph information fusion approach, multi-scale features are collected and processed from the backbone network. The generated high-order association enhancement features are then distributed to the backbone network and the neck network through a dedicated channel, achieving fine-grained information flow and representation collaboration across the entire model. The dynamic upsampling module in the neck network interpolates the input features into a continuous space, generating a content-aware sampling point resampled continuous map that preserves complete facial visual feature information of the driver.

6. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (4), the driver's eye state analysis is achieved by calculating the eye aspect ratio between the height and width of the eyes. The key points of each eye of the driver include the inner corner of the eye, the outer corner of the eye, two points on the upper eyelid, and two points on the lower eyelid. The position coordinates of each key point in the image are recorded, and the eye aspect ratio is calculated. It can be calculated using the following formula: in, and These represent the coordinates of the outer corner and inner corner of the eye, respectively. and This represents the coordinates of two key points on the upper eyelid. and This represents the coordinates of two key points on the lower eyelid; This indicates the calculation of the Euclidean distance between two points; calculate the left eye in the driver's facial region image. The aspect ratio of the right eye The final aspect ratio of the driver's eyes was obtained. , When three consecutive frames of driver facial images At that time, it was determined that the driver was in a closed-eye state.

7. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (4), the driver's head posture analysis first obtains the following key points of the driver's face from the image: the tip of the nose, the chin, the inner and outer corners of the eyes, and the left and right corners of the mouth. The key point of the nose is used as the center point of the head, the key point of the chin is used to determine the vertical direction of the head, and the key points of the eyes and the key points of the mouth are used to determine the horizontal direction and posture of the head, respectively. Secondly, a standard 3D model of the human head is obtained, and the 3D coordinates of facial key points are extracted from the model. The key points of the standard 3D model and the facial key points obtained from the image are input into the PnP algorithm to calculate the camera extrinsic parameters and obtain the rotation matrix. Finally, the Euler angles of the head are extracted from the rotation matrix. The calculation formula is as follows: in, Represents the arctangent function. It is a rotation matrix The Line number Column elements; When the driver's facial image is detected in five consecutive frames When this occurs, it is determined to be an abnormal state.

8. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (4), the driver's yawning behavior detection is achieved by calculating the mouth's aspect ratio. Based on six key points of the mouth, the mouth's aspect ratio is calculated. The formula is as follows: in, and These represent the coordinates of the left and right corners of the mouth, respectively. and This represents the coordinates of two key points on the upper lip. and This represents the coordinates of two key points on the lower lip; This indicates the calculation of the Euclidean distance between two points; when the driver's facial image is detected for 2 consecutive seconds... At that moment, it was determined that the driver was yawning and was showing signs of fatigue.

9. The vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in claim 1, characterized in that, In step (5), a weighted voting mechanism is used to calculate the driver fatigue index. The formula is as follows: in, This represents the percentage of time spent with eyes closed; it is the ratio of the number of frames in the closed-eye state to the number of frames during the detection time. Indicates the Euler angles of the head. This indicates the frequency of yawning, counting the number of yawns within the detection period. Indicates steering wheel vibration deviation; Based on the driver fatigue index calculated in real time, a tiered early warning mechanism is adopted: Initial warning: The system uses flashing icons on the vehicle's dashboard to alert the driver. Intermediate warning: The buzzer inside the car sounded intermittently for 30 seconds. Emergency Alarm: The system will automatically reduce the vehicle's speed and transmit the vehicle's location data to the cloud monitoring center to report the situation to the traffic police department.

10. A vehicle-mounted intelligent monitoring system based on multi-mode communication and behavior recognition, characterized in that: The system is used to implement the vehicle-mounted intelligent monitoring method based on multi-mode communication and behavior recognition as described in any one of claims 1-9, and the system comprises: Data acquisition module: used to acquire network signal strength data, vehicle driving status data, and real-time images and video data from the driver's cab through onboard data acquisition equipment inside the vehicle; Multi-mode communication module: It adopts multi-mode communication to transmit the data collected by the vehicle data acquisition device to various modules in the vehicle intelligent monitoring system, and automatically switches the communication network based on the prediction of network signal strength according to historical data and formulate multi-factor decision-making strategies to achieve real-time data transmission. Driver face localization and key point extraction module: Apply a pre-trained target detection model adapted to nighttime lighting environment to analyze and process real-time images of the cab, locate the driver's facial area, and obtain facial key points through key point detection; Driver fatigue analysis module: Based on the detection results of the driver's facial key points, combined with the Deepsort target tracking algorithm, the module processes the cab video data, tracks the driver's dynamic fatigue characteristics across frames, analyzes the driver's eye state, head posture and yawning behavior, and comprehensively calculates the driver's fatigue index. Early warning module: Based on the calculated driver fatigue index, it establishes a tiered early warning mechanism to promptly remind drivers to adjust their driving behavior and coordinate with the vehicle system and monitoring center to take emergency measures.

Images