Driving reminder method and apparatus

Abstract

The application discloses a driving reminding method and device, and relates to the technical field of driving data processing, and comprises the following steps: acquiring driving environment data and multi-modal data comprising vehicle operation parameters and user driving operation data in at least one dimension during user driving of a vehicle; performing driving behavior recognition based on the multi-modal data to obtain user driving behavior data; determining a driving risk level of the user according to the multi-modal data, the driving behavior data and the driving environment data; acquiring and executing a response strategy corresponding to the driving risk level to remind the user to drive, so that flexible and effective driving reminding is achieved.

Description

Technical Field

[0001] This application relates to the field of driving data processing technology, and in particular to a driving reminder method and device. Background Technology

[0002] With the development and popularization of vehicles, more and more users are choosing to travel by car, which provides great convenience and greatly improves the ease of travel. However, in the road traffic environment, traffic accidents not only waste users' time but also their resources. According to relevant data research, dangerous driving behavior is the main cause of traffic accidents. In order to reduce dangerous driving behavior and thus traffic accidents, warnings about dangerous driving behavior can be issued. However, how to improve the effectiveness of dangerous driving behavior identification, as well as the effectiveness and flexibility of warnings about dangerous driving behavior, is a key focus for both users and vehicle providers. Summary of the Invention

[0003] In view of the above problems, this application provides a driving reminder method and device that overcomes or at least partially solves the above problems, and the technical solution is as follows: A driving alert method includes: acquiring multimodal data and driving environment data during a user's driving process; the multimodal data includes vehicle operating parameters and the user's driving operation data in at least one dimension; performing driving behavior recognition based on the multimodal data to obtain the user's driving behavior data; determining the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data; acquiring a response strategy corresponding to the driving risk level, and executing the response strategy to provide a driving alert to the user.

[0004] The driving reminder method provided in this application acquires driving environment data and multimodal data, including vehicle operating parameters and user driving operation data, during the user's driving process. This improves the comprehensiveness of the acquired data by introducing multimodal data, avoiding the impact of relying solely on single-dimensional data for driving behavior recognition on the accuracy of the acquired driving behavior data. Furthermore, since driving environments, vehicle operating conditions, and user driving operations may differ, introducing driving environment data enhances the effectiveness of subsequent multimodal data processing. Based on the acquired driving environment data and multimodal data, driving behavior recognition is first performed using the multimodal data to obtain the user's driving behavior data. This approach, starting from multimodal data, enhances the effectiveness of driving behavior data obtained from diverse multimodal data. Furthermore, based on the multimodal data, driving behavior data, and driving environment data, the user's driving risk level is determined. Then, a corresponding response strategy is executed to provide driving reminders to the user. This process, combining multimodal data, driving behavior data, and driving environment data to further determine the driving risk level, improves the effectiveness of the determined risk level. By executing the corresponding response strategy, driving reminders are provided to the user, enhancing the effectiveness and accuracy of driving reminders. Moreover, different response strategies are used for different driving risk levels, increasing the flexibility of driving reminders.

[0005] Optionally, the step of recognizing driving behavior based on the multimodal data to obtain the user's driving behavior data includes: constructing a relationship structure graph between the user and the vehicle based on the multimodal data; extracting features from each node in the relationship structure graph to obtain node features of each node; and inputting the node features into a fully connected layer for index prediction processing of each driving behavior to obtain the predicted index of each driving behavior of the user as the driving behavior data.

[0006] In this optional implementation, during the process of obtaining user driving behavior data through multimodal data-based driving behavior recognition, since the multimodal data includes vehicle operating parameters and user driving operation data, a user-vehicle relationship structure graph can be constructed first based on the multimodal data. Then, features are extracted from each node in the relationship structure graph to obtain node features. Finally, the node features are input into a fully connected layer for indicator prediction processing of each driving behavior, obtaining the predicted indicators of each user's driving behavior as driving behavior data. In this way, by constructing a relationship structure graph and extracting features from each node, the extracted node features have both temporal and spatial dimensions, thereby improving the effectiveness and accuracy of indicator prediction processing, and thus improving the effectiveness and accuracy of the obtained predicted indicators of each driving behavior, that is, improving the effectiveness and accuracy of the obtained driving behavior data.

[0007] Optionally, the relational structure graph is generated using the acquisition time of each sensor in the multimodal data as the time step; the nodes in the relational structure graph include sub-data in the multimodal data; and nodes in the relational structure graph that have spatial and / or temporal relationships are connected by edges.

[0008] In this optional implementation, the nodes in the relational structure graph may include sub-data from the multimodal data. Nodes with spatial and / or temporal relationships are connected by edges. Thus, nodes are determined from the sub-data in the multimodal data, and the connecting edges are determined according to the spatial and / or temporal relationships between the sub-data, thereby obtaining a relational structure graph composed of nodes and connecting edges between nodes. This improves the effectiveness of the generated relational structure graph and the degree of representation of the sub-data and the relationships between the sub-data in the multimodal data.

[0009] Optionally, determining the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data includes: calculating a first risk score based on the multimodal data and the driving environment data, and calculating a second risk score based on the driving behavior data; the first risk score characterizes the data risk level of the multimodal data in the driving environment corresponding to the driving environment data; the second risk score characterizes the behavioral risk level of the driving behavior; and the driving risk level is determined based on the first risk score and the second risk score.

[0010] In this optional implementation, in determining a user's driving risk level, on the one hand, a first risk score is calculated based on multimodal data and driving environment data to characterize the degree of data risk in the multimodal data under the driving environment; on the other hand, a second risk score is calculated based on driving behavior data to characterize the degree of behavioral risk in driving behavior; and then the driving risk level is determined based on the first risk score and the second risk score. In this way, the driving risk level is determined from two dimensions: the degree of data risk and the degree of behavioral risk, thereby improving the effectiveness of the determined driving risk level.

[0011] Optionally, calculating the first risk score based on the multimodal data and the driving environment data includes: reading the mapped parameter allocation weights from the weight mapping data based on the driving environment data; storing the mapping relationship between the driving environment data and the parameter allocation weights in the weight mapping data; obtaining the parameter values ​​of preset parameters in the multimodal data; and calculating the first risk score based on the parameter values ​​and the parameter allocation weights.

[0012] In this optional implementation, during the calculation of the first risk score based on multimodal data and driving environment data, the parameter allocation weights are first read from the weighted mapping data based on the driving environment data. Then, the parameter values ​​of preset parameters in the multimodal data are obtained, and the first risk score is calculated based on the parameter values ​​and parameter allocation weights. In this way, the parameter allocation weights are determined through the driving environment data, and the first risk score is calculated through the parameter values ​​and parameter allocation weights. This ensures that the impact of the driving environment on the user's driving is fully considered during the calculation of the first risk score, which represents the data risk level of multimodal data, thereby improving the matching degree between the obtained first risk score and the driving environment.

[0013] Optionally, the step of calculating the first risk score based on the parameter value and the parameter allocation weight includes: if the parameter value of the target preset parameter in the multimodal data is found to be abnormal, then obtaining at least one associated parameter of the target preset parameter; allocating the target allocation weight corresponding to the target preset parameter to the at least one associated parameter to obtain the target parameter allocation weight; and calculating the first risk score based on the parameter value and the target parameter allocation weight.

[0014] In this optional implementation, if the parameter value of the target preset parameter in the multimodal data is abnormal during the calculation of the first risk score based on the parameter value and parameter allocation weight, the target allocation weight corresponding to the target preset parameter is allocated to the associated parameter of the preset parameter. The first risk score is then calculated based on the target parameter allocation weight obtained after allocating the parameter value and weight of the preset parameter other than the target preset parameter. In this way, the corresponding target allocation weight is allocated to the associated parameter in the case of abnormal parameter value, ensuring the stability of the target parameter allocation weight, thereby reducing the impact of abnormal parameter value on the first risk score and improving the accuracy of the calculated first risk score.

[0015] Optionally, calculating the second risk score based on the driving behavior data includes: obtaining a preset weight for each risky driving behavior and a prediction index for each risky driving behavior in the driving behavior data; and calculating the second risk score based on the preset weight and prediction index for each risky driving behavior.

[0016] In this optional implementation, preset weights for each driving behavior can be configured. In the process of calculating the degree of behavioral risk for characterizing driving behavior, a second risk score can be calculated based on the preset weights of each driving behavior and prediction indicators. In this way, in the process of calculating the second risk score, prediction indicators of multiple driving behaviors are fully integrated, thereby improving the degree of characterization of the degree of behavioral risk of driving behavior by the calculated second risk score.

[0017] Optionally, the method is applied to an in-vehicle terminal; the step of performing driving behavior recognition based on the multimodal data to obtain the user's driving behavior data includes: inputting the multimodal data into a behavior recognition model to perform driving behavior recognition, and obtaining the driving behavior data output by the behavior recognition model; the behavior recognition model is deployed on the in-vehicle terminal.

[0018] In this optional implementation, the driving reminder method can be applied to the vehicle terminal. In the process of obtaining the user's driving behavior data by performing driving behavior recognition based on multimodal data, the driving behavior data is obtained by inputting the multimodal data into the behavior recognition model deployed on the vehicle terminal. This eliminates the need for the vehicle terminal to obtain the data, transmit it to the server, and then process the driving behavior data on the server side, which would otherwise result in a long recognition time. This improves the efficiency of driving behavior recognition. Furthermore, by using the behavior recognition model to perform driving behavior recognition, the efficiency of driving behavior recognition is further improved.

[0019] Optionally, the method further includes: uploading the multimodal data and the driving behavior data to a server via an encrypted communication channel; the server training a baseline recognition model based on the multimodal data and the driving behavior data to obtain an updated recognition model, and generating model update data based on the updated recognition model and the baseline recognition model; and obtaining the model update data sent by the server through the encrypted communication channel to update the behavior recognition model deployed on the vehicle terminal based on the model update data.

[0020] In this optional implementation, multimodal data and driving behavior data are uploaded to the server to train the model. Then, the behavior recognition model deployed on the vehicle terminal is updated based on the updated recognition model obtained from the model training, thereby improving the recognition capability of the behavior recognition model.

[0021] Optionally, after the step of performing driving behavior recognition based on the multimodal data to obtain the user's driving behavior data is executed, the method further includes: detecting whether the prediction index of risky driving behavior in the driving behavior data is greater than a preset threshold; if so, performing the step of determining the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data.

[0022] In this optional implementation, if the predicted index of risky driving behavior in the acquired driving behavior data is greater than a preset threshold, then the driving risk level is determined and the corresponding response strategy is executed. If the predicted index of risky driving behavior is less than or equal to the preset threshold, then no processing is performed. In this way, driving reminders are issued when the predicted index of the user's risky driving behavior is greater than the preset threshold, thereby improving the effectiveness of driving reminders and avoiding the waste of computing resources caused by determining the driving risk level when the user does not have risky driving behavior.

[0023] A driving reminder device, the device comprising: The acquisition module is used to acquire multimodal data and driving environment data during the user's driving process; the multimodal data includes vehicle operating parameters and the user's driving operation data in at least one dimension; The behavior recognition module is used to perform driving behavior recognition based on the multimodal data to obtain the user's driving behavior data; The risk level determination module is used to determine the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data. The response execution module is used to obtain the response strategy corresponding to the driving risk level and execute the response strategy to provide driving reminders to the user.

[0024] A vehicle that includes the driving reminder device as described above.

[0025] An electronic device includes: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described driving reminder methods.

[0026] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described driving reminder methods.

[0027] A computer program product includes a computer program that, when executed by a processor, implements the steps of any of the above-described driving reminder methods.

[0028] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0029] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is an illustrative flowchart of a driving reminder method provided in an embodiment of this application. Figure 1 ; Figure 2 This is a schematic flowchart illustrating the driving behavior recognition process of a behavior recognition model provided in an embodiment of this application; Figure 3 This is a schematic flowchart illustrating a process for determining a driving risk level, as provided in an embodiment of this application. Figure 4 This is an illustrative flowchart of a driving reminder method provided in an embodiment of this application. Figure 2 ; Figure 5 This is a schematic structural diagram of a driving reminder device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0031] In practical applications, traffic accidents caused by improper driving behavior account for a large proportion of all traffic accidents, among which distracted driving or fatigued driving accounts for a significant proportion. Based on this, cameras can be used to collect images of the driver's eyes, and fatigued driving can be identified based on these images. However, identifying fatigued driving solely based on eye images cannot identify other non-eye signals, resulting in an incomplete identification of fatigued or distracted driving. Furthermore, single-eye image recognition is easily affected by lighting conditions, occlusion, and low light at night, impacting the accuracy of the identification. To address this, the driving reminder method provided in this embodiment acquires multimodal data, including vehicle operating parameters and user driving operation data in at least one dimension, during the user's driving process. It then performs driving behavior recognition based on the multimodal data, thereby improving the effectiveness of driving behavior data obtained from more comprehensive multimodal data recognition while enhancing the comprehensiveness of the data. In practice, once driving behavior data is obtained, driving reminders can be given to users based on this data. However, to avoid the limitations of relying solely on driving behavior data for reminders, this embodiment provides a driving reminder method that, after obtaining the user's driving behavior data, also determines the user's driving risk level based on multimodal data, driving behavior data, and driving environment data. Driving reminders are then given based on this risk level. By combining multimodal data, driving behavior data, and driving environment data to determine the user's driving risk level, the effectiveness of the determined risk level is improved. Furthermore, during the process of giving driving reminders based on the risk level, a response strategy corresponding to that risk level can be executed. This allows for different response strategies to be applied to users with different driving risk levels, thereby improving the effectiveness and flexibility of the driving reminders.

[0032] like Figure 1 As shown, Figure 1 This is an illustrative flowchart of a driving reminder method provided in an embodiment of this application. Figure 1 The method includes: Step 101: Obtain multimodal data and driving environment data during the user's driving process.

[0033] The multimodal data in this embodiment can be used to characterize user-related information and vehicle-related information during the user's driving process. Optionally, the multimodal data includes vehicle operating parameters and user driving operation information in at least one dimension.

[0034] In the specific execution process, vehicle operating parameters and user driving operation information in at least one dimension can be collected through at least one sensor deployed inside the vehicle. Among them, the sensor can be an image acquisition component for acquiring images of the user, an infrared tracking module for infrared tracking, a torque sensor for acquiring torque information of the user, a pressure sensor array for acquiring the user's sitting posture pressure, or a bus interface for acquiring vehicle operating parameters. This embodiment does not limit the specific sensor.

[0035] To improve the efficiency and effectiveness of data acquisition from various sensors, the image acquisition component can be deployed in the center of the top of the cockpit, the infrared tracking module can be integrated below the instrument panel, the torque sensor can be embedded inside the steering wheel, and the pressure sensor array can be placed under the driver's seat. Specifically, a high-resolution visible light camera can be installed in the center of the top of the cockpit to continuously capture facial image sequences of the driver. A high-resolution visible light camera, for example, 1080P@60fps (1920p resolution) can be used. A 1080-pixel camera (capturing 60 frames per second); an infrared eye-tracking module can be integrated under the dashboard to emit invisible near-infrared light and receive corneal reflections, combining pupil focus changes to calculate the left side of the fixation point and eyelid closure frequency; a high-precision torque sensor can be embedded inside the steering wheel to acquire the amplitude and rate of change of the steering torque applied by the user in real time; the high-precision torque sensor is, for example, a torque sensor with a measurement accuracy within ±0.1 Nm; 8... An 8-pressure sensor array is used to construct a human sitting posture pressure heat map at a sampling frequency of 50 Hz; in addition, ECU (Electronic Control Unit) data frames can be obtained through the vehicle bus interface to obtain vehicle speed, steering angle, brake pedal pressure, gear status, engine speed and / or braking frequency as vehicle operating parameters.

[0036] Correspondingly, user driving operation data in at least one dimension may include at least one of the following: facial image sequence, corneal reflection point, steering torque data, and seating pressure data. Vehicle operating parameters may include at least one of the following: vehicle speed, steering angle, brake pedal pressure, gear status, engine speed, braking frequency, and seat belt status.

[0037] In addition, user driving operation data in at least one dimension may include at least one of the following: facial features, eyelid closure frequency, three-dimensional head posture, hand grip force changes, pupil position, eyelid opening and closing degree, weight shift, and sitting posture. Specifically, facial features can be obtained through feature extraction from facial image sequences captured by a high-resolution visible light camera; eyelid closure frequency and eyelid opening and closing degree can be calculated based on facial image sequences and corneal reflection points; three-dimensional head posture can be obtained through facial image sequences; hand grip force changes can be determined through steering torque data; pupil position can be calculated based on facial image sequences and corneal reflection points; and weight shift and sitting posture can be obtained based on sitting pressure data.

[0038] It should be noted that the user's driving operation data in at least one dimension may include at least one of the following: facial image sequence, corneal reflection point, steering torque data, sitting pressure data, facial features, eyelid closure frequency, three-dimensional head posture, hand grip force change, pupil position, eyelid opening and closing degree, weight shift, and sitting posture.

[0039] In the specific execution process, when acquiring multimodal data, the data collected by at least one sensor can be acquired first, and then the driving operation data can be determined based on the acquired data.

[0040] In practice, in order to ensure that the driving operation data of at least one dimension can be matched on the time axis, during the process of acquiring multimodal data or data collected by at least one sensor, all sensors can be time-aligned with the embedded timestamp module through hardware trigger signals, so that the driving operation data and / or vehicle operating parameters of each dimension in the acquired driving operation data of at least one dimension are aligned in time.

[0041] For example, all sensors are time-aligned with GPS (Global Positioning System) protocol / PTP (Precision Time Protocol) via PPS (Pulse Per Second).

[0042] Furthermore, the multimodal data obtained above can be initial multimodal data. Data preprocessing can be performed on the initial multimodal data to obtain multimodal data. For example, initial multimodal data can be obtained, and the initial multimodal data can be preprocessed to obtain multimodal data.

[0043] Specifically, in the process of preprocessing the initial multimodal data, the initial multimodal data can be segmented according to a preset time window to obtain at least one time window of data. Data normalization, coordinate calibration and / or signal denoising are then performed on the data of each time window to obtain multimodal data.

[0044] Specifically, after obtaining at least one time window of data, the time window data can be temporarily cached to allow the initial multimodal data at different rates to be aligned and integrated on the same time dimension, avoiding data corruption. Then, data normalization, coordinate calibration and / or signal denoising are performed on each time window of data to obtain multimodal data.

[0045] In addition to acquiring multimodal data during the user's driving process, driving environment data can also be acquired. In this embodiment, driving environment data includes data describing the environment in which the vehicle is located, such as time of day and / or weather; for example, the time of day in the driving environment data is daytime and the weather is sunny; or, the time of day in the driving environment data is nighttime and the weather is rainy. In addition, driving environment data may also include vehicle speed.

[0046] The time period can be determined based on the vehicle clock and GPS positioning. For example, if the vehicle clock is from 6:00 to 18:00, the time period can be determined to be daytime; if the vehicle clock is from 18:00 to 6:00, the time period can be determined to be nighttime.

[0047] Weather conditions can be obtained through rain gauges, visibility sensors, and / or meteorological data interfaces. For example, if a rain gauge detects rainfall data, the weather is determined to be rainy; if the rain gauge does not detect rainfall and the visibility sensor collects visibility greater than a visibility threshold, the weather is determined to be sunny; if the rain gauge does not detect rainfall and the visibility sensor collects visibility less than or equal to a visibility threshold, the weather is determined to be foggy; the visibility threshold can be 500 meters.

[0048] In addition, weather data can be obtained by calling the meteorological data interface.

[0049] The above sections have provided specific explanations of multimodal data and driving environment data. In this embodiment, during the driving reminder process, multimodal data and driving environment data generated during the user's driving process are first acquired.

[0050] Step 102: Perform driving behavior recognition based on multimodal data to obtain the user's driving behavior data.

[0051] In specific implementation, after acquiring multimodal data and driving environment data during the user's driving process, driving behavior recognition is first performed based on the multimodal data to obtain the user's driving behavior data. The driving behavior data in this embodiment may include driving behaviors and their prediction indicators. The prediction indicator for any driving behavior includes the predicted probability of that driving behavior occurring. Driving behavior refers to the user's description of their current state or behavior. Optionally, driving behavior includes baseline behavior, mild fatigue behavior, severe fatigue behavior, distracted behavior, agitated behavior, and / or abnormal behavior.

[0052] In some embodiments, in the process of obtaining user driving behavior data by recognizing driving behavior based on multimodal data, a relationship structure graph between the user and the vehicle can be constructed first based on the multimodal data. Then, feature extraction is performed on each node in the relationship structure graph to obtain the node features of each node. The node features are then input into a fully connected layer for index prediction processing of each driving behavior to obtain the predicted index of each user's driving behavior as driving behavior data.

[0053] Optionally, the relational structure graph is generated using the acquisition time of each sensor in the multimodal data as the time step. Nodes in the relational structure graph include sub-data in the multimodal data, and nodes with spatial and / or temporal relationships in the relational structure graph are connected by edges. In this embodiment, the relational structure graph can be constructed using the acquisition time of each sensor frame as the time step. Nodes include: facial feature nodes, eyelid closure frequency nodes, head three-dimensional posture nodes, hand grip force change nodes, pupil position nodes, eyelid opening and closing degree nodes, focus offset nodes, sitting posture nodes, and / or operating parameter nodes. Based on the above nodes, it is determined that physically related nodes within the same frame have spatial connection edges, and that nodes of the same type in adjacent frames have temporal edges. In this way, a relational structure graph composed of multimodal data is obtained, which can characterize the temporal and spatial relationships of each sub-data.

[0054] Furthermore, after obtaining the relational structure graph, features can be extracted from each node in the relational structure graph to obtain the node features of each node. It should be noted that during the feature extraction process of each node in the relational structure graph, the spatiotemporal relationship of the nodes should be fully considered.

[0055] The node features in this embodiment can be used to characterize the node's own data, the node's position in the relational structure graph, and / or its association relationships. Optionally, node features include node data features and / or node spatiotemporal relationship features. Node data features can be features characterizing the node's own data, while node spatiotemporal relationship features can be features characterizing the node's change patterns over time and / or the strength of its spatial association with adjacent nodes.

[0056] For example, the node features of a facial feature node may include the node data feature of facial key point coordinates, as well as the spatiotemporal relationship features of the node, which include the changing trends of the facial feature node and related facial feature nodes (facial feature nodes in adjacent time steps) and the association strength with related nodes (nodes that have connecting edges to the facial feature node). Among them, the association strength may include Euclidean distance.

[0057] For example, the node features of an eyelid closure frequency node may include the node data feature of the number of blinks per unit time, and may also include the spatiotemporal relationship features of the node, including the duration of the number of blinks per unit time and the correlation strength between the eyelid closure frequency node and related nodes. The duration of the number of blinks per unit time can be obtained by analyzing the eyelid closure frequency node and related eyelid closure frequency nodes.

[0058] For example, the node data features in the node features of the pupil position node may include pupil coordinates and / or gaze angle, and the spatiotemporal relationship features of the node may include the trend of pupil coordinate changes and / or the correlation strength between the pupil position node and related nodes.

[0059] It should be noted that the above description of node features is merely exemplary. In actual execution, specific node features can be configured and extracted according to actual needs, and this embodiment does not impose any limitations on them.

[0060] Based on the node features obtained, the node features can be input into the fully connected layer for indicator prediction processing of various driving behaviors, and the predicted indicators of each user's driving behavior can be obtained as driving behavior data.

[0061] It should be noted that the process of recognizing driving behavior based on multimodal data input to a behavior recognition model and obtaining the driving behavior data output by the behavior recognition model can be performed through the behavior recognition model itself to improve the efficiency and convenience of driving behavior recognition. In an optional implementation provided in this embodiment, during the process of obtaining user driving behavior data based on multimodal data, the multimodal data is input to a behavior recognition model for driving behavior recognition, and the driving behavior data output by the behavior recognition model is obtained, thereby improving the efficiency of driving behavior recognition.

[0062] The behavior recognition model in this embodiment can be a Spatio-Temporal Graph Neural Network (ST-GNN) model. The ST-GNN model may include a structure graph construction layer, a lightweight graph convolutional layer, a temporal modeling layer, and a fully connected layer.

[0063] like Figure 2 As shown, Figure 2 This is a schematic flowchart illustrating the driving behavior recognition process of a behavior recognition model provided in an embodiment of this application. The process includes: Step 201: Construct a relational structure graph based on multimodal data through a structure graph construction layer.

[0064] In practice, the input to the structure graph construction layer can be multimodal data, which can be heterogeneous. For example, multimodal data includes the following four sub-data categories: facial key points, i.e., facial features; eye movement trajectories (pupil position and / or eyelid closure frequency); hand pressure (changes in hand grip strength); and vehicle operating parameters.

[0065] After acquiring multimodal data, the structure graph construction layer generates connecting edges based on the spatiotemporal adjacency relationships of the sub-data corresponding to each node. Then, using each sub-data in the multimodal data as a node, it constructs a relational structure graph based on these connecting edges. In this way, unstructured heterogeneous sensor data, i.e., multimodal data, is transformed into spatiotemporal graph structured data (i.e., a relational structure graph) that can be processed by the behavior recognition model. This enables structured modeling of the spatial and temporal features of the multimodal data, resulting in the relational structure graph.

[0066] It should be noted that, during the process of inputting multimodal data into the structure graph construction layer, multimodal data can be input into the structure graph construction layer according to time steps.

[0067] Step 202: Extract features based on the relational structure graph using a lightweight graph convolutional layer to obtain the first feature vector.

[0068] In this embodiment, the lightweight graph convolutional layer can adopt a layered graph convolutional structure, with each layer using separable convolution and channel pruning; for example, the lightweight graph convolutional layer includes a first GCN (Graph Convolutional Network) and a second GCN; wherein, the first GCN is used to map the node data in the relational graph to a second feature vector; the second GCN is used to compress the second feature vector into a first feature vector.

[0069] The first feature vector includes a feature vector of the first dimension, such as a 32-dimensional feature vector, and the second feature vector includes a feature vector of the second dimension, such as a 64-dimensional feature vector. For example, the first-layer GCN maps the node data in the relational graph to a 64-dimensional feature vector, and the second-layer GCN compresses the 64-dimensional feature vector into a 32-dimensional feature vector.

[0070] It should be noted that the lightweight graph convolutional layers employ adjacency matrix normalization and residual connections, ensuring that the number of parameters in each layer is less than the first preset parameter count. Furthermore, an attention mechanism can be introduced to dynamically weight edge weights, enhancing the contribution of key nodes. The first feature vector retains both the fusion information of spatial features and the temporal sequence of the time steps, matching the input requirements of subsequent temporal modeling layers. The first preset parameter count can be 1.2 megabytes.

[0071] Step 203: Perform feature fusion on the first feature vector through the temporal modeling layer to obtain the target feature vector.

[0072] In this embodiment, the temporal modeling layer can be an LSTM (Long Short-Term Memory) network. The temporal modeling layer can be a single-layer bidirectional LSTM with 48 hidden units. It can suppress redundant temporal information through a gating mechanism and prevent overfitting by setting Dropout (rate=0.3). To achieve a lightweight behavior recognition model, the number of parameters in this temporal modeling layer can be less than the second preset number of parameters; where the second preset number of parameters can be 0.7 megabytes.

[0073] Specifically, in the process of feature fusion of the first feature vector, the first feature vector can be fused into a single-dimensional first-dimensional feature vector to extract the temporal evolution pattern of driving behavior. In this way, feature fusion is performed on the first feature vector output by the first feature vector to obtain the target feature vector. Through feature fusion, the deep extraction of the first feature vector is achieved, realizing the dual fusion of spatial and temporal features.

[0074] For example, a 32-dimensional feature vector is input into the temporal modeling layer, which then merges the 32-dimensional feature vector into a single 32-dimensional feature vector, and determines the single 32-dimensional feature vector as the target feature vector.

[0075] Step 204: Based on the target feature vector, the fully connected layer performs index prediction processing for each driving behavior to obtain the predicted index of each user's driving behavior as driving behavior data.

[0076] The fully connected layer in this embodiment can be a classification and inference head that includes a softmax function.

[0077] In the specific execution process, the fully connected layer predicts the predicted index under each driving behavior based on the target feature vector, and obtains the predicted index of the user under each driving behavior as driving behavior data. In addition, the target driving behavior with the largest predicted index can also be determined as driving behavior data. This embodiment does not limit this.

[0078] For example, driving behavior includes baseline behavior, distracted behavior, fatigued behavior, impatient behavior, dangerous operation behavior, and abnormal behavior. The fully connected layer calculates the prediction index corresponding to each driving behavior based on the target feature vector, and determines each driving behavior and its corresponding prediction index as driving behavior data.

[0079] The above describes the process of driving behavior recognition using a behavior recognition model. It should also be noted that the driving processing method provided in this embodiment can be applied to in-vehicle terminals, and the aforementioned behavior recognition model can be deployed on these terminals. To avoid the behavior recognition model being too large and not meeting the deployment requirements of in-vehicle terminals, the behavior recognition model deployed on these terminals can be a lightweight model. Specifically, it can be optimized using weighted quantization INT8 (8-bit Integer), layer fusion GCN (Graph Convolutional Network) + LSTM, and TensorRT (Tensor Runtime) engine. On the hardware platform, the graph propagation operator is rewritten using CUDA (Compute Unified Device Architecture) kernel functions, and sparse matrices are used to accelerate graph convolution operations. The final model has a total of 2.08M parameters, and under 1080p input and a 30Hz frame rate, the average inference latency reaches 67ms, meeting automotive-grade real-time requirements.

[0080] Based on the aforementioned driving behavior recognition model deployed through the vehicle-mounted terminal, to improve the driving behavior recognition capability of the model, it can be updated. In one optional implementation of this embodiment, during the update process, the vehicle-mounted terminal first uploads multimodal data and driving behavior data to the server via an encrypted communication channel. This encrypted communication channel enhances data transmission security. After receiving the multimodal data and driving behavior data, the server trains the baseline recognition model based on the data to obtain an updated recognition model. Then, based on the updated and baseline recognition models, it generates model update data and sends it to the vehicle-mounted terminal. The vehicle-mounted terminal receives the model update data sent by the server via the encrypted communication channel and updates the behavior recognition model deployed on the terminal based on this data.

[0081] Optionally, the model update data includes a model incremental update package, which may contain weight change parameters. In this way, the model incremental update package is sent instead of the entire updated recognition model, thereby improving data transmission efficiency and thus improving the update efficiency of the behavior recognition model.

[0082] In practice, the vehicle terminal can upload multimodal data and driving behavior data to the server via an encrypted communication channel at preset intervals (every hour or every 100 kilometers traveled) for model training. Besides uploading multimodal data and driving behavior data, time and / or driving environment data, as well as other data, can also be uploaded; this embodiment does not limit the scope of the upload. It should be noted that the uploaded multimodal data and driving behavior data can be anonymized to protect user data privacy.

[0083] After receiving multimodal data and driving behavior data, the server can incrementally train the baseline recognition model through a federated learning framework to obtain an updated recognition model. During the incremental training process, the multimodal data and driving behavior data, as well as historical multimodal data and historical driving behavior data, can be merged, and differential privacy technology can be used to add noise to prevent reverse inference of users. Furthermore, the training can adopt adaptive learning rate and online batch normalization to cope with data distribution drift.

[0084] After the server obtains the updated recognition model through training, it generates an incremental update package containing the weight change parameters and sends it to the vehicle terminal. The vehicle terminal adopts a hot-loading mechanism to update the behavior recognition model without affecting real-time inference.

[0085] It should be noted that once the next preset cycle is reached, the server can update the recognition model to the baseline recognition model for model training.

[0086] In addition to constructing a relational structure graph and obtaining driving behavior data from it, in the process of obtaining user driving behavior data by recognizing driving behavior based on multimodal data, one can first calculate at least one dimension of behavioral indicator features based on multimodal data, then input the at least one dimension of behavioral indicator features into a behavior classification model for behavior classification processing, and obtain the driving behavior data output by the behavior classification model.

[0087] Specifically, in the process of calculating at least one dimension of behavioral indicator features based on multimodal data, a fatigue index can be calculated based on facial features and corneal reflective points in the multimodal data; a grip strength and posture coupling index can be constructed based on steering torque data and sitting posture pressure data; and / or, a driving operation stability index can be calculated based on vehicle operating parameters. The fatigue index, grip strength and posture coupling index, and / or driving operation stability index are then identified as behavioral indicator features. The specific calculation process for each of the three behavioral indicator features is explained below.

[0088] (1) Calculate the fatigue index based on facial features and corneal reflection points in multimodal data.

[0089] In practice, during the calculation of the fatigue index based on facial features and corneal reflection points in multimodal data, at least one fatigue sub-index can be calculated first based on the facial features and corneal reflection points, and then the fatigue index can be calculated based on the at least one fatigue sub-index. Optionally, the fatigue sub-index may include eyelid closure duration, eyelid closure frequency, and / or gaze deviation angle.

[0090] The duration of eyelid closure includes the percentage of time the eyelid is closed per unit time. Specifically, in calculating the duration of eyelid closure, the frame intervals of complete and / or partial eyelid closure can be identified based on the continuous data of corneal reflection points in multimodal data to obtain the cumulative closure duration. The ratio of the cumulative closure duration to the unit time is then calculated to obtain the duration of eyelid closure.

[0091] Eyelid closure frequency includes the number of complete blinks per unit time. Specifically, in calculating eyelid closure frequency, blinking behavior can first be identified based on facial features and / or corneal reflex points, and the number of blinks per unit time can be calculated as the eyelid closure frequency.

[0092] The fixation deviation angle includes the angle between the driver's pupil fixation point and the visual center reference, with the vehicle's direction of travel as the visual center reference. Specifically, in calculating the fixation deviation angle, the visual center is first determined, then the coordinates of the fixation point are calculated based on the corneal reflectance, and finally the deviation angle is calculated based on the coordinates of the visual center and the fixation point as the fixation deviation angle.

[0093] After obtaining the eyelid closure duration, eyelid closure frequency, and / or gaze deviation angle, the eyelid closure duration, eyelid closure frequency, and / or gaze deviation angle can be normalized first, and then the fatigue index can be calculated by assigning weights according to the pre-configured sub-indices.

[0094] For example, if the pre-configured sub-index weights are 5:3:2, and the calculated eyelid closure duration is a, eyelid closure frequency is b, and fixation deviation angle is c, then the fatigue index is... .

[0095] (2) Construct a grip force and sitting posture coupling index based on steering torque data and sitting posture pressure data.

[0096] In practice, when constructing the grip strength and sitting posture coupling index based on steering torque data and sitting posture pressure data, the steering torque data and sitting posture pressure data can be extracted separately to obtain grip strength features and sitting posture features. Then, the grip strength features and sitting posture features can be fused to obtain the grip strength and sitting posture coupling index.

[0097] Specifically, in the process of extracting features from steering torque data to obtain grip strength features, the user's pre-stored reference torque amplitude can be obtained. The ratio between the reference torque amplitude and the torque amplitude in the steering torque data is calculated, and the grip strength type is identified as the grip strength feature based on the ratio.

[0098] For example, the ratio of the torque amplitude to the reference torque amplitude is calculated. If the ratio is less than the ratio threshold of 0.5, the grip strength type is determined to be low grip strength; if the ratio is greater than or equal to the ratio threshold, it is determined to be effective grip strength.

[0099] In addition, the system can detect whether the single-hand torque amplitude is abnormal (the rate of change is greater than a first speed threshold or less than a second speed threshold). If so, the grip strength characteristic is determined to include a sudden drop in single-hand torque amplitude; otherwise, no action is taken. Alternatively, the torque change rate can be calculated based on the torque amplitude, and grip strength loss of control behavior can be determined as a grip strength characteristic based on the torque change rate. Specifically, if the torque change rate is greater than the change rate threshold, grip strength loss of control is determined. Optionally, the first speed threshold is greater than the second speed threshold. The first speed threshold can be 5, and the second speed threshold can be 0.2. The change rate threshold can be 0.2.

[0100] (3) Calculate driving operation stability index based on vehicle operating parameters.

[0101] In practice, during the calculation of driving operation stability indicators based on vehicle operating parameters, at least one driving operation sub-indicator can be calculated first based on the vehicle operating parameters, and then the driving operation stability indicator can be calculated based on the at least one driving operation sub-indicator. Optionally, the driving operation indicators include the standard deviation of lateral acceleration and / or the steering reversal frequency. The standard deviation of lateral acceleration can be calculated by taking the square root of the sum of the squared deviations of the average lateral acceleration within a preset time window from each real-time value. The steering reversal frequency includes the number of times the user reverses the steering wheel per unit time.

[0102] In the process of calculating the driving operation stability index based on at least one driving operation sub-index, each driving operation sub-index can first be normalized, and then the at least one normalized driving operation sub-index can be weighted and summed to obtain the driving operation stability index.

[0103] For example, if the pre-configured driving operation sub-indicators are weighted at 5:5, the standard deviation of lateral acceleration is d, and the steering reversal frequency is e, then the driving operation stability index is: .

[0104] The above provides a detailed explanation of the calculation process for the three behavioral indicators. It should be noted that the fatigue index is used to characterize eye fatigue characteristics, the grip strength and posture coupling index is used to supplement non-eye fatigue indicators and solve the problem of fatigue omission caused by relying solely on eye recognition, and the driving operation stability index is used to infer the user's physiological and / or psychological state through the vehicle operation status.

[0105] Based on the aforementioned behavioral indicator features, these features can be input into a classification model for behavior classification, and the driving behavior data output by the classification model can be obtained. The classification model can be a multi-layer perceptron (MLP) classifier to improve the accuracy of the obtained driving behavior data.

[0106] In practice, to improve the effectiveness of driving reminders, a driving reminder is issued when the predicted index of risky driving behavior in the user's driving behavior data is greater than a preset threshold. If the predicted index of risky driving behavior in the driving behavior data is less than or equal to the preset threshold, it means that the user does not have dangerous driving behavior and there is no need to issue a driving reminder. In this case, no action needs to be taken.

[0107] In some embodiments, after obtaining the user's driving behavior data, it is detected whether the predictive index of risky driving behavior in the driving behavior data is greater than a preset threshold; if so, step 103 is executed to determine the user's driving risk level based on multimodal data, driving behavior data, and driving environment data; if not, no processing is required. The preset threshold can be 0.5, or it can be configured according to the actual scenario; this embodiment does not limit this.

[0108] Step 103: Determine the user's driving risk level based on multimodal data, driving behavior data, and driving environment data.

[0109] In practice, after acquiring multimodal data, driving behavior data, and driving environment data, the user's driving risk level is determined. In this embodiment, the driving risk level characterizes the user's perception of the vehicle's driving risk; specifically, the driving risk level is directly proportional to the actual driving risk.

[0110] Specifically, the process of determining a user's driving risk level based on multimodal data, driving behavior data, and driving environment data can be achieved through the following operations: Step 103-1: Calculate the first risk score based on multimodal data and driving environment data.

[0111] The first risk score in this embodiment can be used to characterize the data risk level of multimodal data in the driving environment corresponding to the driving environment data.

[0112] In practical implementation, during the calculation of the first risk score based on multimodal data and driving environment data, the parameter allocation weights can first be read from the weight mapping data based on the driving environment data. Then, the parameter values ​​of preset parameters in the multimodal data are obtained, and the first risk score is calculated based on the parameter values ​​and parameter allocation weights. In this way, by combining driving environment data, the risk characterization of multimodal data is determined, improving the adaptability of the obtained first risk score to the driving environment, i.e., improving the effectiveness of the first risk score. Optionally, the weight mapping data can be used to store the mapping relationship between driving environment data and parameter allocation weights. For the weight mapping relationship, for nighttime environments, the weight of facial features can be reduced, while the weight of eye movement and seat pressure can be increased; for high-speed environments, the weight of steering and seat belt status can be increased. The seat belt status can include both connected and disconnected states.

[0113] In the specific execution process, when retrieving the mapped parameters and assigning weights from the weight mapping table based on the driving environment data, the driving environment can first be determined based on the driving environment data, and then the parameter assignment weights mapped to the driving environment can be retrieved from the weight mapping table. Optionally, the weight mapping data can be used to store the mapping relationship between the driving environment and the parameter assignment weights.

[0114] Specifically, in the process of determining the driving environment based on driving environment data, the speed environment can be determined based on the vehicle speed contained in the driving environment data, the time period environment can be determined based on the time period contained in the driving environment data, and / or the weather environment can be determined based on the weather contained in the driving environment data. Then, the speed environment, time period environment and / or weather environment are combined to obtain the driving environment.

[0115] For example, if the vehicle speed is less than or equal to 60 km / h, the speed environment is determined to be low speed; if the vehicle speed is greater than 60 km / h but less than or equal to 100 km / h, the speed environment is determined to be medium speed; and if the vehicle speed is greater than 100 km / h, the speed environment is determined to be high speed. Furthermore, the speed environment, time period environment, and / or weather environment can be combined into a driving environment, and then the parameters for assigning weights to the driving environment mapping can be read from the weighted mapping data.

[0116] Specifically, the mapping relationship between the driving environment and parameter allocation weights can be shown in Table 1 below.

[0117]

[0118] Table 1 It should be noted that the driving environment and parameter labels and weights shown in Table 1 are exemplary and can be configured according to the actual scenario. This embodiment does not limit them here.

[0119] In practice, after reading the parameter allocation weights from the weighted mapping data based on the driving environment data, the first risk score can be calculated based on the parameter allocation weights and the parameter values ​​of preset parameters in the multimodal data. Specifically, in the process of calculating the first risk score based on the parameter allocation weights and the parameter values ​​of preset parameters in the multimodal data, the parameter values ​​of each preset parameter and the allocation weights can be weighted and fused to obtain the first risk score.

[0120] For example, the first risk score can be calculated as follows:

[0121] in, Assign weights to the i-th preset parameter. Let be the parameter value of the i-th preset parameter; where the parameter value of the i-th preset parameter can be its normalized parameter value.

[0122] In specific implementation, in order to reduce the impact of abnormal parameter values ​​on the first risk score during the calculation of the first risk score based on parameter values ​​and parameter allocation weights, in an optional implementation of this embodiment, if the parameter value of the target preset parameter in the multimodal mode is found to be abnormal, at least one associated parameter of the target preset parameter is obtained, the target allocation weight corresponding to the target preset parameter is allocated to at least one associated parameter to obtain the target parameter allocation weight, and the first risk score is calculated based on the parameter value and the target parameter allocation weight.

[0123] In the specific execution process, the associated parameters of each preset parameter can be pre-configured, or the associated parameters of the preset parameters can be parameters that have a temporal and / or spatial relationship with the preset parameters. In the process of allocating the target allocation weight corresponding to the target preset parameter to at least one associated parameter, the target allocation weight can be evenly distributed among at least one associated parameter. Furthermore, if there is a connection weight between the preset parameter and the associated parameter in the relationship structure diagram, the target allocation weight can be evenly distributed among at least one associated parameter according to the connection weight between the associated parameter and the target allocation weight. This embodiment does not impose any limitations on this. For example, as shown in Table 1, in the nighttime + low-speed scenario, facial feature anomalies are obtained from multimodal data. The associated parameters of facial features are determined to include eye movement weight and seat pressure weight. The facial feature weight is then evenly distributed to the eye movement weight and seat pressure weight. The target parameter allocation weights obtained after the allocation are: facial feature weight is 0, eye movement weight is 0.5, seat pressure weight is 0.4, steering weight is 0.05, and seat belt status weight is 0.05.

[0124] Step 103-2: Calculate the second risk score based on driving behavior data.

[0125] The second risk score in this embodiment can be used to characterize the degree of behavioral risk of driving behavior.

[0126] In practice, preset weights can be configured for each risky driving behavior in advance. During the calculation of the second risk score based on driving behavior data, the preset weights for each risky driving behavior and the predictive indicators for each risky driving behavior in the driving behavior data can be obtained first. Then, the second risk score is calculated based on the preset weights and predictive indicators for each risky driving behavior. In this embodiment, risky driving behaviors may include driving behaviors other than the baseline driving behavior.

[0127] For example, risky driving behaviors include fatigue, distraction, and impatience; the pre-configured weights are: 0.4 for fatigue, 0.3 for distraction, and 0.3 for impatience; the second risk score is obtained by weighting and summing the pre-configured weights and prediction indicators for each risky driving behavior.

[0128] It should be noted that the above description of the preset weights is merely exemplary. The specific behaviors and preset weights included in risky driving behaviors can be configured according to actual needs, and this embodiment does not limit them here.

[0129] Step 103-3: Determine the driving risk level based on the first risk score and the second risk score.

[0130] Based on the calculation of the first risk score and the second risk score, the driving risk level can be determined.

[0131] Specifically, in the process of determining the driving risk level based on the first risk score and the second risk score, the target risk score can be calculated first based on the first risk score and the second risk score, and then the risk level corresponding to the target risk score can be determined as the driving risk level.

[0132] In calculating the target risk score, the sum or weighted sum of the first risk score and the second risk score can be used as the target risk score. For example, the product of the first risk score and 0.5 and the product of the second risk score and 0.5 can be used as the target risk score. Alternatively, the average of the first risk score and the second risk score can be used as the target risk score, but this embodiment does not limit this.

[0133] In addition to calculating the target risk score based on the first risk score and the second risk score mentioned above, the first risk score or the second risk score can also be used as the target risk score.

[0134] In practice, the correspondence between risk scores and risk levels can be pre-configured. For example, if the target risk level is greater than or equal to 98, the driving risk level is determined to be level four; if the target risk level is greater than or equal to 95 and less than 98, the driving risk level is determined to be level three; if the target wind level is greater than or equal to 90 and less than 95, the driving risk level is determined to be level two; and if the target wind level is greater than or equal to 85 and less than 90, the driving risk level is determined to be level one.

[0135] like Figure 3 As shown, Figure 3 This is a schematic flowchart illustrating a process for determining a driving risk level, as provided in an embodiment of this application. The process includes: Step 301: Read the mapping parameters from the weight mapping data based on the driving environment data and assign weights.

[0136] Step 302: Obtain the parameter values ​​of preset parameters in the multimodal data.

[0137] Step 303: If an abnormal value of the target preset parameter is detected, the parameter allocation weight is updated to obtain the target parameter allocation weight.

[0138] Optionally, the target allocation weight corresponding to the target preset parameter in the target parameter allocation weight is 0.

[0139] Specifically, the update process for parameter allocation weights involves assigning the target allocation weight corresponding to the target preset parameter to at least one associated parameter of the target preset parameter to obtain the target parameter allocation weight.

[0140] Step 304: Calculate the first risk score by assigning weights based on the preset parameter values ​​and the target parameter.

[0141] Step 305: Obtain the preset weights of each risky driving behavior and the predictive indicators of each risky driving behavior in the driving behavior.

[0142] Step 306: Calculate the second risk score based on the preset weights and prediction indicators of each risky driving behavior.

[0143] It should be noted that the execution order of steps 301 to 304 and steps 305 to 306 is not limited in this embodiment.

[0144] Step 307: Calculate the target risk score based on the first risk score and the second risk score.

[0145] Step 308: Determine the risk level corresponding to the target risk score as the driving risk level.

[0146] Step 104: Obtain the response strategy corresponding to the driving risk level and execute the response strategy to provide driving reminders to the user.

[0147] In practice, different response strategies can be configured for different driving risk levels. By executing the response strategy corresponding to the driving risk level, driving reminders can be given to users, thereby improving the flexibility of driving reminders and enhancing users' perception of driving reminders.

[0148] For example, the response strategy corresponding to the first-level risk level could be: play a voice reminder “Please concentrate” through the vehicle audio system; in the process of implementing this response strategy, a TTS (Text-to-Speech) engine can be used to generate natural speech to reduce latency; The response strategy corresponding to the Level 2 risk level can be: applying slight tactile feedback through the linear vibration motor built into the steering wheel; when this response strategy is executed, it can be driven by the PWM (Pulse Width Modulation) signal controlled by CAN FD (Controller Area Network Flexible Data-Rate) instructions. The response strategy corresponding to the Level 3 risk level can be: to limit the vehicle speed to the limit speed of 80 km / h through ECU (Electronic Control Unit) speed control, and prohibit speeding acceleration; this response strategy can be executed by the powertrain controller. The response strategy corresponding to the Level 4 risk level can be: seat belt warning, emergency hazard lights and / or voice reminder; specifically, the seat belt retractor's built-in motor can tighten to a preset tension of 600 Newtons within 200 milliseconds, all turn signals can be activated through the Body Control Module (BCM), the navigation system can push voice and visual prompts "It is recommended to pull over and rest", and the nearest service area or safe parking spot can be searched through the navigation application.

[0149] Specifically, during the execution of the response strategy, execution instructions can be sent to the execution unit corresponding to the response strategy to perform the response; for example, sending a "please concentrate" broadcast instruction to the vehicle audio system, or sending a steering wheel vibration instruction to the PWM, or sending a speed limit instruction to the ECU, or sending a turn signal activation instruction and / or seat belt tightening instruction to the BCM.

[0150] It should be noted that during the execution of the response strategy, all instructions are transmitted via the CAN FD bus with frame IDs: 0x1F0~0x1F3, corresponding to the response strategies for risk levels one through four, respectively. For example, the response strategy identified as 0x1F0 corresponds to the first-level risk level, the response strategy identified as 0x1F1 corresponds to the second-level risk level, the response strategy identified as 0x1F2 corresponds to the third-level risk level, and the response strategy identified as 0x1F3 corresponds to the fourth-level risk level. A timestamp synchronization mechanism can be used to ensure that the delay between the instruction and the execution unit is less than 20 milliseconds. Each instruction can also be accompanied by a CRC (Cyclic Redundancy Check) verification and retransmission mechanism to ensure that the instruction can be effectively delivered in an electromagnetic interference environment. Specific execution units, such as BCM, ECU, and / or vehicle audio system, can be configured with a status feedback loop. After the response is executed, an ACK (Acknowledgement) signal is sent to the vehicle terminal to form a closed-loop confirmation.

[0151] In practice, all trigger events can be recorded locally, including timestamps, multimodal data, driving risk levels, and / or execution status. If any execution unit fails, the vehicle terminal can automatically downgrade to the next lower level of response and trigger the instrument panel malfunction indicator light alarm. In addition, self-checks can be performed at preset intervals, such as every 24 hours. Self-checks can include sensor calibration, bus access health detection, and execution unit detection, thereby enabling effective driving reminders.

[0152] like Figure 4 As shown, Figure 4 This is an illustrative flowchart of a driving reminder method provided in an embodiment of this application. Figure 2 The method includes the following steps: Step 401: Obtain multimodal data and driving environment data during the user's driving process.

[0153] Step 402: Input the multimodal data into the behavior recognition model to perform driving behavior recognition and obtain driving behavior data.

[0154] Step 403: Detect whether the predictive index of risky driving behavior in the driving behavior data is greater than a preset threshold; If so, proceed to steps 404 to 406; If not, no action needs to be taken.

[0155] Step 404: Calculate the first risk score based on multimodal data and driving environment data, and calculate the second risk score based on driving behavior data.

[0156] Step 405: Determine the driving risk level based on the first risk score and the second risk score.

[0157] Step 406: Obtain and execute the response strategy corresponding to the driving risk level to provide driving reminders to the user.

[0158] It should be noted that any one or more of steps 401 to 406 can be combined with any one or more of steps 101 to 104 to form a new implementation method according to the needs of implementation and deployment. In addition, any one or more technical features in any of steps 401 to 406 can be selected and combined with any one or more technical features provided in steps 101 to 104 to form a new implementation method according to the actual deployment needs. Alternatively, any one or more technical features in steps 401 to 406 can be replaced with any one or more technical features provided in steps 101 to 104 to form a new implementation method according to the actual deployment needs. These will not be elaborated on here.

[0159] like Figure 5 As shown, Figure 5 A schematic structural diagram of a driving reminder device provided for embodiments of this application, the device comprising: The acquisition module 501 is used to acquire multimodal data and driving environment data during the user's driving process; the multimodal data includes vehicle operating parameters and the user's driving operation data in at least one dimension; The behavior recognition module 502 is used to perform driving behavior recognition based on the multimodal data to obtain the user's driving behavior data; The rating determination module 503 is used to determine the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data; The response execution module 504 is used to obtain the response strategy corresponding to the driving risk level and execute the response strategy to provide driving reminders to the user.

[0160] Optionally, the behavior recognition module 502 is specifically used for: constructing a relationship structure graph between the user and the vehicle based on the multimodal data; extracting features from each node in the relationship structure graph to obtain node features of each node; inputting the node features into a fully connected layer for index prediction processing of each driving behavior to obtain the predicted index of each driving behavior of the user as the driving behavior data.

[0161] Optionally, the relational structure graph is generated using the acquisition time of each sensor in the multimodal data as the time step; the nodes in the relational structure graph include sub-data in the multimodal data; and nodes in the relational structure graph that have spatial and / or temporal relationships are connected by edges.

[0162] Optionally, the level determination module 503 is specifically used to: calculate a first risk score based on the multimodal data and the driving environment data, and calculate a second risk score based on the driving behavior data; the first risk score is used to characterize the data risk level of the multimodal data in the driving environment corresponding to the driving environment data; the second risk score is used to characterize the behavioral risk level of the driving behavior; and determine the driving risk level based on the first risk score and the second risk score.

[0163] Optionally, when calculating the first risk score based on the multimodal data and the driving environment data, the level determination module 503 is specifically used to: read the mapped parameter allocation weights from the weight mapping data based on the driving environment data; store the mapping relationship between the driving environment data and the parameter allocation weights in the weight mapping data; obtain the parameter values ​​of preset parameters in the multimodal data; and calculate the first risk score based on the parameter values ​​and the parameter allocation weights.

[0164] Optionally, the level determination module 503, when calculating the first risk score based on the parameter value and the parameter allocation weight, is specifically used for: if the parameter value of the target preset parameter in the multimodal data is abnormal, then obtaining at least one associated parameter of the target preset parameter; allocating the target allocation weight corresponding to the target preset parameter to the at least one associated parameter to obtain the target parameter allocation weight; and calculating the first risk score based on the parameter value and the target parameter allocation weight.

[0165] Optionally, the level determination module 503, when calculating the second risk score based on the driving behavior data, is specifically used to: obtain the preset weights of each risky driving behavior and the prediction indicators of each risky driving behavior in the driving behavior data; and calculate the second risk score based on the preset weights and prediction indicators of each risky driving behavior.

[0166] Optionally, the device operates on the vehicle terminal; the behavior recognition module 502 is specifically used to: input the multimodal data into the behavior recognition model for driving behavior recognition, and obtain the driving behavior data output by the behavior recognition model; the behavior recognition model is deployed on the vehicle terminal.

[0167] Optionally, the device is further configured to: upload the multimodal data and the driving behavior data to a server via an encrypted communication channel; the server trains a baseline recognition model based on the multimodal data and the driving behavior data to obtain an updated recognition model, and generates model update data based on the updated recognition model and the baseline recognition model; and acquires the model update data sent by the server via the encrypted communication channel to update the behavior recognition model deployed on the vehicle terminal based on the model update data.

[0168] Optionally, the device is further configured to: detect whether the predicted index of risky driving behavior in the driving behavior data is greater than a preset threshold; if so, perform the operation of determining the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data.

[0169] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0170] Figure 6 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application.

[0171] For example, such as Figure 6 As shown, the vehicle includes a memory 601 and a processor 602. The memory 601 stores executable program code 6011, and the processor 602 is used to call and execute the executable program code 6011 to perform a driving reminder method.

[0172] This embodiment can divide the vehicle into functional modules based on the above method example. For example, each function can be assigned to a separate module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. When dividing each functional module according to its corresponding function, the vehicle may include: an acquisition module, a behavior recognition module, a level determination module, and a response execution module, etc. It should be noted that all relevant content of each step involved in the above method embodiment can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0173] The vehicle provided in this embodiment is used to execute the driving reminder method described above, and thus can achieve the same effect as the above implementation method.

[0174] When using integrated units, the vehicle may include a processing module and a storage module. The processing module is used to control and manage the vehicle's movements. The storage module is used to support the vehicle in executing relevant program code and data.

[0175] The processing module may be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0176] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described driving alert method embodiments.

[0177] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described driving reminder method embodiments when it is run.

[0178] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0179] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described driving reminder method embodiments.

[0180] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described driving reminder method embodiments.

[0181] The beneficial effects of the above embodiments can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0182] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0183] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0184] In the description of this application, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0185] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0186] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A driving reminder method, characterized in that, The method includes: Acquire multimodal data and driving environment data during the user's driving process; the multimodal data includes vehicle operating parameters and the user's driving operation data in at least one dimension; Based on the multimodal data, driving behavior recognition is performed to obtain the user's driving behavior data; The user's driving risk level is determined based on the multimodal data, the driving behavior data, and the driving environment data. Obtain the response strategy corresponding to the driving risk level, and execute the response strategy to provide driving reminders to the user.

2. The method according to claim 1, characterized in that, The process of recognizing driving behavior based on the multimodal data to obtain the user's driving behavior data includes: A relationship structure diagram between the user and the vehicle is constructed based on the multimodal data; Feature extraction is performed on each node in the relational structure graph to obtain the node features of each node; The node features are input into a fully connected layer for index prediction processing of each driving behavior, and the predicted indexes of each driving behavior of the user are obtained as the driving behavior data.

3. The method according to claim 2, characterized in that, The relational structure graph is generated using the acquisition time of each sensor in the multimodal data as the time step; the nodes in the relational structure graph include sub-data in the multimodal data; nodes in the relational structure graph that have spatial and / or temporal relationships are connected by edges.

4. The method according to claim 1, characterized in that, Determining the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data includes: A first risk score is calculated based on the multimodal data and the driving environment data, and a second risk score is calculated based on the driving behavior data; the first risk score is used to characterize the data risk level of the multimodal data in the driving environment corresponding to the driving environment data; the second risk score is used to characterize the behavioral risk level of the driving behavior. The driving risk level is determined based on the first risk score and the second risk score.

5. The method according to claim 4, characterized in that, The calculation of the first risk score based on the multimodal data and the driving environment data includes: The weights are assigned to the parameters mapped in the weight mapping data based on the driving environment data; the weight mapping data stores the mapping relationship between the driving environment data and the parameter allocation weights. Obtain the parameter values ​​of preset parameters from the multimodal data, and calculate the first risk score based on the parameter values ​​and the weights assigned to the parameters.

6. The method according to claim 5, characterized in that, The calculation of the first risk score based on the parameter values ​​and the weights assigned to the parameters includes: If the parameter value of the target preset parameter in the multimodal data is found to be abnormal, then at least one associated parameter of the target preset parameter is obtained. The target allocation weights corresponding to the target preset parameters are assigned to the at least one associated parameter to obtain the target parameter allocation weights; The first risk score is calculated based on the parameter values ​​and the weights assigned to the target parameters.

7. The method according to claim 4, characterized in that, The calculation of the second risk score based on the driving behavior data includes: Obtain the preset weights of each risky driving behavior and the prediction indicators of each risky driving behavior in the driving behavior data; The second risk score is calculated based on the preset weights and prediction indicators of each risky driving behavior.

8. The method according to claim 1, characterized in that, The method is applied to an in-vehicle terminal; the step of recognizing driving behavior based on the multimodal data to obtain the user's driving behavior data includes: The multimodal data is input into the behavior recognition model to perform driving behavior recognition, and the driving behavior data output by the behavior recognition model is obtained; the behavior recognition model is deployed on the vehicle terminal.

9. The method according to claim 8, characterized in that, The method further includes: The multimodal data and the driving behavior data are uploaded to the server through an encrypted communication channel; the server trains the baseline recognition model based on the multimodal data and the driving behavior data to obtain an updated recognition model, and generates model update data based on the updated recognition model and the baseline recognition model. The model update data sent by the server through the encrypted communication channel is obtained, and the behavior recognition model deployed on the vehicle terminal is updated based on the model update data.

10. The method according to claim 1, characterized in that, After the operation of recognizing driving behavior based on the multimodal data to obtain the user's driving behavior data is executed, the method further includes: Detect whether the predictive index of risky driving behavior in the driving behavior data is greater than a preset threshold; If so, perform the operation of determining the user's driving risk level based on the multimodal data, the driving behavior data, and the driving environment data.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the driving reminder method as described in any one of claims 1 to 10.

Images