A voice and visual interaction control method for safe driving

Abstract

This invention discloses a safe driving voice and visual interaction control method. The method includes: synchronously collecting and preprocessing the driver's visual data and voice interaction data; extracting key visual features based on the preprocessed visual data and calculating visual state feature values; triggering standardized voice interaction based on the visual state feature values, calculating a voice activity score in conjunction with the preprocessed voice interaction data, and determining the driver's voice response delay level based on the voice activity score; matching based on a predefined 3D virtual guide action sequence according to the voice response delay level, triggering a linkage response after matching to form a non-intrusive driving reminder with light, sound, and shape linkage; and completing closed-loop feedback control based on the execution state of the 3D virtual guide action sequence and the non-intrusive driving reminder with light, sound, and shape linkage. The method provided by this invention can reduce the monitoring misjudgment rate and achieve non-intrusive safety reminders.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a safe driving voice-visual interaction control method. BACKGROUND

[0002] With the popularization of vehicle intelligent driving technology, driver distraction and fatigue driving have become the main inducement of road traffic accidents. Real-time monitoring and safety reminders of the driver's state are the core link to ensure road safety. The current mainstream driving state monitoring technology mostly uses single visual monitoring or simple voice prompt scheme, which has high monitoring misjudgment rate, strong intrusive reminding method, poor interactive adaptability and other problems, and is difficult to meet the high-precision, low-interference monitoring needs of safe driving.

[0003] The above-mentioned prior art generally has the following defects: first, the monitoring misjudgment rate is high, the single visual mode is easily disturbed by the driver's eye features (such as small eyes), wearing sunglasses, changes in light environment, etc., and frequent fatigue / distraction false alarms, which even affect the UBI insurance declaration results; second, the reminding method is intrusive, and traditional sound, vibration and other interruptive reminders are mostly used, which easily interfere with the normal driving operation of the driver; third, the interaction has no adaptive closed loop, 3D interaction, voice prompt, light reminder and other modules are independent of each other, cannot be linked according to the driver's state classification, and have no complete closed loop of "state monitoring-classified reminder-response feedback"; fourth, there is no redundancy checking mechanism, and no visual and voice dual-channel judgment logic is constructed, so that the single mode misjudgment directly triggers the reminder, further reducing the monitoring reliability. SUMMARY

[0004] In view of the above-mentioned deficiencies of the current intelligent driving technology, the present application provides a safe driving voice-visual interaction control method, which can reduce the monitoring misjudgment rate and realize non-intrusive safety reminders.

[0005] To achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions:

[0006] A safe driving voice-visual interaction control method, which is realized based on the hardware architecture of a core computing unit, a display interaction unit, a visual perception unit, a voice interaction unit and a power execution unit, comprising:

[0007] Synchronously collecting visual data and voice interaction data of the driver and pre-processing them;

[0008] Extracting key visual features based on the pre-processed visual data, and calculating visual state feature values;

[0009] Triggering standardized voice interaction based on the visual state feature values, calculating a voice activity score in combination with the pre-processed voice interaction data, and determining the driver's voice response delay level according to the voice activity score;

[0010] Based on the voice response delay level, a matching is performed on a predefined 3D virtual guide action sequence. Once the matching is completed, a linkage response is triggered to form a non-intrusive driving reminder with light, sound, and shape linkage.

[0011] Based on the execution status of the 3D virtual guide body's action sequence and the light, sound, and shape-linked non-intrusive driving reminder, closed-loop feedback control is completed.

[0012] According to one aspect of the present invention, the simultaneous acquisition and preprocessing of the driver's visual data and voice interaction data specifically comprises: the core computing unit simultaneously acquiring and preprocessing the visual data and voice interaction data; performing lens distortion correction, face region ROI extraction and illumination normalization preprocessing sequentially on the visual data; and performing vehicle environment noise reduction, acoustic echo cancellation and voice endpoint detection preprocessing sequentially on the voice interaction data.

[0013] According to one aspect of the present invention, the step of extracting key visual features based on preprocessed visual data and calculating visual state feature values ​​specifically includes:

[0014] Visual feature points are selected based on the preprocessed visual data, and then labeled and normalized.

[0015] Key eye visual feature points are selected from the normalized visual feature points, and the area of ​​the eye region is calculated.

[0016] Key torso visual feature points are selected from the normalized visual feature points, and the body offset angle is calculated.

[0017] Visual state feature values ​​are obtained based on the area of ​​the eye region and the body offset angle.

[0018] According to one aspect of the present invention, in the extraction of key visual features based on preprocessed visual data, the extracted key visual features are 14 preset visual feature points, which are numbered as follows: 0-nose, 1-left eye, 2-right eye, 3-left ear, 4-right ear, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 9-left pupil, 10-right pupil, 11-outer corner of left eye, 12-outer corner of right eye, 13-right elbow.

[0019] According to one aspect of the present invention, the step of selecting key eye visual feature points from normalized visual feature points and calculating the eye region area is specifically calculated using the following formula:

[0020]

[0021] In the formula, This represents the original total area of ​​both eyes. Let be the area of ​​the bounding rectangle of the face.

[0022] According to one aspect of the present invention, the step of selecting key torso visual feature points from normalized visual feature points and calculating the body offset angle is specifically calculated using the following formula:

[0023]

[0024] In the formula, V is the direction vector of the torso's central axis L. The direction vector is in the vertical direction. For vectors with vector The dot product; For vectors The modulus length; For vectors The length of the module.

[0025] According to one aspect of the present invention, the visual state feature value is obtained based on the area of ​​the eye region and the body offset angle, and the specific calculation formula is as follows:

[0026]

[0027] In the formula, The area of ​​the eye region. This refers to the body's offset angle.

[0028] According to one aspect of the present invention, in the step of triggering standardized voice interaction based on the visual state feature value and calculating the voice activity score in combination with the preprocessed voice interaction data, the specific calculation formula is as follows:

[0029]

[0030] In the formula, T represents the voice response delay, and E represents the response validity indicator.

[0031] According to one aspect of the present invention, the step of matching based on a predefined 3D virtual guide action sequence according to the voice response delay level, and triggering a linkage response after matching to form a light, sound, and shape linkage non-intrusive driving reminder specifically includes:

[0032] Predefined 3D virtual guide body action sequence;

[0033] Match the 3D virtual guide action sequence according to the voice response delay level;

[0034] Based on the matched 3D virtual guide action sequence, a linkage response is triggered, and the core computing unit synchronously schedules and drives the display interaction unit and the voice interaction unit to form a non-intrusive driving reminder with light, sound and shape linkage.

[0035] According to one aspect of the present invention, the closed-loop feedback control specifically includes: turning off the light, sound, and shape linkage non-intrusive driving reminder only when the core computing unit synchronously detects a preset effective active action and effective voice response from the driver; starting a 15-minute countdown when the driver leaves the driver's seat, retaining the previously triggered 3D virtual guide action sequence and the light, sound, and shape linkage non-intrusive driving reminder state during the countdown period, and automatically resetting all states after the countdown reaches 15 minutes.

[0036] The advantages of this invention are as follows: This invention effectively solves the problems of high misjudgment rate and highly intrusive reminders in traditional driving monitoring systems by employing technologies such as dual-channel redundant perception (visual and voice), precise calculation of 14 feature points on the face and torso, adaptive action triggering of a 3D virtual guide, non-intrusive reminders through light, sound, and shape linkage, and closed-loop status management. This invention adopts a tiered adaptive reminder strategy to adapt to different levels of distraction and fatigue. 3D virtual guidance and ambient lighting work together to achieve uninterrupted high wake-up. Simultaneously, a status timing and retention mechanism is included to improve interaction continuity and monitoring reliability, significantly increasing the accuracy of driving status monitoring, optimizing the in-vehicle interactive experience, and effectively ensuring driving safety. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic flowchart of a voice and visual interaction control method for safe driving according to the present invention.

[0039] Figure 2 This is a schematic diagram of the structure of a safe driving voice and visual interaction control system according to the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] like Figure 1As shown, a safe driving voice and vision interaction control method is implemented based on a hardware architecture comprising a core computing unit, a display interaction unit, a visual perception unit, a voice interaction unit, and a power execution unit. The method includes the following specific steps:

[0043] Step S1: Simultaneously collect and preprocess the driver's visual data and voice interaction data;

[0044] Step S1 involves simultaneously collecting and preprocessing the driver's visual and voice interaction data, specifically including:

[0045] In practical use, the device for collecting the driver's visual and voice interaction data uses a core computing unit as the control center, and builds a hardware architecture that includes five major units: core computing, display interaction, visual perception, voice interaction, and power execution. Specifically:

[0046] 1. Core computing unit: It adopts an edge computing main control unit, such as the Jetson series edge computing development board, equipped with a graphics memory module, and is equipped with wireless communication components and antennas. It undertakes the overall machine's collaborative scheduling, logic judgment and data processing work; it is compatible with high-capacity high-speed solid-state storage media (such as SSD) for system firmware, algorithm model storage and temporary caching of triggered acquisition data.

[0047] 2. Display and Interaction Unit: Equipped with flexible touch display components, such as a high-resolution flexible AMOLED touch screen, for rendering and visualizing 3D virtual guide bodies; supporting video driving components (such as HDMI driver boards) and high-definition video transmission cables to ensure stable transmission of video signals; integrated linear RGB LED running light module for light prompt output with light, sound and shape linkage.

[0048] 3. Visual perception unit: It adopts vehicle-mounted visual acquisition components, such as CSI interface camera modules, which have field of view specifications adapted to vehicle scenarios and can be directly connected to the core computing unit to achieve low-latency and high-definition acquisition of driver's facial visual data.

[0049] 4. Voice Interaction Unit: Adopts integrated in-vehicle audio components, such as USB interface audio modules, integrating sound cards, microphones and speakers, which can be directly connected to the core computing unit to complete the acquisition and playback of voice signals in the cabin.

[0050] 5. Power Actuation Unit: Equipped with multi-degree-of-freedom attitude adjustment load-bearing components, such as a servo gimbal structure composed of dual digital servos, equipped with digital servos of different rotation ranges and torque specifications, and matching adapter connectors, serving as the load-bearing structure of the whole machine, realizing dynamic calibration and locking of the acquisition angle.

[0051] In actual use, the data acquisition device is fixed in the driver's cab above the center console or the right-hand side of the instrument panel (driver's right front) via a servo gimbal. After power-on, the servo gimbal dynamically adjusts to the optimal acquisition angle, and the angle is locked and not adjusted again within a single driving cycle.

[0052] In practical use, the synchronous acquisition and parallel transmission of data are as follows: After the device starts up, the system clock of the core computing unit is used as the global unified time reference, and a unified timestamp is added to each set of visual data frames and audio frames. Visual data is directly connected to the core computing unit through the CSI interface of the camera module, using a single-channel video stream real-time transmission mode, with a default acquisition frame rate of 30fps and an acquisition resolution of 1920×1080; the acquired raw video stream is not persistently stored locally by default, but is directly output in real time; only when the trigger flag is set to 1, the single frame image of the video stream corresponding to the current timestamp is captured and saved to the temporary buffer of the SSD, waiting for the background algorithm to call; after the algorithm call is completed, the corresponding data in the buffer is immediately cleared and the flag is reset to 0. Audio data is directly connected to the core computing unit through the USB interface of the USB audio module, using a single-channel audio stream real-time transmission mode, with a default sampling rate of 48kHz, 16-bit quantization precision, dual-channel acquisition, and a pickup range covering the core area of ​​the driver's seat. When the voice interaction link triggers the semantic parsing requirement, and it is necessary to match the driver's facial visual state at the corresponding time, the state is automatically set to 1, and the video frame with the corresponding timestamp is captured and cached, so as to realize the on-demand linkage of voice and visual data.

[0053] In practical use, the simultaneous acquisition and preprocessing of the driver's visual data and voice interaction data specifically involves: the core computing unit simultaneously acquiring and preprocessing the visual data and voice interaction data, wherein:

[0054] Visual data preprocessing: For the original video stream frames, lens distortion correction (based on the camera module's factory calibration parameters), face region ROI extraction (locking the driver's face area and filtering redundant background pixels), and illumination normalization processing (adaptive gamma correction to eliminate the impact of strong light, backlight, and low light environments on the image) are performed in sequence.

[0055] Voice data preprocessing: For the raw audio stream, in-vehicle environment noise reduction (based on spectral subtraction to eliminate steady-state noise such as engine noise, wind noise, and road noise), acoustic echo cancellation (filtering the echo of prompts played by the speaker), and voice endpoint detection (VAD) are performed in sequence to accurately segment the driver's voice segments from the silent environmental segments.

[0056] Step S2: Extract key visual features based on the preprocessed visual data and calculate the visual state feature values;

[0057] In step S2, key visual features are extracted based on the preprocessed visual data, and visual state feature values ​​are calculated. Specifically:

[0058] Step S21: Select visual feature points based on the preprocessed visual data, and then label and normalize them; specifically:

[0059] 1. Selecting visual feature points based on preprocessed visual data

[0060] Based on the preprocessed visual data, namely the real-time image data of the driver's face and upper body after lens distortion correction, face ROI extraction, and illumination normalization processing by the core computing unit, 14 effective visual feature points (numbered 0-13) that can be stably collected by the camera module in the vehicle environment were selected, covering key parts of the driver's face and upper body. The selection criteria are that these 14 points are highly identifiable, less affected by interference from the vehicle environment, and can effectively reflect the driver's fatigue state and body posture. The specific numbers and corresponding parts are as follows: 0-nose, 1-left eye, 2-right eye, 3-left ear, 4-right ear, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 9-left pupil, 10-right pupil, 11-outer corner of left eye, 12-outer corner of right eye, 13-right elbow.

[0061] 2. Label and preprocess the visual feature points.

[0062] (1) Relative coordinate labeling: Taking the 0th nose (nose tip) feature point as the origin (0,0), the relative coordinates of the 13 feature points are labeled. ,in, , These are the horizontal and vertical relative coordinates of each feature point relative to the origin of the nose tip; Mark the location as visible. This indicates that the feature point is occluded (invisible), and only applies to [specific features]. The effective feature points are then used for subsequent calculations.

[0063] (2) Coordinate normalization processing: for all Relative coordinates of effective feature points Normalization is performed to eliminate scale interference caused by face size and shooting distance. The normalization formula is:

[0064]

[0065] In the formula, and It is the relative coordinate value of the current feature point. and All valid feature points The maximum and minimum values ​​of the coordinates. and All valid feature points The maximum and minimum values ​​of the coordinates.

[0066] Step S22: Select key eye visual feature points from the normalized visual feature points and calculate the area of ​​the eye region; specifically:

[0067] 1. Select the following feature points: 1-left eye, 9-left pupil, 11-outer corner of left eye; 2-right eye, 10-right pupil, 12-outer corner of right eye, a total of 6 eye feature points with obvious features that can stably reflect the degree of eye opening.

[0068] 2. Calculate the area of ​​the eye region.

[0069] (1) Sum of the original areas of both eyes

[0070] Arrange the three feature points of each eye according to the eye contour (left eye: 11-outer corner of left eye → 1-left eye → 9-left pupil; right eye: 12-outer corner of right eye → 2-right eye → 10-right pupil). The three feature points of each eye form a closed triangular region. First, calculate the original area of ​​each eye separately, and then sum the two to obtain the original total area of ​​both eyes. .

[0071] Calculate the original area of ​​the left and right eyes (taking the left eye as an example):

[0072]

[0073] In the formula, , The normalized relative coordinates of feature point 11 at the outer corner of the left eye; , , where is the normalized relative coordinate of feature point 1 in the left eye; , Normalized relative coordinates of feature point 9 in the left pupil; original area of ​​the right eye. The calculation method is the same as for the left eye; only the coordinates of the corresponding feature points need to be replaced.

[0074] Formula for calculating the original total area of ​​both eyes: ;

[0075] (2) Eye scale normalization

[0076] Eye area The calculation formula is as follows:

[0077]

[0078] In the formula, The area of ​​the bounding rectangle of the face is calculated using the following formula:

[0079]

[0080] In the formula, The width of the left and right ears is calculated based on the normalized relative coordinates of feature points 3 (left ear) and 4 (right ear), specifically the widths of both ears. The absolute value of the coordinate difference; The facial height is fitted to OpenFace by the core computing unit using the OpenFace algorithm to fit the facial contour of the preprocessed driver's face image output in step S1.

[0081] Step S23: Select key torso visual feature points from the normalized visual feature points and calculate the body offset angle; specifically:

[0082] 1. Select the following feature points: 0-nose, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 13-right elbow, a total of 6 feature points that can reflect the posture of the torso.

[0083] 2. Calculate the body offset angle

[0084] (1) Extract the direction vector of the trunk midline

[0085] O, the origin of the nose tip Based on this, coordinate fitting is performed on 5 selected torso-related feature points to obtain the torso midline L, and the direction vector V of the torso midline L is extracted:

[0086]

[0087] In the formula, , These are the relative coordinates of the origin O at the tip of the nose. , It is the relative coordinate of the lower end point of the trunk midline L, which is obtained by fitting the normalized coordinates of five feature points: 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, and 13-right elbow.

[0088] (2) Calculate the body offset angle

[0089] Calculate the relationship between the trunk's central axis L and the vertical direction ( The angle between the positive axis and the positive axis. That is, the body offset angle. The calculation formula is:

[0090]

[0091] In the formula, Let be a directed vector in the vertical direction (positive y-axis direction), with a value of ( , For vectors with vector The dot product; For vectors The modulus length; For vectors The module length is fixed at 1.

[0092] Body offset angle The range of values ​​is ,in, This indicates that the driver's body is completely upright and not deviated from the vertical direction; The larger the value, the more pronounced the driver's body deviation and the higher the risk of abnormal posture.

[0093] Step S24: Based on the area of ​​the eye region and the body offset angle, calculate the visual state feature value. Specifically:

[0094] Based on the 14 preprocessed feature points, the area of ​​the eye region was determined through spatial geometric fitting. Angle of body offset By fusing the two indicators and eliminating the dimensional differences, a unified visual state feature value is obtained. The calculation formula is:

[0095]

[0096] The weighting was based on the fact that eye fatigue (degree of eye opening) had a greater impact on the driver's state than body posture deviation. The corresponding weight is 0.6. The corresponding weight is 0.4; The larger the value, the higher the risk of fatigue; The larger the value, the more severe the attitude deviation. After fusion, The larger the diameter, the worse the driver's condition.

[0097] Furthermore, the rules for determining driver distraction based on visual state feature values ​​are as follows:

[0098] 1. When At that time: the driver's eyes were fully open ( Larger), upright posture ( (Smaller), without fatigue or abnormal posture;

[0099] 2. When If the driver is at risk of insufficient eye opening (fatigue) or body deviation (abnormal posture), return to step S1 to collect visual data and monitor in real time until... Or, continuous monitoring may confirm that the driver is fatigued.

[0100] Step S3: Trigger standardized voice interaction based on the visual state feature value, calculate the voice activity score by combining the preprocessed voice interaction data, and determine the driver's voice response delay level based on the voice activity score;

[0101] In step S3, standardized voice interaction is triggered based on the visual state feature values. A voice activity score is calculated using the preprocessed voice interaction data, and the driver's voice response delay level is determined based on the voice activity score. Specifically:

[0102] 1. Trigger standardized voice interaction based on the visual state feature values, and preprocess them; specifically:

[0103] Based on the driver's visual state feature value output in step S2 As the trigger for voice interaction: when And all 10 consecutive tests met the requirements. If the driver is visually distracted, the system initiates a voice interaction process. If the condition of 10 consecutive responses is not met, the system returns to step S2 for visual feature monitoring. Specifically, the voice interaction process involves proactively initiating a standardized voice interaction through the USB audio module's speaker: "Please respond within 10 seconds so the system can confirm your attention level." Only the driver's voice response "Received" is accepted; gestures, button presses, touch controls, and all other non-voice feedback are rejected. A maximum of 5 rounds of standardized voice interaction are performed. If the driver does not respond effectively within 5 rounds, it is directly judged as high latency or no response.

[0104] Upon initiating a standardized voice interaction, a 10-second timer is activated, serving as the maximum allowed response time for the driver. Failure to respond within this timeframe is considered an invalid response. The core computing unit uses the built-in in-vehicle voice recognition model for real-time transcription to determine the validity of the response.

[0105] (1) If the transcription result is "received", it is marked as a valid response. );

[0106] (2) If the transcription result is not "received", there is no voice input, or the voice is unclear and unrecognizable, it is marked as an invalid response. ).

[0107] 2. Calculate the speech activity score based on the preprocessed speech interaction data.

[0108] (1) Voice activity score The calculation formula is:

[0109]

[0110] In the formula, T is the voice response delay (unit: seconds), with a maximum timing of 10 seconds; E is the response validity indicator (E=1 for a valid response, E=0 for an invalid response).

[0111] (2) Voice activity Scoring rules: The initial base score is 100 points. 10 points are deducted for every second of delay. A delay of more than 10 seconds is counted as 10 seconds. 20 points are deducted for invalid responses. No points are deducted for valid responses. The minimum score is 0 points. Negative scores are not allowed.

[0112] 3. Based on voice activity score Determine voice activity level to obtain the corresponding voice response latency level.

[0113] (1) When If the driver's voice response is timely and effective, and the driver is in a normal state, return to step S1 to collect data again.

[0114] (2) When If there is a possibility of misjudgment, repeat step S3 voice interaction judgment;

[0115] (3) When Time: Determined as slight delay, proceed to step S4;

[0116] (4) When Time: Determined as moderate delay, proceed to step S4;

[0117] (5) When Time: Determined as severe delay, proceed to step S4.

[0118] Step S4: Based on the voice response delay level, match the predefined 3D virtual guide action sequence. After the matching is completed, trigger the linkage response to form a light, sound and shape linkage non-intrusive driving reminder.

[0119] In step S4, matching is performed based on the predefined 3D virtual guide action sequence according to the voice response delay level. After matching is completed, a linkage response is triggered to form a non-intrusive driving reminder with light, sound, and shape linkage. Specifically:

[0120] Based on the execution status of the 3D virtual guide body's action sequence and the light, sound, and shape-linked non-intrusive driving reminder, closed-loop feedback control is completed.

[0121] Predefined 3D virtual guide body action sequence;

[0122] Step S41: Predefine the 3D virtual guide body action sequence; specifically:

[0123] Sequence 1: Slow respiratory pulsation;

[0124] Sequence 2: Three blinks + pointing gesture;

[0125] Sequence 3: Rapidly wave hands and cross them in refusal.

[0126] Step S42: Match the 3D virtual guide action sequence according to the voice response delay level; specifically:

[0127] Based on the driver's voice response delay level determined in step S3, The binding with the 3D virtual guide body action sequence is as follows:

[0128] 1. When (With slight delay) trigger sequence 1;

[0129] 2. When (Medium delay) triggers sequence 2;

[0130] 3. When (Severe delay) triggers sequence 3.

[0131] Upon entering this step, sequence 1 is triggered by default, providing the driver with a 5-second adaptation buffer period; after the 5-second adaptation period, the sequence will proceed as follows: The system will automatically switch to either sequence 2 or sequence 3 based on the corresponding threshold.

[0132] Step S43: Based on the matched 3D virtual guide action sequence, a linkage response is triggered. The core computing unit synchronously schedules and drives the display interaction unit and the voice interaction unit to form a light, sound, and shape linkage non-intrusive driving reminder. Specifically:

[0133] Based on the action sequence of the 3D virtual guide, the core computing unit synchronously drives the display interaction unit (RGB LED marquee module) and the voice interaction unit (USB audio module) to operate, forming a non-intrusive reminder with light, sound, and shape linkage, specifically as follows:

[0134] 1. Sequence 1 (slightly delayed, ): No beep, the scrolling light outputs yellow light, flashing at a low frequency of 1-2Hz;

[0135] 2. Sequence 2 (moderately delayed) The voice prompt says "You are driving while fatigued, please take a rest," and the scrolling light outputs a yellow light that flashes at a high frequency of 3-5Hz.

[0136] 3. Sequence 3 (severely delayed, The voice prompt says "Please stop and rest immediately," and the scrolling red light flashes at a high frequency of 3-5Hz.

[0137] The RGB LED marquee module consists of 8-10 linearly arranged RGB LED beads, fixedly installed on the edge of the device base; it uses 5V DC power supply, compatible with the HDMI driver board and USB audio module power supply, and is powered by the device itself; the GPIO interface of the RGB LED marquee module directly connects to the core computing unit without the need for additional adapters; the LED beads support free switching between yellow and red light, and the brightness is adjustable. The blinking frequency is adjustable from 1-2Hz for low frequency and 3-5Hz for high frequency.

[0138] Step S5: Based on the action sequence of the 3D virtual guide and the execution status of the light, sound and shape linkage non-intrusive driving reminder, complete the closed-loop feedback control.

[0139] In step S5, based on the action sequence of the 3D virtual guide and the execution state of the light, sound, and shape-linked non-intrusive driving reminder, closed-loop feedback control is completed, specifically as follows:

[0140] 1. Valid Response Determination

[0141] The light, sound, and visual linkage non-intrusive reminder can only be turned off when the driver simultaneously completes a valid active action and a valid voice response. The valid active action is a nod of confirmation from the driver captured and recognized by the camera module (number of head movements up and down). Next, swing amplitude Effective voice response is determined by the USB audio module, the built-in in-vehicle voice recognition model, and the voice clarity. The designated confirmation messages (Received, Close notification, I have rested, Continue driving).

[0142] Only when the core computing unit synchronously detects a valid active action and a valid voice response will the light, sound and shape linkage non-intrusive reminder be immediately turned off, including: turning off the marquee lights, stopping the voice prompts, and terminating the rendering of the 3D virtual guide body.

[0143] 2.15-minute status timing rules

[0144] If the camera module does not detect a driver for 3 consecutive seconds, a 15-minute countdown will start immediately. When the driver returns to the driver's seat, the countdown will pause and retain the remaining time. When the driver leaves again, the countdown will resume from the remaining time. The countdown will terminate normally after 15 minutes and automatically reset all statuses. When the driver returns and completes a valid response, the countdown will terminate early and the timing data will be cleared.

[0145] 3. State preservation mechanism

[0146] Within the 15-minute timer period, the previously triggered action sequence (Sequence 1, Sequence 2, Sequence 3) is retained. Score threshold, light, sound and shape linkage non-intrusive reminder status (light color, flashing frequency, prompt sound status).

[0147] If the timer is less than 15 minutes: The driver returns to the driver's seat and triggers the fatigue monitoring process again (from step S2 to step S4). The last saved action sequence is directly used for execution. By default, sequence 1 is run first for a 5-second buffer before the saved target sequence is directly triggered without step-by-step progression.

[0148] When the timer reaches 15 minutes: all reserved states are automatically cleared, and the initial default rules are restored; when fatigue monitoring is triggered again, step S4 is strictly followed, starting with a 5-second buffer of sequence 1 by default, and then... Threshold matching sequence 2 or sequence 3.

[0149] Special rules for multiple departures or returns: If the driver leaves or returns to the driver's seat multiple times within the time period, the status is retained based solely on the action sequence triggered last time. The time is calculated by adding up the cumulative departure time, and all statuses are automatically reset when the cumulative departure time reaches 15 minutes.

[0150] Example 2

[0151] like Figure 2 As shown, this embodiment applies to the safe driving voice and visual interaction control method described in Embodiment 1. The system includes a core computing unit, a display interaction unit, a visual perception unit, a voice interaction unit, and a power execution unit.

[0152] In practical use, the display interaction unit, visual perception unit, voice interaction unit, and power execution unit all establish bidirectional communication connections with the core computing unit. The specific connection relationships and selection of each unit are as follows:

[0153] 1. Core computing unit: It adopts Jetson Orin Nano Super Developer Kit (8GB video memory), equipped with a wireless network card and dual-band antenna, and is responsible for the overall system coordination scheduling, logic judgment and data processing; it is equipped with a 500GB M.2 NVMePCIe3.0×4 SSD for firmware, model storage and trigger-based temporary data caching.

[0154] 2. Display and Interaction Unit: Equipped with a 6.67-inch Tianma flexible AMOLED touchscreen (2400×1080 resolution) for 3D virtual guide rendering; it comes with a 5V power supply HDMI driver board and a DP to HDMI 4K transmission cable to ensure stable video signal transmission; it integrates a linear RGB LED running light module for light, sound, and shape-linked lighting prompts. Video signals are transmitted via the HDMI interface, and the running light is controlled via the GPIO interface, both directly connected to the core computing unit.

[0155] 3. Visual perception unit: It adopts a 77° field of view IMX219 camera module, equipped with a 22-pin-CSI interface, which is directly connected to the core computing unit to achieve low-latency and high-definition acquisition of driver's facial visual data.

[0156] 4. Voice Interaction Unit: It adopts a USB audio module that integrates a USB driverless sound card, an omnidirectional microphone and a speaker. It connects to the core computing unit via USB to complete the acquisition and playback of cockpit voice.

[0157] 5. Power Actuation Unit: Equipped with dual digital servo gimbals, integrating a 270° rotation range 20kg torque digital servo and a 180° rotation range 25kg torque digital servo. It is directly connected to the core computing unit through the servo control interface, and is equipped with matching servo disk accessories. It serves as the load-bearing structure of the whole machine, realizing dynamic calibration and locking of the acquisition angle.

[0158] In practical use, the specific functions of each unit are as follows:

[0159] 1. Core Computing Unit: Responsible for overall machine coordination scheduling, logical judgment, and full-process data processing. Specifically, it performs synchronous preprocessing of visual data and voice interaction data, extraction and normalization calculation of 14 feature points, and calculation of visual state feature values. With voice activity score It performs calculations, matches the action sequence of the 3D virtual guide body, issues commands for light, sound, and shape linkage, and provides closed-loop feedback control; at the same time, it uses a built-in SSD to store system firmware, algorithm models, and temporarily cache triggered acquisition data.

[0160] 2. Visual Perception Unit: Performs driver visual data acquisition tasks, acquires high-definition image data of the face and upper body in real time, and provides raw data for visual feature extraction and effective active action recognition of the driver.

[0161] 3. Voice Interaction Unit: Performs cockpit voice interaction data collection tasks, completes standardized voice interaction broadcasts and driver response collection, executes graded voice prompt playback, and provides voice data support for voice activity score calculation.

[0162] 4. Display and Interaction Unit: Executes 3D virtual guide action sequence rendering and display, outputs corresponding light signals through integrated RGB running lights, and synchronously completes light, sound and shape linkage non-intrusive driving reminders.

[0163] 5. Power Actuation Unit: Performs dynamic calibration and locking functions for the acquisition angle. After power-on, it automatically adjusts to the optimal acquisition angle and maintains stable posture within a single driving cycle to ensure visual acquisition effect.

[0164] In actual use, after the system is powered on, the power execution unit first completes the acquisition perspective calibration and locking; the visual perception unit and the voice interaction unit simultaneously acquire multimodal data of the driver and transmit it to the core computing unit; the core computing unit sequentially completes data preprocessing, visual state determination, voice activity calculation, 3D action sequence triggering, light, sound and shape linkage reminders and closed-loop feedback control; the display interaction unit outputs visual reminders in real time until a valid response from the driver is detected or the state reset conditions are met, thus completing the entire process of safe driving interaction control.

[0165] The advantages of this invention are as follows: This invention effectively solves the problems of high misjudgment rate and highly intrusive reminders in traditional driving monitoring systems by employing technologies such as dual-channel redundant perception (visual and voice), precise calculation of 14 feature points on the face and torso, adaptive action triggering of a 3D virtual guide, non-intrusive reminders through light, sound, and shape linkage, and closed-loop status management. This invention adopts a tiered adaptive reminder strategy to adapt to different levels of distraction and fatigue. 3D virtual guidance and ambient lighting work together to achieve uninterrupted high wake-up. Simultaneously, a status timing and retention mechanism is included to improve interaction continuity and monitoring reliability, significantly increasing the accuracy of driving status monitoring, optimizing the in-vehicle interactive experience, and effectively ensuring driving safety.

[0166] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A voice-visual interactive control method for safe driving, characterized in that, The method is implemented based on a hardware architecture comprising a core computing unit, a display interaction unit, a visual perception unit, a voice interaction unit, and a power execution unit, including: Simultaneously collect and preprocess the driver's visual and voice interaction data; Key visual features are extracted from the preprocessed visual data, and visual state feature values ​​are calculated. Based on the visual state feature values, standardized voice interaction is triggered. The voice activity score is calculated by combining the preprocessed voice interaction data, and the driver's voice response delay level is determined based on the voice activity score. Based on the voice response delay level, a matching is performed on a predefined 3D virtual guide action sequence. Once the matching is completed, a linkage response is triggered to form a non-intrusive driving reminder with light, sound, and shape linkage. Based on the execution status of the 3D virtual guide body's action sequence and the light, sound, and shape-linked non-intrusive driving reminder, closed-loop feedback control is completed.

2. The safe driving voice and visual interaction control method according to claim 1, characterized in that, The process of simultaneously collecting and preprocessing the driver's visual and voice interaction data specifically involves: the core computing unit simultaneously collecting and preprocessing the visual and voice interaction data; performing lens distortion correction, face region ROI extraction, and illumination normalization preprocessing sequentially on the visual data; and performing vehicle environment noise reduction, acoustic echo cancellation, and voice endpoint detection preprocessing sequentially on the voice interaction data.

3. The safe driving voice and visual interaction control method according to claim 2, characterized in that, The extraction of key visual features based on preprocessed visual data and the calculation of visual state feature values ​​specifically include: Visual feature points are selected based on the preprocessed visual data, and then labeled and normalized. Key eye visual feature points are selected from the normalized visual feature points, and the area of ​​the eye region is calculated. Key torso visual feature points are selected from the normalized visual feature points, and the body offset angle is calculated. Visual state feature values ​​are obtained based on the area of ​​the eye region and the body offset angle.

4. The safe driving voice and visual interaction control method according to claim 3, characterized in that, In the process of extracting key visual features based on preprocessed visual data, the extracted key visual features are 14 preset visual feature points, which are numbered as follows: 0-nose, 1-left eye, 2-right eye, 3-left ear, 4-right ear, 5-left shoulder, 6-right shoulder, 7-left elbow, 8-right elbow, 9-left pupil, 10-right pupil, 11-outer corner of left eye, 12-outer corner of right eye, 13-right elbow.

5. The safe driving voice and visual interaction control method according to claim 4, characterized in that, The step involves selecting key eye visual feature points from the normalized visual feature points and calculating the eye region area using the following formula: In the formula, This represents the original total area of ​​both eyes. Let be the area of ​​the bounding rectangle of the face.

6. The safe driving voice and visual interaction control method according to claim 5, characterized in that, The step involves selecting key torso visual feature points from normalized visual feature points and calculating the body offset angle. The specific calculation formula is as follows: In the formula, V is the direction vector of the torso's central axis L. The direction vector is in the vertical direction. For vectors with vector The dot product; For vectors The modulus length; For vectors The length of the module.

7. The safe driving voice and visual interaction control method according to claim 6, characterized in that, The visual state feature value is obtained based on the area of ​​the eye region and the body offset angle, and the specific calculation formula is as follows: In the formula, The area of ​​the eye region. This refers to the body's offset angle.

8. The safe driving voice and visual interaction control method according to claim 1, characterized in that, The standardized voice interaction triggered based on the visual state feature value, combined with the preprocessed voice interaction data to calculate the voice activity score, is specifically calculated using the following formula: In the formula, T represents the voice response delay, and E represents the response validity indicator.

9. The safe driving voice and visual interaction control method according to claim 1, characterized in that, The process of matching based on the predefined 3D virtual guide action sequence according to the voice response delay level, and triggering a linkage response after matching to form a non-intrusive driving reminder with light, sound, and shape linkage, specifically involves: Predefined 3D virtual guide body action sequence; Match the 3D virtual guide action sequence according to the voice response delay level; Based on the matched 3D virtual guide action sequence, a linkage response is triggered, and the core computing unit synchronously schedules and drives the display interaction unit and the voice interaction unit to form a non-intrusive driving reminder with light, sound and shape linkage.

10. The safe driving voice and visual interaction control method according to claim 1, characterized in that, The closed-loop feedback control is as follows: the light, sound, and shape linkage non-intrusive driving reminder is turned off only when the core computing unit synchronously detects the preset effective active action and effective voice response of the driver; when the driver leaves the driver's seat, a 15-minute countdown is started, and the action sequence of the 3D virtual guide and the state of the light, sound, and shape linkage non-intrusive driving reminder are retained during the countdown period. After the countdown reaches 15 minutes, all states are automatically reset.

Images