Driver safe driving prompting system and method based on AR glasses

Abstract

The invention relates to a driver safe driving prompting method based on AR glasses, and the method comprises the steps: obtaining a forward video, collected by the AR glasses, of a first visual angle of a driver in a vehicle driving process; tracking a fixed marker in the forward video based on a target tracking algorithm of correlation filtering to form a dynamic anchor frame; according to the dynamic anchor frame, intercepting a front view subgraph in a video frame of the forward video; obtaining a vehicle lateral deviation state, a vehicle angle deviation state and a sight attention deviation state according to the front view subgraph, and determining a vehicle risk level; and outputting prompt information according to the vehicle risk level. While effective transmission of the prompt information and the navigation information is ensured, the interference to the driving task is reduced, so that the overall driving safety is improved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and more specifically, to a driver safety driving prompt system and method based on AR glasses. Background Technology

[0002] Human-computer interaction (HCI) technology studies the exchange of information and commands between users and systems. With the development of HCI and artificial intelligence, HCI has evolved from simple single-channel information exchange to multi-channel collaborative information exchange. In the field of intelligent transportation, driver assistance systems (ADAS) are a typical example of multi-channel information exchange systems. Cars use multi-source sensors to perceive the surrounding environment and extract relevant information to output commands to the driver. The driver can also control the car through commands and operations, achieving a human-machine co-driving interaction mode.

[0003] The common human-computer interaction methods in the existing technology mainly include: (1) central control screen display: displaying prompt text, graphics and vehicle status information on the central control LCD screen; (2) head-up display (HUD): projecting brief key information such as speed and navigation on the windshield; (3) sound and light alarm device: providing prompts through buzzers, instrument panel indicator lights and voice broadcasts. The above interaction methods have the following shortcomings: (1) obvious eye movement, increasing secondary risks: the central control screen is usually located below the driver's line of sight, and the driver needs to significantly shift his line of sight or even turn his head to obtain information, which can easily cause "the prompt system to introduce new attention distraction". Although the HUD is located in the area in front of the line of sight, the display range is limited and the content is complex, which can easily obstruct the road view. (2) insufficient environmental adaptability: the HUD projection effect is significantly affected by external light. In strong light, backlight, night and extreme weather such as rain and fog, the visibility and contrast are easily reduced, resulting in unclear key information. (3) Focusing only on vehicle status and lacking driver status monitoring: Traditional ADAS systems focus on the vehicle's movement relative to the lane and surrounding obstacles, but pay insufficient attention to the driver's attention distribution, gaze deviation and distraction behavior, making it difficult to identify high-risk situations caused by prolonged gaze deviation, playing on mobile phones, talking to passengers, etc. (4) Eye-tracking-based gaze detection is costly and complex to deploy: Some studies use eye-tracking devices to collect driver gaze trajectories, but eye trackers are expensive, have high wearing thresholds and deviate from daily driving habits, making it difficult to promote on a large scale among ordinary vehicles and the general driving population. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, this application provides a driver safety driving prompt system and method based on AR glasses.

[0005] Firstly, a method for providing driver safety prompts based on AR glasses is provided, including: During vehicle operation, the driver's first-person forward-facing video, captured by AR glasses, is obtained. The target tracking algorithm based on correlation filtering tracks fixed landmarks in the forward video to form dynamic anchor boxes; based on the dynamic anchor boxes, a forward view sub-image is extracted from the video frames of the forward video. A lane line detection method based on anchors is adopted to determine lane lines based on the forward view sub-image and to determine the lane center line based on the lane lines. Calculate the distance between the vehicle's center point and the lane center line to obtain the vehicle's lateral offset; calculate the vehicle's angular offset based on the lane center line and the vehicle's current orientation; calculate the offset of the driver's line of sight attention point relative to the line of sight calibration point to obtain the line of sight attention offset. The lateral offset state of the vehicle is determined based on the vehicle's lateral offset, current speed, and current acceleration. The vehicle angle offset state is determined based on the vehicle angle offset, the vehicle's current speed, and the vehicle's current acceleration. The gaze attention shift is input into the gaze attention analysis model to obtain the gaze attention shift state; the gaze attention analysis model includes a two-layer bidirectional gated loop unit and a multi-head self-attention mechanism connected in sequence; The vehicle risk level is determined based on the vehicle's lateral deviation, angular deviation, and visual attention shift. Based on the vehicle's risk level, output a warning message.

[0006] In one embodiment, extracting a forward view sub-image from a video frame of the forward video based on a dynamic anchor frame includes: At a predetermined position within the relatively dynamic anchor frame, a sub-image of the forward view is captured, which includes core road information.

[0007] In one embodiment, calculating the vehicle angle offset based on the lane centerline and the vehicle's current orientation includes: When the lane centerline is a straight line, calculate the angle between the direction of the lane centerline and the current direction of the vehicle, and use it as the vehicle angle offset. When the lane centerline is a curve, determine the tangent at the starting position of the lane centerline, calculate the angle between the tangent and the vehicle's current orientation, and use it as the vehicle's angular offset.

[0008] In one embodiment, determining the vehicle lateral offset state based on the vehicle's lateral offset, current vehicle speed, and current vehicle acceleration includes: Calculate the comprehensive index of vehicle lateral offset state :

[0009] in, , , All are weighting coefficients. The vehicle's current speed. The vehicle's current acceleration. This represents the lateral offset of the vehicle. Calculate dynamic threshold time :

[0010] in, Based on the base threshold time, , The attenuation coefficient; like If so, the vehicle's lateral deviation is considered normal. This is the time value from the detection of the vehicle's lateral deviation to the current time; like and If so, the vehicle's lateral deviation is considered a normal dangerous condition; The distinguishing value for differentiating between ordinary dangerous conditions and emergency dangerous conditions; like and If the vehicle is in a lateral deviation state, then the vehicle is in an emergency danger state.

[0011] In one embodiment, determining the vehicle angle offset state based on the vehicle angle offset, the vehicle's current speed, and the vehicle's current acceleration includes: Calculate the comprehensive index of vehicle angular offset state :

[0012] in, , , All are weighting coefficients. The vehicle's current speed. The vehicle's current acceleration. This represents the vehicle's angular offset. Calculate dynamic threshold time :

[0013] in, Based on the base threshold time, , The attenuation coefficient; like If so, the vehicle's angular deviation is in a normal state. This is the time value from the detection of the vehicle's angular deviation to the current time; like and If so, the vehicle's angular deviation state is a normal dangerous state; The distinguishing value for differentiating between ordinary dangerous conditions and emergency dangerous conditions; like and If the vehicle's angle deflection is then considered an emergency danger state.

[0014] In one embodiment, determining the vehicle risk level based on the vehicle's lateral drift state, vehicle angular drift state, and visual attention drift state includes: The lateral deviation of a vehicle includes normal state, ordinary danger state and emergency danger state, with corresponding danger scores of 0, 1 and 3 respectively; The vehicle angle deviation status includes normal status, ordinary danger status and emergency danger status, with corresponding danger scores of 0, 1 and 3 respectively; The state of visual attention deviance includes normal state, ordinary danger state and emergency danger state, with corresponding danger scores of 0, 2 and 4 respectively; The weights of vehicle lateral deviation, vehicle angular deviation, and visual attention deviation are set, and each is multiplied by its corresponding state hazard score to obtain three weighted values. These values ​​are then summed to obtain the final state hazard score. The vehicle risk level is determined based on the final hazard score.

[0015] Secondly, a driver safety driving prompt system based on AR glasses is provided, including AR glasses, a computing unit, an in-vehicle sensing module, and a communication module; AR glasses are used to capture forward-facing video from the driver's first-person perspective and send it to the computing unit via a communication module; The computing unit is used to implement the above-mentioned driver safety driving prompt method based on AR glasses, and to send vehicle risk level prompt information to AR glasses through the communication module; The vehicle-mounted sensing module is used to acquire the vehicle's current speed and current acceleration, and then sends them to the computing unit via the communication module.

[0016] In one embodiment, the AR glasses present vehicle risk level information in the form of text and / or voice.

[0017] In one embodiment, the communication module is also used to receive roadside equipment information and other intelligent connected vehicle information, and send them to AR glasses, which then present the received information in the driver's field of vision.

[0018] Compared with the prior art, this application has the following beneficial effects: The driver safety driving prompt system and method based on AR glasses of this application realizes the collaborative judgment of the vehicle's lane departure status and the driver's visual attention status, and constructs a hierarchical prompt mechanism. While ensuring the effective transmission of prompt information and navigation information, it reduces interference with the driving task itself, thereby improving the overall driving safety. Attached Figure Description

[0019] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart of a driver safety prompt method based on AR glasses is shown; Figure 2 This diagram illustrates the display of prompts in AR glasses. Detailed Implementation

[0020] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0021] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution of this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0022] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0023] AR glasses offer a flexible human-computer interaction method. By wearing AR glasses (such as the Microsoft HoloLens 2 helmet), drivers can ensure their gaze remains forward while simultaneously perceiving the driving environment through their forward-facing camera. This environmental awareness provides alerts about driving risks, and the system can also detect the driver's attention level, providing prompts when the driver is not focused on driving. The AR glasses display and output prompts via voice, and can also present navigation information. This AR-based driver safety alert system creates a completely new driving experience, making the entire driving process safer.

[0024] This application provides a method for providing safe driving prompts to drivers based on AR glasses. Figure 1 A flowchart of a driver safety prompt method based on AR glasses is shown. See [link / reference]. Figure 1 The method mainly includes the following steps: Step S1: During vehicle operation, acquire the driver's first-person forward-facing video captured by the AR glasses.

[0025] After the vehicle is started or the system is powered on, the AR glasses activate the forward-facing camera to capture forward-facing video from the driver's first-person perspective. The driver is required to maintain a normal forward-looking driving posture, at which point the center point of their line of sight (usually corresponding to the vanishing point of the road or the center line of the road ahead) is recorded as the line-of-sight calibration point.

[0026] Step S2: Based on the target tracking algorithm of correlation filtering, track fixed landmarks in the forward video to form dynamic anchor frames; based on the dynamic anchor frames, extract the forward view sub-image from the video frames of the forward video.

[0027] A fixed marker is preset or affixed in the vehicle's forward field of view, such as a high-contrast graphic below the windshield or on the left side of the dashboard. This fixed marker is detected in the initial video frame, and its location is used as the initial anchor frame. A target tracking algorithm based on correlation filtering is then employed to continuously track the fixed marker, updating the position and scale of the dynamic anchor frame to ensure it tightly surrounds the marker.

[0028] Further, at a predetermined position relative to the dynamic anchor frame, a forward view sub-image is captured, which includes core road information. Here, the forward view sub-image can be captured at a predetermined distance from the anchor frame marker, according to a preset size. The core road information refers to the current lane and the adjacent lane area.

[0029] This step effectively ensures the stability and clarity of lane line recognition video data captured from the driver's perspective. Even if there is shaking or other visual interference during vehicle movement, the dynamic anchor frame can still ensure that lane line recognition is not affected, effectively improving the accuracy and reliability of lane line detection.

[0030] Step S3: Using an anchor-based lane detection method, lane lines are determined based on the forward view sub-image, and lane centerlines are determined based on the lane lines. The centerline of the lane lines on both sides is the lane centerline.

[0031] Specifically, Ultra Fast Lane Detection (UFLD) can be used for lane detection, which requires less computation, has high real-time performance, and improves the detection accuracy of lane lines in complex curves and partially occluded scenarios by utilizing global features.

[0032] Step S4: Calculate the distance between the vehicle center point and the lane center line to obtain the vehicle lateral offset; calculate the vehicle angular offset based on the lane center line and the vehicle's current orientation; calculate the offset of the driver's line of sight attention point relative to the line of sight calibration point to obtain the line of sight attention offset.

[0033] Specifically, based on the lane centerline and the vehicle's current orientation, the vehicle's angular offset is calculated, including: When the lane centerline is a straight line, calculate the angle between the direction of the lane centerline and the current direction of the vehicle, and use it as the vehicle angle offset. When the lane centerline is curved, determine the tangent at the starting point of the lane centerline, and calculate the angle between the tangent and the vehicle's current orientation, which is taken as the vehicle's angular offset. Here, the starting point of the lane centerline refers to the point on the lane centerline closest to the vehicle.

[0034] Here, the gaze calibration point refers to the vanishing point of the road in the forward field of view sub-map, which serves as the reference position when the driver's gaze is fully focused on the driving task. The driver's gaze attention point refers to the center point of the video data collected through the AR glasses, which can approximate the location where the driver's gaze is focused.

[0035] Step S5: Determine the vehicle's lateral offset state based on the vehicle's lateral offset, current speed, and current acceleration.

[0036] Step S6: Determine the vehicle angle offset state based on the vehicle angle offset, the vehicle's current speed, and the vehicle's current acceleration.

[0037] Step S7: Input the gaze attention offset into the gaze attention analysis model to obtain the gaze attention offset state; the gaze attention analysis model includes a two-layer bidirectional gated loop unit and a multi-head self-attention mechanism connected in sequence.

[0038] Here, the gaze attention shift is input to a two-layer bidirectional gated recurrent unit, which encodes the shift along the forward and reverse time directions to capture the contextual dependencies between time points. The features output by the two-layer bidirectional gated recurrent unit are processed by a multi-head self-attention mechanism to output the probability distribution of the three states (normal, ordinary danger, and emergency danger) in the gaze attention shift state.

[0039] The gaze attention analysis model is an optimized gaze attention analysis model obtained by globally searching the key hyperparameters of the network using the antlion optimization algorithm. The antlion algorithm adaptively guides the hyperparameter combination to converge to a high fitness region by simulating the predation behavior of ants and antlions in the search space. The fitness function is constructed with indicators such as classification accuracy, macro average F1 score and recall rate of different categories to achieve performance balance for the three gaze attention states, with a focus on improving the ability to identify emergency danger categories.

[0040] Step S8: Determine the vehicle risk level based on the vehicle's lateral deviation, angular deviation, and visual attention shift.

[0041] Step S9: Output a prompt message based on the vehicle risk level.

[0042] This embodiment achieves coordinated judgment of vehicle lane departure status and driver visual attention status, and constructs a hierarchical prompting mechanism. While ensuring the effective transmission of prompting information and navigation information, it reduces interference with the driving task itself, thereby improving overall driving safety.

[0043] In one embodiment, step S5, determining the vehicle lateral offset state based on the vehicle's lateral offset, current vehicle speed, and current vehicle acceleration, includes: Calculate the comprehensive index of vehicle lateral offset state :

[0044] in, , , All are weighting coefficients. The vehicle's current speed. The vehicle's current acceleration. This represents the lateral offset of the vehicle. Calculate dynamic threshold time :

[0045] in, Based on the base threshold time, , The attenuation coefficient; like If so, the vehicle's lateral deviation is considered normal. This is the time value from the detection of the vehicle's lateral deviation to the current time; like and If so, the vehicle's lateral deviation is considered a normal dangerous condition; The distinguishing value for differentiating between ordinary dangerous conditions and emergency dangerous conditions; like and If the vehicle is in a lateral deviation state, then the vehicle is in an emergency danger state.

[0046] In this embodiment, when a small lateral drift of the vehicle is detected and this drift persists for a period of time, a timer is automatically started, and the timer threshold is dynamically adjusted based on the vehicle's speed and acceleration. As the vehicle's speed and acceleration increase, the timer threshold decreases accordingly, thus responding more quickly to potential hazards. When the timer exceeds a preset threshold, the system infers that the driver may have poor driving habits, such as inattentiveness or a slightly dangerous driving state. In an emergency, when the system detects a large lateral drift of the vehicle and the driver has not activated the turn signal, the system identifies this situation as a possible dangerous driving state or a sign of an impending event.

[0047] In one embodiment, step S6, determining the vehicle angle offset state based on the vehicle angle offset, the vehicle's current speed, and the vehicle's current acceleration, includes: Calculate the comprehensive index of vehicle angular offset state :

[0048] in, , , All are weighting coefficients. The vehicle's current speed. The vehicle's current acceleration. This represents the vehicle's angular offset. Calculate dynamic threshold time :

[0049] in, Based on the base threshold time, , The attenuation coefficient; like If so, the vehicle's angular deviation is in a normal state. This is the time value from the detection of the vehicle's angular deviation to the current time; like and If so, the vehicle's angular deviation state is a normal dangerous state; The distinguishing value for differentiating between ordinary dangerous conditions and emergency dangerous conditions; like and If the vehicle's angle deflection is then considered an emergency danger state.

[0050] In this embodiment, depending on the specific situation of angular deviation, the system not only monitors the changes in the vehicle's angle relative to the lane centerline in real time, but also analyzes the driving state based on multiple data such as the vehicle's speed and acceleration. When the system detects a slight but continuous deviation from the driving trajectory, it activates a timer. The trigger threshold of the timer decreases with the intensity of the vehicle's movement to adapt to more risky dynamic environments. Once the timer reaches a preset limit, the system infers that the driver may have made an unintentional driving deviation or a slightly risky operation. In sudden or extreme deviation situations, such as when the driver fails to use the turn signal and causes a significant angular deviation or loss of directional control, the system will quickly identify and switch to a high-level danger state.

[0051] In one embodiment, step S8, determining the vehicle risk level based on the vehicle's lateral deviation state, vehicle angular deviation state, and visual attention deviation state, includes: The lateral deviation of a vehicle includes normal state, ordinary danger state and emergency danger state, with corresponding danger scores of 0, 1 and 3 respectively; The vehicle angle deviation status includes normal status, ordinary danger status and emergency danger status, with corresponding danger scores of 0, 1 and 3 respectively; The state of visual attention deviance includes normal state, ordinary danger state and emergency danger state, with corresponding danger scores of 0, 2 and 4 respectively; Distraction of gaze / attention: Normal state, that is, focusing the gaze on the area directly related to the driving task; In ordinary dangerous situations, the line of sight is briefly deviated, such as quickly checking the in-vehicle screen, having a brief conversation with the occupants, or briefly observing the scenery outside the vehicle; In emergency situations, the gaze is severely deviated for an extended period, such as continuously browsing a mobile phone, engaging in prolonged conversations with the face turned to the side, or experiencing obvious fatigue and drowsiness.

[0052] Set weights w3 for vehicle lateral deviation state w1, vehicle angular deviation state w2, and visual attention deviation state, satisfying w1>w2=w3. Multiply each weighted value by its corresponding state hazard score to obtain three weighted values, and sum them to obtain the final state hazard score.

[0053] Table 1 shows the hazard scores for three driving conditions.

[0054] Table 1

[0055] Finally, the vehicle risk level is determined based on the final hazard score.

[0056] Here, vehicle risk levels are divided into 5 levels: Level 0 warning, with a final danger score of 0: indicates that the driving condition is safe; Level 1 warning, the final danger score is 1: indicating that there is a slight risk; Level 2 warning, the final danger score is 2: indicating that increased vigilance is required; Level 3 warning, with a final danger score of 3: indicating a relatively serious danger trend; Level 4 warning, with a final danger score of ≥4: indicates that the vehicle and driver are in an extremely high-risk situation.

[0057] This application also provides a driver safety driving prompt system based on AR glasses, including AR glasses, a computing unit, an in-vehicle sensing module, and a communication module; AR glasses are used to capture forward-facing video from the driver's first-person perspective and send it to the computing unit via a communication module; The computing unit is used to implement the driver safety driving prompt method based on AR glasses in the above embodiment, and to send vehicle risk level prompt information to AR glasses through the communication module; The vehicle-mounted sensing module is used to acquire the vehicle's current speed and current acceleration, and then sends them to the computing unit via the communication module.

[0058] Furthermore, the AR glasses present vehicle risk level information in the form of text and / or voice. Figure 2 This diagram illustrates the display of prompts in AR glasses.

[0059] Specifically, different warning levels are indicated by different colors and voice prompts to indicate the current danger status: Level 0 warning: Displays a warning such as "Drive Safely," with the text displayed as solid white text. Level 1 prompt: White text + slow speech; Level 2 prompt: Red text + slow speech; Level 3 alert: Flashing red text + emergency voice message; Level 4 alert: Flashing red text + emergency voice message + beeping sound.

[0060] The combination of visual and auditory cues constitutes a multimodal cuing mechanism, which ensures that information can be perceived in a timely manner while minimizing the negative impact of excessive cues on the driver's attention and operation.

[0061] Furthermore, the communication module is also used to receive information from roadside equipment and other intelligent connected vehicles, and send it to the AR glasses, which then display the received information in the driver's field of vision.

[0062] Roadside equipment provides information including real-time road condition detection, traffic flow status, lane status, road hazard warnings, and speed limit reminders. It can also obtain information on vehicles and pedestrians in blind spots, achieving beyond-line-of-sight perception, unaffected by weather conditions. Real-time road condition information includes the number, location, and speed of vehicles on the current road segment, and intelligently analyzes congestion levels. Traffic flow status includes road-level queuing status and congestion levels; lane status includes lane-level queuing status, congestion levels, lane purpose, direction of travel, and whether the lane is open or closed; road hazard warnings detect obstacles, water accumulation, potholes, or other dangerous conditions in real-time, or when severe weather such as rain, snow, or fog increases the risk of driving on that section of road, and send this information to vehicles within a certain distance via voice prompts; speed limit reminders sense vehicle speed and direction in real-time, and when a vehicle's speed exceeds the road segment's speed limit, a speed limit reminder is sent via voice prompts.

[0063] Other information sent by intelligent connected vehicles includes collision warnings, lane change warnings, and status information of other road users, all delivered via voice. Collision warnings require real-time perception of the vehicle's position, speed, and direction. When the vehicle determines there is a risk of collision with another vehicle, it sends a warning to the driver via voice. Lane change warnings require real-time perception of the vehicle's absolute position, combined with high-precision road maps and lane recognition information, to determine the vehicle's relative position within the lane and send this information via voice. Status information of other road users includes the connected vehicle's position, speed, acceleration, and orientation. This information is delivered via voice, but only within a certain distance.

[0064] Furthermore, since AR glasses do not have GPS location information, they need to connect to other GPS devices or GPS-enabled mobile phones across platforms to obtain the driver's GPS location information; based on the input location point and destination, online map API functions are called to obtain navigation content.

[0065] Compared with the prior art, this application has the following technical advantages: 1. The design is novel, using AR glasses as an interaction method to provide drivers with a variety of safe driving prompts, realizing a new type of head-mounted personal driving safety prompt system.

[0066] 2. The operation is simple and convenient. The driver does not need to take his hands off the steering wheel, switch his gaze between multiple interactive devices, or install a separate eye-tracking device in the car, which ensures that the driver can concentrate on driving.

[0067] 3. The forward-facing video of the AR glasses serves as a unified data entry point, enabling the acquisition of vehicle lane position and analysis of driver's gaze direction. This allows for collaborative assessment of vehicle status and driver attention status, avoiding the system complexity and data fusion difficulties caused by the dispersed deployment of multiple sensors in traditional systems.

[0068] 4. The analysis decouples the lateral offset and directional angular offset of the vehicle relative to the lane. Based on the magnitude and duration of the two types of offsets, multiple levels of danger are defined. The influence of vehicle speed and acceleration on the degree of danger is then comprehensively considered to more accurately reflect the real risks under different working conditions.

[0069] 5. By separately judging and integrating the vehicle's lateral deviation, angular deviation, and visual attention status, a five-level warning system from safety to major risk is constructed, making the warnings easier to understand and accept. It can provide drivers with progressive warnings and reduce the shock, distraction, and secondary risks caused by sudden and strong alarms.

[0070] 6. It provides a natural and comfortable interaction method, combining information presentation with head position, maintaining traditional driving habits, reducing the interference of too much information and the impact of information presentation on the observation of normal driving roads.

[0071] 7. By using augmented reality technology, virtual information blends more naturally with the real world, and virtual information is presented in the correct position in the real world.

[0072] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for providing safe driving prompts to drivers based on AR glasses, characterized in that, include: During vehicle operation, the driver's first-person forward-facing video, captured by AR glasses, is obtained. A target tracking algorithm based on correlation filtering is used to track fixed landmarks in the forward video and form dynamic anchor frames. Based on the dynamic anchor frame, a forward view sub-image is extracted from the video frame of the forward video; A lane line detection method based on line anchors is adopted to determine lane lines based on the forward view sub-image and to determine the lane center line based on the lane lines. Calculate the distance between the vehicle's center point and the lane centerline to obtain the vehicle's lateral offset. Calculate the vehicle angle offset based on the lane centerline and the vehicle's current orientation; Calculate the offset of the driver's line of sight attention point relative to the line of sight calibration point to obtain the line of sight attention offset; The vehicle's lateral offset state is determined based on the vehicle's lateral offset, current speed, and current acceleration. The vehicle angle offset state is determined based on the vehicle angle offset, the vehicle current speed, and the vehicle current acceleration. The gaze attention offset is input into the gaze attention analysis model to obtain the gaze attention offset state; the gaze attention analysis model includes a two-layer bidirectional gated loop unit and a multi-head self-attention mechanism connected in sequence. The vehicle risk level is determined based on the vehicle's lateral deviation state, vehicle's angular deviation state, and visual attention deviation state. Based on the vehicle's risk level, output a prompt message.

2. The method as described in claim 1, characterized in that, in, Based on the dynamic anchor frame, a forward view sub-image is extracted from the video frame of the forward video, including: At a predetermined position relative to the dynamic anchor frame, a forward view sub-image is captured, which includes core road information.

3. The method as described in claim 1, characterized in that, in, Based on the lane centerline and the vehicle's current orientation, calculate the vehicle's angular offset, including: When the lane centerline is a straight line, calculate the angle between the direction of the lane centerline and the current direction of the vehicle, and use it as the vehicle angle offset. When the lane centerline is a curve, determine the tangent at the starting position of the lane centerline, and calculate the angle between the tangent and the vehicle's current orientation, which is used as the vehicle's angular offset.

4. The method as described in claim 1, characterized in that, in, Based on the vehicle's lateral offset, current speed, and current acceleration, the vehicle's lateral offset state is determined, including: Calculate the comprehensive index of vehicle lateral offset state : in, , , All are weighting coefficients. The vehicle's current speed. The vehicle's current acceleration. This represents the lateral offset of the vehicle. Calculate dynamic threshold time : in, Based on the base threshold time, , The attenuation coefficient; like If so, the vehicle's lateral deviation is considered normal. This is the time value from the detection of the vehicle's lateral deviation to the current time; like and If so, the vehicle's lateral deviation is considered a normal dangerous condition; The distinguishing value for differentiating between ordinary dangerous conditions and emergency dangerous conditions; like and If the vehicle is in a lateral deviation state, then the vehicle is in an emergency danger state.

5. The method as described in claim 1, characterized in that, in, Based on the vehicle angular offset, the vehicle's current speed, and the vehicle's current acceleration, the vehicle angular offset state is determined, including: Calculate the comprehensive index of vehicle angular offset state : in, , , All are weighting coefficients. The vehicle's current speed. The vehicle's current acceleration. This represents the vehicle's angular offset. Calculate dynamic threshold time : in, Based on the base threshold time, , The attenuation coefficient; like If so, the vehicle's angular deviation is in a normal state. This is the time value from the detection of the vehicle's angular deviation to the current time; like and If so, the vehicle's angular deviation state is a normal dangerous state; The distinguishing value for differentiating between ordinary dangerous conditions and emergency dangerous conditions; like and If the vehicle's angle deflection is then considered an emergency danger state.

6. The method as described in claim 1, characterized in that, in, Based on the vehicle's lateral deviation state, vehicle's angular deviation state, and visual attention deviation state, the vehicle risk level is determined, including: The vehicle's lateral deviation status includes a normal status, a normal dangerous status, and an emergency dangerous status, with corresponding danger scores of 0, 1, and 3, respectively. The vehicle angle deviation status includes normal status, ordinary danger status and emergency danger status, with corresponding danger scores of 0, 1 and 3 respectively; The state of visual attention shift includes a normal state, a normal dangerous state, and an emergency dangerous state, with corresponding state danger scores of 0, 2, and 4, respectively. The weights of the vehicle's lateral deviation state, vehicle's angular deviation state, and visual attention deviation state are set, and multiplied by the corresponding state hazard score to obtain three weighted values, which are then summed to obtain the final state hazard score. The vehicle risk level is determined based on the final state hazard score.

7. A driver safety prompting system based on AR glasses, characterized in that, Includes AR glasses, computing unit, vehicle-mounted sensing module, and communication module; The AR glasses are used to capture forward-facing video from the driver's first-person perspective and send it to the computing unit via the communication module; The computing unit is used to implement the driver safety driving prompt method based on AR glasses as described in any one of claims 1-6, and to send vehicle risk level prompt information to the AR glasses through the communication module; The vehicle-mounted sensing module is used to acquire the vehicle's current speed and current acceleration, and sends them to the computing unit via the communication module.

8. The system as described in claim 7, characterized in that, The AR glasses present vehicle risk level information in the form of text and / or voice.

9. The system as described in claim 7, characterized in that, The communication module is also used to receive information from roadside equipment and other intelligent connected vehicles, and send it to the AR glasses, which then display the received information in the driver's field of vision.

Images