Augmented reality head-up display method, augmented reality device and readable storage medium

By using augmented reality technology in cars, risk warning images are generated based on the vehicle's operating conditions and environmental information, the problem of low intelligence in the existing technology is solved, and timely reminders and safety improvements are achieved for potential traffic risks.

CN118981109AInactive Publication Date: 2024-11-19GOERTEK INC

Patent Information

Application Number
CN202310572556.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing head-up display and augmented reality technology applications in cars are low in intelligence, and cannot effectively remind drivers of potential traffic risks, resulting in safety hazards.

Method used

By establishing a communication link with the vehicle, obtaining operating conditions and driving environment information, determining whether the vehicle is in a risk scenario, and generating risk warning image content after determination, presenting it in real time on the window of the augmented reality device, including risk image identification, Location distance information and coping strategies.

Benefits of technology

It improves the intelligence of the vehicle driving process, promptly pushes risk warning information, reduces driver distraction and safety hazards, and avoids traffic accidents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981109A_ABST
    Figure CN118981109A_ABST
Patent Text Reader

Abstract

The invention discloses an augmented reality head-up display method, augmented reality equipment and a readable storage medium, and relates to the technical field of augmented reality, and the augmented reality head-up display method comprises the steps: building communication link connection with a vehicle, and obtaining the operation condition information and the driving environment information of the vehicle through a communication link; determining whether the vehicle enters a risk scene according to the operation condition information and the driving environment information; and after determining that the vehicle is in the risk scene, generating risk early warning image content corresponding to the risk scene, and presenting the risk early warning image content on the current window of the augmented reality device. The intelligent degree of the vehicle driving process can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and in particular to an augmented reality head-up display method, an augmented reality device, and a readable storage medium. Background Art

[0002] Head Up Display (HUD), also known as the head-up display system, projects important driving information such as speed and navigation onto the windshield in front of the driver, allowing the driver to see important driving information such as speed and navigation without lowering or turning his head. Augmented Reality (AR) is a technology that cleverly integrates virtual information with the real world. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0003] With the development of automobiles, head-up display and augmented reality technology have been applied in automobiles. Combining head-up display and augmented reality technology in automobiles allows drivers to see images that integrate virtual images with real environments without lowering their heads, providing convenience for drivers to drive. However, currently, only augmented reality head-up display technology can be used in automobiles to display navigation routes, vehicle speed, etc., and the degree of intelligence in the vehicle driving process is low. Summary of the invention

[0004] The main purpose of this application is to provide an augmented reality head-up display method, an augmented reality device and a readable storage medium, aiming to solve the problem of low intelligence level in the vehicle driving process.

[0005] To achieve the above object, the present application provides an augmented reality head-up display method, which is applied to an augmented reality device, and the method includes: Establishing a communication link connection with the vehicle, and obtaining operating condition information and driving environment information of the vehicle through the communication link; Determining whether the vehicle enters a risk scenario based on the operating condition information and the driving environment information; After determining that the vehicle is in a risk scenario, a risk warning image content corresponding to the risk scenario is generated, and the risk warning image content is presented on a current window of the augmented reality device.

[0006] Optionally, the step of determining whether the vehicle enters a risk scenario according to the operating condition information and the driving environment information includes: Determine, based on the operating condition information and the driving environment information, a risk probability value of a collision between the vehicle and a target object, wherein the target object is a pedestrian, a pothole on the road surface, a building, or another vehicle whose distance from the vehicle is less than or equal to a preset distance; If the risk probability value is less than a preset probability threshold, it is determined that the vehicle is not in a risk scenario; If the risk probability value is greater than or equal to a preset probability threshold, it is determined that the vehicle is in a risk scenario.

[0007] Optionally, the step of presenting the risk warning image content on the current window of the augmented reality device includes: The risk warning image content is presented in a preset target area on the current window of the augmented reality device, wherein the preset target area is an area that matches the orientation of the target object.

[0008] Optionally, before the step of generating risk warning image content corresponding to the risk scenario, the method further includes: Determining the probability interval in which the risk probability value lies; According to the probability interval, query and obtain the risk level mapped by the probability interval from a preset scenario type mapping table; If the mapped risk level belongs to the first risk level, executing: the step of generating the risk warning image content corresponding to the risk scenario; If the mapped risk level belongs to the second risk level, the vibration module of the augmented reality device is controlled to generate vibration of a preset duration, and the step of generating risk warning image content corresponding to the risk scenario is executed, wherein the second risk level is greater than the first risk level.

[0009] Optionally, after the step of querying and obtaining the risk level of the probability interval mapping from a preset scenario type mapping table, the method further includes: If the mapped risk level belongs to the third risk level, response strategy information for avoiding collision between the vehicle and the target object is generated and sent to the vehicle, wherein the response strategy information is used to trigger the vehicle to actively intervene in executing a response strategy to avoid collision risk, and the third risk level is greater than the second risk level.

[0010] Optionally, the step of generating the response strategy information for avoiding the collision between the vehicle and the target object includes: Obtaining the type of the target object; According to the type of the target object, response strategy information for avoiding a collision between the vehicle and the target object is generated.

[0011] Optionally, the step of generating, according to the type of the target object, the response strategy information for avoiding the collision between the vehicle and the target object comprises: If the type of the target object is the pothole on the road, the response strategy information for avoiding the collision between the vehicle and the target object is determined as follows: the vehicle is decelerated to a speed less than a preset speed threshold; If the type of the target object is a pedestrian, a building or another vehicle, the response strategy information for avoiding a collision between the vehicle and the target object is determined as: vehicle deceleration and braking.

[0012] Optionally, the risk warning image content includes at least one of risk image identification information, location distance information between the vehicle and the target object, environmental image information of an area corresponding to the target object, and risk avoidance response strategy information.

[0013] The present application also provides an augmented reality device, which is a physical device. The augmented reality device includes: a memory, a processor, and a program of the augmented reality head-up display method stored in the memory and executable on the processor. When the program of the augmented reality head-up display method is executed by the processor, the steps of the augmented reality head-up display method as described above can be implemented.

[0014] The present application also provides a readable storage medium, which is a computer-readable storage medium, on which is stored a program for implementing an augmented reality head-up display method, and the program for implementing the augmented reality head-up display method is executed by a processor to implement the steps of the augmented reality head-up display method as described above.

[0015] The present application also provides a computer program product, including a computer program, which implements the steps of the augmented reality head-up display method as described above when executed by a processor.

[0016] The present application proposes an augmented reality head-up display method, an augmented reality device, and a readable storage medium. The technical solution of the present application is to establish a communication link connection with the vehicle, obtain the vehicle's operating condition information and driving environment information through the communication link; determine whether the vehicle has entered a risk scene based on the operating condition information and driving environment information; after determining that the vehicle is in a risk scene, generate risk warning image content corresponding to the risk scene, and present the risk warning image content on the current window of the augmented reality device, so as to push the risk warning message to the user in a timely manner, and solve the problem of safety hazards caused by the user's distraction while driving and incomplete observation of the road conditions. For example, when the user drives through an intersection without a signal light and turns his head to the right to observe the road conditions, the vehicle detects a collision risk with other vehicles approaching quickly on the left. At this time, the present application reminds the user of the traffic danger on the left in a timely manner without turning his head, so as to avoid accidents.

[0017] Compared with the current vehicle-mounted HUD which only displays simple information such as vehicle speed and navigation line, this application determines whether the vehicle has entered a risk scenario based on the vehicle's operating condition information and driving environment information. After determining that the vehicle is in a risk scenario, it generates risk warning image content corresponding to the risk scenario. The risk warning image content varies with different risk scenarios, thereby making the content of the augmented reality head-up display rich and variable. At the same time, it can push risk warning messages to users in a timely manner, solving the problem of safety hazards caused by users' distraction while driving and incomplete observation of road conditions, thereby improving the intelligence of the vehicle driving process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 This is a flowchart of the first embodiment of the augmented reality head-up display method of the present application; Figure 2 This is a schematic diagram of the field of view FOV of the augmented reality head-up display in the embodiment of the present application; Figure 3 This is a schematic diagram of a message push scenario of an augmented reality head-up display in an embodiment of the present application; Figure 4A schematic diagram of a scenario in which an augmented reality device performs risk warning prompts in an embodiment of the present application; Figure 5 This is a schematic diagram of the module structure of the augmented reality head-up display device according to an embodiment of the present application; Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the augmented reality device in this embodiment.

[0021] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0023] Embodiment 1 With the development of automobiles, head-up display and augmented reality technology have been applied in automobiles. Combining head-up display and augmented reality technology in automobiles allows drivers to see images that integrate virtual images with real environments without lowering their heads, providing convenience for drivers to drive. However, currently, only augmented reality head-up display technology can be used in automobiles to display navigation routes, vehicle speed, etc., and the degree of intelligence in the vehicle driving process is low.

[0024] Based on this, please refer to Figure 1 This embodiment provides an augmented reality head-up display method, which is applied to an augmented reality device, and the method includes: Step S10, establishing a communication link connection with the vehicle, and obtaining the operating condition information and driving environment information of the vehicle through the communication link; In this embodiment, the operating condition information refers to the working condition of the vehicle during driving. For example, according to the movement form of the car, it may include driving conditions such as acceleration, deceleration, turning, uphill and downhill, and parking. According to the driver's control method, it may include gear shifting, coasting (gear disengagement coasting, neutral coasting, acceleration coasting, parking coasting), braking (emergency braking, speed control braking, brake braking), throttle speed control, steering, reversing and other working conditions. Driving environment information refers to the road environment information in the surrounding area of ​​the vehicle (the range area within a preset distance from the vehicle), including the distance between the vehicle and surrounding pedestrians or other vehicles and other traffic objects, and the road conditions on which the vehicle is driving (for example, whether the road surface is uneven, whether the road surface is slippery, etc.). Therefore, it is easy to understand that the current driving scene of the vehicle can be analyzed based on the current operating condition information and driving environment information of the vehicle, so as to facilitate the subsequent determination of whether the vehicle enters a risk scene based on the current driving scene.

[0025] In this embodiment, the operating condition information and driving environment information of the vehicle can be monitored by on-board sensors, for example, steering sensors, wheel sensors, side slip sensors, acceleration sensors, throttle and brake sensors, millimeter-wave radars, cameras, pressure sensors and other on-board sensors can be used to monitor the operating condition information and driving environment information of the vehicle, and the monitored operating condition information and driving environment information can be sent to the augmented reality device.

[0026] Step S20, determining whether the vehicle enters a risk scenario based on the operating condition information and the driving environment information; In this embodiment, the risk scenario refers to a scenario where there is a high probability that the vehicle will have a safety accident such as a collision. The risk scenario includes but is not limited to driver fatigue driving, speeding, driving on a severely bumpy road, vehicle skidding, wheel locking, AEB (Autonomous Emergency Braking) activation, ESP (Electronic Stability Program) activation, and the driver's hands are off the steering wheel for a preset period of time. It is easy to understand that when the vehicle is not in a risk scenario, it means that the vehicle is in a safe scenario. It should be noted that when the vehicle collides, and within a preset time after the collision, for example, the preset time is 60 seconds, it still falls into the category of risk scenarios.

[0027] Step S30, after determining that the vehicle is in a risk scenario, generating risk warning image content corresponding to the risk scenario, and presenting the risk warning image content on the current window of the augmented reality device.

[0028] Exemplarily, the risk warning image content includes at least one of risk image identification information, location distance information between the vehicle and the target object, environmental image information of the area corresponding to the target object, and risk avoidance response strategy information.

[0029] The risk warning image content can be presented through the dynamic display area of ​​the current window of the augmented reality device, where the dynamic display area serves as an area for the driver to observe the traffic information of the real environment. When pedestrians or other vehicles pass in front of the vehicle and there is a potential safety risk, the system of the augmented reality device can sense the position of the pedestrian through sensors and mark the pedestrian. The warning information is superimposed and displayed in the form of a dynamic mark in the area where it overlaps with the real pedestrian through the augmented reality device, prompting the driver to observe and avoid it to avoid accidents.

[0030] In this embodiment, the augmented reality device is worn on the user's head.

[0031] The current view window of the augmented reality device refers to the maximum range of XR (Extended Reality) content images that the user can see under the current head posture. The current head posture may include the spatial position and angle of the current head, wherein the angle may include a pitch angle (rotated based on the X-axis), a yaw angle (rotated based on the Y-axis), and a roll angle (rotated based on the Z-axis).

[0032] In one embodiment, the user's head posture information (i.e., current head posture) can be dynamically detected by an inertial sensor and / or camera carried by the augmented reality device itself, wherein the camera can be one or more of a TOF (Time of Flight) camera, an infrared camera, a millimeter wave camera, and an ultrasonic camera. In another embodiment, the user's head posture information can be sent to the augmented reality device in real time by other devices that are connected to the augmented reality device for communication, thereby completing the dynamic detection of the head posture information. For example, a camera installed in the activity venue where the augmented reality device is used tracks and locates the augmented reality device (or the user's head), obtains the user's head posture information, and sends the head posture information to the augmented reality device in real time, so that the augmented reality device can obtain the dynamically detected head posture information in real time.

[0033] In this embodiment, it is known to those skilled in the art that, for extended reality technology, in order to simulate the changes in human eye sensory perception in the real world and enhance the user's sense of immersion in extended reality content, the visual field images that the user can see are often different under different head posture information. The current window refers to the visual field window that can be seen corresponding to the current head posture (different head posture information corresponds to different windows). In other words, under the current head posture (i.e., a specific eye position), the maximum range of the XR content image that the user can see is the user's current window image. It is easy for those skilled in the art to understand that during the content display process of the augmented reality device, the user's head posture information can change in real time, and the augmented reality device can collect or obtain the user's head posture information in real time to update the current window.

[0034] At present, the in-vehicle AR HUD (augmented reality head-up display) projects the image onto the film on the windshield and reflects it to the human eye. Due to optical limitations, the display area is only a small area in front of the driver, with a horizontal FOV of 20° and a vertical FOV of 7°. Among them, the field of view angle FOV refers to the range that the lens can cover (objects exceeding this angle will not be collected in the lens). The range of scenery that a camera lens can cover is usually expressed in angles. This angle is called the field of view angle FOV (angle of view), such as Figure 2 shown.

[0035] As more and more car companies begin to connect and adapt AR / VR devices to their own new energy vehicles. Compared with the in-vehicle HUD, augmented reality devices (such as AR glasses) have a larger FOV, a wider virtual image distance, and can display in 3D. Moreover, they can display 360° with the head without being restricted by the screen, making them more suitable for intelligent driving assistance. Figure 3 shown.

[0036] That is to say, compared with AR HUD, augmented reality devices (such as AR glasses) can rotate with the head, and the display screen is always displayed in front of the user, so that the user can pay more attention to the prompt information in a timely manner. Based on this, the embodiment of the present application proposes an augmented reality head-up display method, an augmented reality device and a readable storage medium. The technical solution of the present application is to establish a communication link connection with the vehicle, and obtain the vehicle's operating condition information and driving environment information through the communication link; determine whether the vehicle has entered a risk scene based on the operating condition information and driving environment information; after determining that the vehicle is in a risk scene, generate risk warning image content corresponding to the risk scene, and present the risk warning image content on the current window of the augmented reality device, so as to push the risk warning message to the user in time, and solve the problem of safety hazards caused by users' distraction while driving and incomplete observation of road conditions. For example, when a user drives through an intersection without traffic lights and turns right to observe the road conditions, the vehicle detects a collision risk with other vehicles approaching quickly on the left. At this time, the present application provides information reminders on the current window of the augmented reality device, such as Figure 4 As shown, the user can be informed of the traffic danger on the left in time without turning his head, thus avoiding accidents.

[0037] Compared with the current vehicle-mounted HUD which only displays simple information such as vehicle speed and navigation line, the embodiment of the present application determines whether the vehicle has entered a risk scenario based on the vehicle's operating condition information and driving environment information. After determining that the vehicle is in a risk scenario, a risk warning image content corresponding to the risk scenario is generated. The risk warning image content varies with different risk scenarios, thereby making the content of the augmented reality head-up display rich and variable. At the same time, it is possible to push risk warning messages to users in a timely manner, thereby solving the problem of safety hazards caused by users' distraction while driving and incomplete observation of road conditions, thereby improving the intelligence level of the vehicle driving process.

[0038] In an practicable manner, the step of determining whether the vehicle enters a risk scenario based on the operating condition information and the driving environment information includes: Step A10, determining a risk probability value of a collision between the vehicle and a target object according to the operating condition information and the driving environment information, wherein the target object is a pedestrian, a pothole on the road, a building, or another vehicle whose distance from the vehicle is less than or equal to a preset distance; Step A20, if the risk probability value is less than a preset probability threshold, determining that the vehicle is not in a risk scenario; Step A30: If the risk probability value is greater than or equal to a preset probability threshold, it is determined that the vehicle is in a risk scenario.

[0039] If the risk probability value is greater than or equal to the preset probability threshold, it is determined that the vehicle is in a risk scenario. If the risk probability value is less than the preset probability threshold, it is determined that the vehicle is in a safe scenario. The preset probability threshold can be set by those skilled in the art according to actual conditions and is not specifically limited in this embodiment.

[0040] In this embodiment, the current driving scene of the vehicle can be determined according to the operating condition information and the driving environment information, and then it is determined whether the vehicle enters the risk scene according to the current driving scene. It can be understood that different driving scenes correspond to different risk probability values ​​of traffic accidents. For example, the current driving scene is that the vehicle is driving at high speed or overspeed, and the distance between the vehicle and the vehicle in front or pedestrians is less than the safe distance, and the risk probability value of traffic accidents is high. The current driving scene is that the vehicle is driving at a speed required by traffic control (not overspeeding) and keeps a safe distance from the vehicle in front or pedestrians, and the risk probability value of traffic accidents is low. For another example, the current driving scene is that the vehicle is driving on a bumpy road and there are many obstacles around, and the risk probability value of traffic accidents is high. The current driving scene is that the vehicle is driving on a flat road and there are few obstacles around, and the risk probability value of traffic accidents is low. For another example, the current driving scene is that the vehicle is driving on a sharp bend or steep slope road, and the risk probability value of traffic accidents is high. The current driving scene is that the vehicle is driving on a straight road, and the risk probability value of traffic accidents is low. For another example, the current driving scene is that the tires of the vehicle are locked or skidding, and the risk probability value of a traffic accident is high. It should be noted that the above examples of the current driving scene are only helpful for understanding the present embodiment and do not constitute a limitation on the present embodiment.

[0041] This embodiment dynamically obtains the vehicle's current operating condition information and driving environment information, determines the vehicle's current driving scenario based on the operating condition information and driving environment information, and analyzes the risk probability value of a traffic accident corresponding to the current driving scenario, thereby accurately detecting whether the vehicle has entered a risk scenario.

[0042] In addition, those skilled in the art can understand that the current operating condition information and driving environment information of the vehicle can be comprehensively considered to identify other vehicles, pedestrians or other obstacles that may collide with the vehicle, predict the collision or the trend of collision, evaluate the potential risk of collision of the vehicle, and analyze and obtain the risk probability value of traffic accident of the vehicle in the current driving scene. In a possible embodiment, the step of determining the risk probability value of traffic accident corresponding to the current driving scene includes: Step B10, determining a traffic object within a preset range around the vehicle according to the driving environment information, and determining a time distance between the vehicle and the traffic object according to the operating condition information; It should be noted that the time interval between a vehicle and a traffic object refers to the estimated time interval for the vehicle and the traffic object to arrive at the same location, or the estimated time interval for the vehicle and the traffic object to collide in the future based on the movement trend of the vehicle and the movement trend of the traffic object. The traffic object is a car, a person, or an animal around the vehicle. The vehicle's speed and direction are detected by the vehicle's onboard sensors to predict the vehicle's movement trend, and the vehicle's speed and direction are detected by the onboard sensors to predict the movement trend of the traffic object. The time interval between the vehicle and the traffic object is determined based on the movement trend of the vehicle and the movement trend of the traffic object.

[0043] Step B20, evaluating the risk probability value of the vehicle having a traffic accident according to the time interval.

[0044] In this embodiment, the risk probability value is evaluated by time distance (i.e., the evaluation standard of the risk level). This embodiment considers the complexity of the traffic environment in which the vehicle is located, collects the driving environment information of the vehicle through the on-board sensor, and determines the time distance between the vehicle and the traffic object based on the driving environment information and the operating condition information, and finally evaluates the risk probability value of the vehicle having a traffic accident based on the time distance, abandoning the previous direct evaluation of the risk probability value based on the distance between the vehicle and the obstacle. This embodiment also combines the movement speed and movement direction of the obstacle for comprehensive evaluation, thereby avoiding misjudgment and omission of potential collision risks, and further more accurately detecting whether the vehicle has entered a risk scene.

[0045] Exemplarily, the step of presenting the risk warning image content on the current window of the augmented reality device includes: Step C10: presenting the risk warning image content in a preset target area on the current window of the augmented reality device, wherein the preset target area is an area that matches the orientation of the target object.

[0046] For example, when a user is driving while wearing an augmented reality device, if the vehicle's radar sensor detects an accident risk on the left side (the distance between vehicles is too close, there is a risk of vehicle collision), a reminder will be given in the area on the side corresponding to the accident on the user's augmented reality device, such as Figure 5 shown.

[0047] This embodiment presents the risk warning image content in a preset target area on the current window of the augmented reality device, wherein the preset target area is an area that matches the orientation of the target object, thereby realizing a special information prompt method to push the message to the user in a timely and accurate manner, thereby solving the problem of the user being distracted while driving and not fully observing the road conditions, which may lead to the risk of safety accidents. For example, when the user passes through an intersection without a signal light and turns his head to the right to observe the road conditions, the vehicle detects that there is a vehicle approaching quickly on the left and there is a risk of collision. A reminder can be given in the left display area of ​​the augmented reality device, so that the user can be aware of the traffic danger on the left in a timely manner without turning his head, and take subsequent measures to avoid accidents.

[0048] In a possible implementation, before the step of generating risk warning image content corresponding to the risk scenario, the method further includes: Step D10, determining the probability interval in which the risk probability value lies; Step D20, according to the probability interval, querying from a preset scenario type mapping table to obtain the risk level mapped to the probability interval; Step D30, if the mapped risk level belongs to the first risk level, executing: the step of generating the risk warning image content corresponding to the risk scenario; Step D40, if the mapped risk level belongs to the second risk level, control the vibration module of the augmented reality device to generate vibration of a preset duration, and execute: the step of generating the risk warning image content corresponding to the risk scenario, wherein the second risk level is greater than the first risk level.

[0049] In this embodiment, the scenario type mapping table has multiple probability intervals, and each probability interval maps a type of risk scenario, and different probability intervals map different types of risk scenarios. It is easy to understand that different risk scenario types correspond to different risk levels. That is to say, different risk scenario types have different risk probabilities of traffic accidents. For example, the risk probability values ​​corresponding to risk scenarios such as driver fatigue driving, vehicle high-speed driving, and vehicle driving on a violently bumpy road surface can be divided into a first probability interval, and the risk probability values ​​corresponding to risk scenarios such as the probability value of the predicted vehicle collision is greater than the first preset value and less than the second preset value, the vehicle skids, the wheel locks, and the vehicle overspeeding are divided into a second probability interval, and the risk probability values ​​corresponding to the probability value of the predicted vehicle collision is greater than the second preset value, AEB is activated, and ESP is activated are divided into a third probability interval, and the risk probability value corresponding to the vehicle being in collision is divided into a fourth probability interval. Among them, it is easy to understand that the first probability interval is smaller than the second probability interval, the second probability interval is smaller than the third probability interval, and the third probability interval is smaller than the fourth probability interval. It should be noted that the above examples of the division of probability intervals are only helpful for understanding this embodiment and do not constitute a limitation on the division of probability intervals.

[0050] This embodiment determines the probability interval in which the risk probability value is located, and based on the probability interval, queries the risk level mapped by the probability interval from a preset scene type mapping table, so that after determining that the vehicle has entered a risk scene, the current driving scene is accurately mapped to the corresponding risk level, thereby facilitating the subsequent output of early warning prompt information corresponding to the mapped risk level. For example, when the risk level is at a higher level, the vibration module of the augmented reality device is controlled to vibrate to increase the danger warning to the driver, thereby improving driving safety.

[0051] In an practicable manner, after the step of querying and obtaining the risk level of the probability interval mapping from the preset scenario type mapping table, the method further includes: Step E10, if the mapped risk level belongs to the third risk level, generate response strategy information for avoiding collision between the vehicle and the target object, and send the response strategy information to the vehicle, wherein the response strategy information is used to trigger the vehicle to actively intervene in executing a response strategy to avoid collision risk, and the third risk level is greater than the second risk level.

[0052] In this embodiment, it should be noted that the third risk level is greater than the second risk level. The third risk level should be the highest risk level.

[0053] This embodiment generates response strategy information for avoiding collision between the vehicle and the target object when it is determined that the mapped risk level is the third risk level, and sends the response strategy information to the vehicle, wherein the response strategy information is used to trigger the vehicle to actively intervene in the execution of the response strategy for avoiding collision risk, so that when it is identified that the current risk level belongs to the highest level, the driver is made aware through early warning prompts and the like that it may be too late to take remedial measures, and the vehicle actively intervenes to perform intelligent unmanned driving, thereby urgently avoiding the vehicle driving risk and improving driving safety.

[0054] Embodiment 2 Based on the first embodiment of the present application, in another embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the step of generating the response strategy information for avoiding the collision between the vehicle and the target object includes: Step F10, obtaining the type of the target object; Step F20: generating, according to the type of the target object, response strategy information for avoiding a collision between the vehicle and the target object.

[0055] The types of the target objects may include potholes, pedestrians, vehicles, and buildings, etc. The response strategy information may be determined according to the type of the target object. It is understandable that the response strategy information corresponds to the type of the target object, and the obstacle avoidance decision corresponding to the type of the target object is pre-calibrated by those skilled in the art and stored in the system of the augmented reality device.

[0056] In one embodiment, if the target object is identified as a pothole, the response strategy information is to slow down and pass; if the target object is identified as a pedestrian, the response strategy information is to slow down and brake, and wait for the pedestrian to walk a preset distance before controlling the vehicle to continue driving; if the target object is identified as a stationary vehicle, the response strategy information is to re-plan the driving route or slow down and brake; if the target object is identified as a moving vehicle, it is determined whether the driving direction of the vehicle is consistent with the driving direction of this vehicle. If it is consistent, the following mode is activated to follow the moving vehicle. If it is inconsistent, the deceleration and braking are performed, and the vehicle is controlled to continue driving after the vehicle has driven a preset distance.

[0057] In a possible implementation manner, the step of generating, according to the type of the target object, the response strategy information for avoiding a collision between the vehicle and the target object includes: Step G10: if the type of the target object is the pothole, determining the response strategy information for avoiding the collision between the vehicle and the target object is: decelerating the vehicle to a speed less than a preset speed threshold; Step G20: If the type of the target object is a pedestrian, a building or another vehicle, the response strategy information for avoiding a collision between the vehicle and the target object is determined as: vehicle deceleration and braking.

[0058] It should be noted that the pothole obstacle represents a pothole in the road.

[0059] In this embodiment, when the target object is determined to be a pothole, the strategy information for avoiding collision between the vehicle and the target object is determined as: the vehicle is decelerated to a speed less than a preset speed threshold. When the target object is a pedestrian, a building or another vehicle, the strategy information for avoiding collision between the vehicle and the target object is determined as: the vehicle is decelerated and braked, thereby generating a targeted collision avoidance strategy according to the current obstacle type, thereby improving the vehicle's obstacle avoidance capability in the face of different obstacle types, and further improving the intelligence and safety of the augmented reality device in guiding the vehicle.

[0060] Embodiment 3 The embodiment of the present invention also provides an augmented reality head-up display device, please refer to Figure 5 The augmented reality head-up display device is applied to an augmented reality device, and the device comprises: An acquisition module 10 is configured to establish a communication link connection with the vehicle and acquire the operating condition information and driving environment information of the vehicle through the communication link; A determination module 20, configured to determine whether the vehicle enters a risk scenario based on the operating condition information and the driving environment information; The presentation module 30 is configured to generate risk warning image content corresponding to the risk scenario after determining that the vehicle is in a risk scenario, and present the risk warning image content on the current window of the augmented reality device.

[0061] Optionally, the determination module 20 is further configured to: Determine, based on the operating condition information and the driving environment information, a risk probability value of a collision between the vehicle and a target object, wherein the target object is a pedestrian, a pothole on the road surface, a building, or another vehicle whose distance from the vehicle is less than or equal to a preset distance; If the risk probability value is less than a preset probability threshold, it is determined that the vehicle is not in a risk scenario; If the risk probability value is greater than or equal to a preset probability threshold, it is determined that the vehicle is in a risk scenario.

[0062] Optionally, the presentation module 30 is further configured to: The risk warning image content is presented in a preset target area on the current window of the augmented reality device, wherein the preset target area is an area that matches the orientation of the target object.

[0063] Optionally, the determination module 20 is further configured to: Determining the probability interval in which the risk probability value lies; According to the probability interval, query and obtain the risk level mapped by the probability interval from a preset scenario type mapping table; If the mapped risk level belongs to the first risk level, executing: the step of generating the risk warning image content corresponding to the risk scenario; If the mapped risk level belongs to the second risk level, the vibration module of the augmented reality device is controlled to generate vibration of a preset duration, and the step of generating risk warning image content corresponding to the risk scenario is executed, wherein the second risk level is greater than the first risk level.

[0064] Optionally, the determination module 20 is further configured to: If the mapped risk level belongs to the third risk level, response strategy information for avoiding collision between the vehicle and the target object is generated and sent to the vehicle, wherein the response strategy information is used to trigger the vehicle to actively intervene in executing a response strategy to avoid collision risk, and the third risk level is greater than the second risk level.

[0065] Optionally, the determination module 20 is further configured to: Obtaining the type of the target object; According to the type of the target object, response strategy information for avoiding a collision between the vehicle and the target object is generated.

[0066] Optionally, the determination module 20 is further configured to: If the type of the target object is the pothole on the road, the response strategy information for avoiding the collision between the vehicle and the target object is determined as follows: the vehicle is decelerated to a speed less than a preset speed threshold; If the type of the target object is a pedestrian, a building or another vehicle, the response strategy information for avoiding a collision between the vehicle and the target object is determined as: vehicle deceleration and braking.

[0067] Optionally, the risk warning image content includes at least one of risk image identification information, location distance information between the vehicle and the target object, environmental image information of an area corresponding to the target object, and risk avoidance response strategy information.

[0068] The augmented reality head-up display device provided in the embodiment of the present invention adopts the augmented reality head-up display method in the above-mentioned embodiment 1 or embodiment 2, which can solve the problem of low intelligence level in the vehicle driving process. Compared with the prior art, the beneficial effects of the augmented reality head-up display device provided in the embodiment of the present invention are the same as the beneficial effects of the augmented reality head-up display method provided in the above-mentioned embodiment, and other technical features in the augmented reality head-up display device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0069] Embodiment 4 An embodiment of the present invention provides an augmented reality device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the augmented reality head-up display method in the above-mentioned embodiment 1.

[0070] Reference below Figure 6 , which shows a schematic diagram of the structure of an augmented reality device suitable for implementing the embodiment of the present disclosure. The augmented reality device in the embodiment of the present disclosure includes but is not limited to an augmented reality (Augmented Reality)—AR device (such as AR glasses or AR helmet). Figure 6 The augmented reality device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0071] like Figure 6 As shown, the augmented reality device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). Various programs and data required for the operation of the augmented reality device are also stored in RAM 1004. The processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface is also connected to the bus 1005.

[0072] Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the augmented reality device to communicate wirelessly or wired with other devices to exchange data. Although the figures show an augmented reality device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0073] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0074] The augmented reality device provided by the present invention adopts the augmented reality head-up display method in the above embodiment, which can solve the problem of low intelligence in the vehicle driving process. Compared with the prior art, the beneficial effects of the augmented reality device provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality head-up display method provided by the above embodiment, and other technical features in the augmented reality device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0075] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0076] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0077] Embodiment 5 An embodiment of the present invention provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the augmented reality head-up display method in the above embodiment.

[0078] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0079] The computer-readable storage medium may be included in the augmented reality device; or may exist independently without being assembled into the augmented reality device.

[0080] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the augmented reality device, the augmented reality device: establishes a communication link connection with the vehicle, and obtains the operating condition information and driving environment information of the vehicle through the communication link; determines whether the vehicle enters a risk scene based on the operating condition information and driving environment information; after determining that the vehicle is in a risk scene, generates risk warning image content corresponding to the risk scene, and presents the risk warning image content on the current window of the augmented reality device.

[0081] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0082] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0084] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned augmented reality head-up display method, which can solve the problem of low intelligence in the vehicle driving process. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality head-up display method provided by the above-mentioned embodiment 1 or embodiment 2, and will not be repeated here.

[0085] Embodiment 6 An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the above-mentioned augmented reality head-up display method when executed by a processor.

[0086] The computer program product provided by the present application can solve the problem of low intelligence level in the vehicle driving process. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality head-up display method provided by the above-mentioned embodiment 1 or embodiment 2, and will not be repeated here.

[0087] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. An augmented reality head-up display method, characterized in that: The augmented reality head-up display method is applied to an augmented reality device, and the method comprises: Establishing a communication link connection with the vehicle, and obtaining operating condition information and driving environment information of the vehicle through the communication link; Determining whether the vehicle enters a risk scenario based on the operating condition information and the driving environment information; After determining that the vehicle is in a risk scenario, a risk warning image content corresponding to the risk scenario is generated, and the risk warning image content is presented on a current window of the augmented reality device.

2. The augmented reality head-up display method according to claim 1, characterized in that: The step of determining whether the vehicle enters a risk scenario according to the operating condition information and the driving environment information includes: Determine, based on the operating condition information and the driving environment information, a risk probability value of a collision between the vehicle and a target object, wherein the target object is a pedestrian, a pothole on the road surface, a building, or another vehicle whose distance from the vehicle is less than or equal to a preset distance; If the risk probability value is less than a preset probability threshold, it is determined that the vehicle is not in a risk scenario; If the risk probability value is greater than or equal to a preset probability threshold, it is determined that the vehicle is in a risk scenario.

3. The augmented reality head-up display method according to claim 2, characterized in that: The step of presenting the risk warning image content on the current window of the augmented reality device comprises: The risk warning image content is presented in a preset target area on the current window of the augmented reality device, wherein the preset target area is an area that matches the orientation of the target object.

4. The augmented reality head-up display method according to claim 2, characterized in that: Before the step of generating risk warning image content corresponding to the risk scenario, the method further includes: Determining the probability interval in which the risk probability value lies; According to the probability interval, query and obtain the risk level mapped by the probability interval from a preset scenario type mapping table; If the mapped risk level belongs to the first risk level, executing: the step of generating the risk warning image content corresponding to the risk scenario; If the mapped risk level belongs to the second risk level, the vibration module of the augmented reality device is controlled to generate vibration of a preset duration, and the step of generating risk warning image content corresponding to the risk scenario is executed, wherein the second risk level is greater than the first risk level.

5. The augmented reality head-up display method according to claim 4, characterized in that: After the step of querying and obtaining the risk level of the probability interval mapping from the preset scenario type mapping table, the method further includes: If the mapped risk level belongs to the third risk level, response strategy information for avoiding collision between the vehicle and the target object is generated and sent to the vehicle, wherein the response strategy information is used to trigger the vehicle to actively intervene in executing a response strategy to avoid collision risk, and the third risk level is greater than the second risk level.

6. The augmented reality head-up display method according to claim 5, characterized in that: The step of generating the response strategy information for avoiding the collision between the vehicle and the target object comprises: Obtaining the type of the target object; According to the type of the target object, response strategy information for avoiding a collision between the vehicle and the target object is generated.

7. The augmented reality head-up display method according to claim 6, characterized in that: The step of generating, according to the type of the target object, the response strategy information for avoiding the collision between the vehicle and the target object comprises: If the type of the target object is the road pothole, the response strategy information for avoiding the collision between the vehicle and the target object is determined as follows: the vehicle is decelerated to a speed less than a preset speed threshold; If the type of the target object is a pedestrian, a building or another vehicle, the response strategy information for avoiding a collision between the vehicle and the target object is determined as: vehicle deceleration and braking.

8. The augmented reality head-up display method according to any one of claims 2 to 7, characterized in that: The risk warning image content includes at least one of risk image identification information, location distance information between the vehicle and the target object, environmental image information of the area corresponding to the target object, and risk avoidance response strategy information.

9. An augmented reality device, characterized in that: The augmented reality device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the augmented reality head-up display method according to any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which is stored a program for implementing the augmented reality head-up display method, and the program for implementing the augmented reality head-up display method is executed by a processor to implement the steps of the augmented reality head-up display method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Cognitive drive assist with variable warning for automated vehicles

    CN107810127A

  • Multi-factor fused dangerous driving behavior grading warning method

    CN109035718A

  • Vehicle auxiliary braking method, device, system and terminal

    CN110422151A

  • Vehicle collision prewarning method, vehicle-mounted terminal and server

    CN110428663A

  • Self-adaptive anti-collision grading early warning method based on forward obstacle recognition

    CN111696387A

Cited By

  • In-vehicle interactive aerial imaging display system for driver

    CN119395892A

  • Head-up display method, device and equipment

    CN121608591A