Vehicle auxiliary driving method, device and system based on head-mounted AR device

The head-mounted AR device in vehicle assisted driving systems addresses the synchronization issue of traditional systems by displaying target images in the driver's field of view, improving safety and perception by reducing distractions and enhancing environmental awareness.

CN119078869APending Publication Date: 2024-12-06GREAT WALL MOTOR CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411333530.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-24
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional vehicle assisted driving systems rely on on-board cameras and sensors that do not synchronize information with the driver's line of sight, causing distraction and prolonged reaction times, leading to safety issues due to visual blind spots and environmental misperception.

Method used

A vehicle assisted driving method using a head-mounted AR device that collects environmental information through sensors and displays target images directly in the driver's field of view to assist in identifying driving risks, enhancing perception and reaction speed.

Benefits of technology

Improves driving safety by providing real-time, intuitive assistance that reduces the need for the driver to divert attention from the road, addressing visual blind spots and enhancing environmental awareness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119078869A_ABST
    Figure CN119078869A_ABST
Patent Text Reader

Abstract

The invention provides a vehicle auxiliary driving method, device and system based on a head-mounted AR device, and is applied to the technical field of intelligent driving, and the method comprises the steps: responding to the starting of a vehicle, collecting environment information through an environment sensor of the vehicle, and determining whether a driving risk exists or not based on the environment information; and in response to the existence of the driving risk, displaying a target image through the head-mounted AR device, the target image being an image for assisting a driver in identifying the driving risk. According to the method, the perception ability and the response speed of the driver to the environment can be greatly improved, and the method for displaying the target image through the head-mounted AR equipment not only improves the driving safety, but also enhances the driving experience. Therefore, the driver can obtain real-time and visual driving assistance, so that the vehicle can be driven more safely, and the driving safety and convenience are greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a vehicle assisted driving method, device and system based on a head-mounted AR device. Background Art

[0002] Traditional vehicle assisted driving systems usually rely on on-board cameras, radars and other sensors to collect environmental information and provide warnings or auxiliary information to the driver through the on-board display or audio system. However, the information display may not be synchronized with the driver's actual line of sight, causing the driver to be distracted and the reaction time to be prolonged, thus affecting driving safety. Summary of the invention

[0003] In view of this, the purpose of this application is to propose a vehicle assisted driving method, device and system based on a head-mounted AR device to avoid distracting the driver when providing assisted driving information to the driver and ensure driving safety.

[0004] Based on the above objectives, the present application provides a vehicle assisted driving method based on a head-mounted AR device, comprising:

[0005] In response to the vehicle being started, collecting environmental information through an environmental sensor of the vehicle and determining whether there is a driving risk based on the environmental information;

[0006] In response to the existence of driving risks, a target image is displayed through a head-mounted AR device, where the target image is an image that assists the driver in identifying driving risks.

[0007] Based on the same inventive concept, the present application also provides a head-mounted AR device, including:

[0008] A camera module configured to capture an image of the driver's field of view;

[0009] A spatial information acquisition module configured to acquire spatial information within the driver's field of view;

[0010] A display screen is configured to display a target image to a driver, wherein the target image is an image for assisting the driver in identifying driving risks.

[0011] Based on the same inventive concept, the present application also provides a vehicle assisted driving system based on a head-mounted AR device, comprising a vehicle and the above-mentioned head-mounted AR device;

[0012] The vehicle comprises:

[0013] An environmental sensor configured to collect environmental information;

[0014] A communication module, configured to connect the vehicle with a head-mounted AR device for information interaction;

[0015] A processor is configured to, in response to vehicle startup, collect environmental information through the vehicle's environmental sensors and determine whether there is a driving risk based on the environmental information; if it is determined that there is a driving risk, send a target image to a head-mounted AR device for display through a communication module, wherein the target image is an image determined based on the environmental information, the field of view image, and the spatial information to assist the driver in identifying the driving risk.

[0016] From the above, it can be seen that the vehicle assisted driving method, device and system based on head-mounted AR device provided by the present application include: when the vehicle is started, environmental information is collected through the vehicle's environmental sensor and whether there is a driving risk based on the environmental information; if there is a driving risk, the target image is displayed through the head-mounted AR device, and the target image is an image that assists the driver in identifying the driving risk; the head-mounted AR device can superimpose the target image in the driver's field of view, so that the driver does not need to distract his attention to check other devices such as the dashboard or the central control screen or can intuitively see the blind spot of the field of view, so as to help the driver better identify and understand the current driving risk and improve the driver's perception and reaction speed to the environment. The method of displaying the target image through a head-mounted AR device improves driving safety, so that the driver can drive the vehicle more safely. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a flow chart of a vehicle assisted driving method based on a head-mounted AR device according to an embodiment of the present application;

[0019] Figure 2 The process diagram of the vehicle assisted driving method based on the head-mounted AR device in the embodiment of the present application is as follows: Figure 1 ;

[0020] Figure 3 The process diagram of the vehicle assisted driving method based on the head-mounted AR device in the embodiment of the present application is as follows: Figure 2 ;

[0021] Figure 4 A schematic diagram of a head-mounted AR device according to an embodiment of the present application;

[0022] Figure 5A schematic diagram of a vehicle assisted driving system based on a head-mounted AR device according to an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of a vehicle assisted driving device based on a head-mounted AR device according to an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] In related technologies, with the development of science and technology, vehicle assisted driving systems have gradually become an important tool for improving driving safety and convenience. Traditional vehicle assisted driving systems usually rely on sensors such as on-board cameras and radars to collect environmental information and provide warnings or auxiliary information to the driver through the on-board display or audio system. However, these traditional methods have some limitations. The information display may not be synchronized with the driver's actual line of sight, causing the driver to be distracted and the reaction time to be prolonged, thus affecting driving safety. For example, the A-pillars (i.e., the pillars supporting the roof) on both sides of the car's front window will block the driver's line of sight, especially when passing through curves or intersections. This structure may hide pedestrians or other vehicles, increasing the risk of collision. For rearview mirror blind spots, although rearview mirrors and side mirrors can help drivers observe the situation behind and to the sides of the vehicle, there is still a certain blind spot range, especially when changing lanes or reversing, vehicles or obstacles at certain angles may not be within the visible range of the mirrors. For rear seat blind spots, rear passengers or objects may block the rear window view, affecting the driver's comprehensive observation of the situation behind the vehicle through the rearview mirror. For low positions in front of the vehicle, such as under the hood, small objects or low obstacles may be hidden, especially when driving on complex terrain or roads with obstacles. These visual blind spots may cause the driver to fail to see other vehicles, pedestrians or obstacles in time, especially when changing lanes, merging, turning left or right or reversing, which may easily lead to collision accidents; on urban roads and residential areas, there are many pedestrians and non-motor vehicles, and the visual blind spots, especially the A-pillar blind spots and the front low blind spots may cause the driver to fail to detect pedestrians or children crossing the road in time, increasing the risk of hitting pedestrians; on highways, the speed is fast, and the visual blind spots, especially the rearview mirror blind spots, may cause the driver to fail to notice other high-speed vehicles when changing lanes, which may easily cause high-speed collisions; in the parking process, especially when reversing into a parking space, the rearview blind spots and the roof blind spots may cause the driver to be unable to fully grasp the situation around the vehicle, increasing the risk of scratches or collisions with obstacles; in emergency avoidance situations that require quick response, the visual blind spots may cause the driver to react untimely and fail to effectively avoid sudden dangers, such as vehicles that suddenly cut in, road obstacles, etc.

[0028] In recent years, augmented reality (AR) technology has been widely used in many fields because it can directly superimpose virtual information on the user's real field of view. As a new type of display device, head-mounted AR devices can project data and images directly into the driver's field of view, thereby providing more intuitive and timely auxiliary information to enhance the driver's environmental perception ability. Although head-mounted AR devices have shown great potential in vehicle assisted driving, how to effectively use head-mounted AR devices to improve driving safety and assistance effects is still a problem worth studying.

[0029] Based on the above problems, the applicant found that: when the vehicle is started, environmental information is collected through the vehicle's environmental sensors and whether there is a driving risk is determined based on the environmental information; if there is a driving risk, the target image is displayed through a head-mounted AR device, and the target image is an image that assists the driver in identifying driving risks; helping the driver to better identify and understand the current driving risks. The head-mounted AR device can superimpose the target image in the driver's field of view, so that the driver does not need to distract his attention to check other devices such as the dashboard or the central control screen or can intuitively see the blind spots of the field of view, which can greatly improve the driver's perception of the environment and reaction speed. The method of displaying the target image through a head-mounted AR device not only improves driving safety, but also enhances the driving experience. The driver can obtain real-time and intuitive driving assistance, so that he can drive the vehicle more safely, greatly improving driving safety and convenience.

[0030] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0031] The present application provides a vehicle assisted driving method based on a head-mounted AR device, such as Figure 1 As shown, in some embodiments, the method is executed by a processor of a vehicle or a processor of a head-mounted AR device. The subsequent embodiments are all illustrated by taking the processor of a vehicle as an example. The vehicle includes an environmental sensor, and the method includes:

[0032] S101, in response to vehicle startup, collecting environmental information through an environmental sensor of the vehicle and determining whether there is a driving risk based on the environmental information;

[0033] In specific implementation, when the vehicle is started, the vehicle's environmental sensors are turned on. Environmental sensors include but are not limited to cameras, lidars, millimeter-wave radars, etc., which are used to collect environmental information around the vehicle in real time. This environmental information may include but is not limited to road conditions, traffic flow, location and size of obstacles, weather conditions, etc. Based on the environmental information, it is assessed whether there is potential driving risk in the current driving environment. The assessment of driving risk is dynamic. As the vehicle moves and the environment changes, the environmental information will be continuously updated and the driving risk will be reassessed to ensure timely response to various emergencies and improve driving safety.

[0034] S102: In response to the existence of driving risk, displaying a target image through a head-mounted AR device, where the target image is an image for assisting the driver in identifying the driving risk.

[0035] In specific implementation, when driving risks are detected, further measures are taken to remind and assist the driver to identify driving risks. In this process, augmented reality (AR) technology is used to display target images to the driver to help the driver better identify and understand the current driving risks. Head-mounted AR devices can superimpose images in the driver's field of view, so that the driver does not need to divert attention to view other devices such as dashboards or central control screens, which can greatly improve the driver's perception and reaction speed of the environment. For example, if an obstacle is detected in front or in the driver's blind spot, the head-mounted AR device can directly highlight the obstacle in the driver's field of view, and may be accompanied by a warning icon or text description. The intuitive prompt can make the driver aware of the potential danger in the first time and take appropriate risk avoidance measures. In addition, head-mounted AR devices can also display more types of auxiliary information, such as navigation routes, lane keeping assistance, speed and distance warnings, etc., to help drivers better understand the surrounding environment and make correct driving decisions.

[0036] The method of displaying target images through head-mounted AR devices not only improves driving safety, but also enhances the driving experience. Drivers can get real-time and intuitive driving assistance, so that they can drive the vehicle more safely, greatly improving driving safety and convenience.

[0037] In this embodiment, the head-mounted AR device can provide real-time visual warnings of potential dangers in the environment, such as obstacles, pedestrians, etc., to reduce the risk of accidents caused by visual blind spots. The target image is superimposed in the driver's field of view, so that the driver can obtain important information without shifting his or her sight, reducing the possibility of accidents due to distraction. The driver no longer needs to frequently check the vehicle's multiple display screens. All necessary information can be directly displayed through the head-mounted AR device, simplifying the operating process and improving driving convenience. In complex traffic environments or severe weather conditions, AR technology can supplement traditional visual blind spots, such as A-pillar blind spots, rearview mirror blind spots, etc., to provide drivers with all-round visual support and more personalized driving assistance. The driver can drive the vehicle more safely, greatly improving driving safety and convenience.

[0038] In some embodiments, Figure 2 As shown, the environmental information includes an environmental image; and displaying the target image through a head-mounted AR device includes:

[0039] S201, collecting a driver's field of view image through a head-mounted AR device, and determining an image in the environment image that is different from the driver's field of view image as a target image;

[0040] In specific implementation, the driver's field of view images are collected through the head-mounted AR device, that is, the images seen by the driver at the current perspective are recorded in real time, and these collected field of view images are compared with the environmental images obtained in real time by the vehicle's environmental sensors. The environmental images can come from the on-board camera and provide a detailed view of the vehicle's surroundings. By analyzing and comparing the two images, the differences between the two are identified. The differences may include but are not limited to obstacles, pedestrians, vehicles, etc. that are blocked by the vehicle structure, or information that the driver cannot clearly identify due to factors such as perspective and light. These images with differences are determined as target images.

[0041] S202: Display the target image in the driver's field of vision through a head-mounted AR device, so that the image observed by the driver is consistent with the environment image.

[0042] In specific implementation, after the image with differences is determined as the target image, the target image (i.e., the element that exists in the environmental image but is missing in the driver's field of view) is superimposed and displayed in the driver's field of view through the head-mounted AR device, so that the driver can obtain additional important information that may be missed or unclear on the basis of the original vision. For example, if the environmental image shows that there is a pedestrian crossing the road in front, and the driver's field of view image does not contain this information due to the occlusion of the vehicle's A-pillar, the image of the pedestrian is overlaid on the corresponding position on the vehicle's A-pillar in the driver's field of view through the head-mounted AR device, so as to display it to the driver, ensuring that the driver can see and respond in time. The driver's environmental perception ability is greatly enhanced, and the risk of accidents caused by visual blind spots, lack of concentration, etc. is reduced. The driver can obtain more comprehensive and accurate environmental information, so as to make safer driving decisions. For another example, if the environmental image shows a pedestrian moving slowly in the blind spot of the driver's field of view, and the driver himself does not notice this due to light reasons, the head-mounted AR device will warn the driver of the existence of the pedestrian by displaying the image of the pedestrian in the driver's field of view. This display may include an outline of the pedestrian, an indication of the direction of movement, and may also include color coding (such as red for danger) to further attract the driver's attention.

[0043] In this way, head-mounted AR devices can improve drivers' awareness of the environment and help them respond faster and more accurately in complex traffic environments, thereby improving driving safety.

[0044] In this embodiment, by directly superimposing elements in the environmental image (such as pedestrians and obstacles) in the driver's field of vision, the head-mounted AR device helps the driver cover the traditional visual blind spot, ensuring that the driver can see important information that may have been missed. At the same time, compared with the signal delay when the environmental image is displayed entirely on the head-mounted AR device, only the target image is displayed in the driver's field of vision through the head-mounted AR device, which can avoid the signal delay of the environmental image causing the driver to react in time. The driver can not only know the situation of the visual blind spot through the target image, but also directly observe the environment through other parts of the head-mounted AR device, which is safer. Directly displaying potential dangers (such as pedestrians crossing the road) in the driver's field of vision, especially when these dangers are not visible in the driver's field of vision, can significantly reduce the risk of accidents. Insufficient light or other visual obstacles may cause the driver to misjudge the environmental situation. The head-mounted AR device helps the driver make correct judgments and reactions by clearly displaying this information. Integrating the target image directly into the driver's natural field of vision reduces the need for the driver to switch his line of sight between multiple devices, thereby reducing the complexity of operation and potential distraction, and reducing the risk of accidents caused by visual blind spots or distraction.

[0045] In some embodiments, Figure 3 As shown, the environmental information includes point cloud data of environmental objects; and displaying the target image through a head-mounted AR device includes:

[0046] S301, performing edge feature detection on the point cloud data of the environmental object to obtain an edge feature image of the environmental object; collecting a field of view image of the driver through a head-mounted AR device, and performing edge detection on the field of view image to obtain a first edge image of the object in the field of view image;

[0047] In specific implementation, point cloud data and edge detection technology are used to enhance the visual information obtained by the driver through the head-mounted AR device. Point cloud data is a collection of data in a 3D coordinate system, which represents the surface of objects in the environment. This data can be obtained through various sensors, such as lidar or millimeter wave radar; then edge feature detection is performed on the point cloud data of environmental objects, and the surface of objects in the environment is converted into an edge feature image to identify the boundaries of objects in the image and represent the outline and structure of objects in the environment.

[0048] After the driver's field of view image is collected through the head-mounted AR device, the field of view image will undergo edge detection processing to extract the edge features of objects in the field of view, so that the edge feature image of the environment and the first edge image in the driver's field of view can be compared subsequently, thereby identifying missing or unclear objects in the field of view.

[0049] S302, determining an image in the edge feature image that is different from the first edge image as a target image;

[0050] In a specific implementation, the difference between the edge feature image of the environmental object and the first edge image in the driver's field of view is identified. These differences may indicate that certain objects in the environment are not fully observed or not observed at all in the driver's field of view. The comparison of the edge feature image with the first edge image can use difference detection techniques in image processing, such as image subtraction, to determine the difference between the two edge images. Environmental objects that the driver may not have noticed are identified, and the identified difference portion will be defined as the target image.

[0051] S303: Display the target image in the driver's field of vision through the head-mounted AR device, so that edge information of the picture observed by the driver is consistent with edge information of the edge feature image.

[0052] In specific implementation, the target image is displayed in the driver's field of view through AR (augmented reality) glasses to ensure that the edge information of the picture observed by the driver is consistent with the edge information of the edge feature image actually existing in the environment; enhance the driver's perception ability and ensure that the driver can fully understand the surrounding environment, especially important information that may not be noticed due to visual field limitations or distraction. Secondly, since environmental information includes point cloud data of environmental objects, that is, point cloud data that can be obtained by lidar or millimeter wave radar; it can avoid the problem of unclear environmental images obtained by other environmental sensors (such as cameras) due to lighting problems or extreme weather, resulting in the inability to accurately identify environmental conditions. In this way, head-mounted AR devices can improve the driver's understanding of the environment and help the driver make faster and more accurate responses in complex traffic environments, thereby improving driving safety.

[0053] In this embodiment, edge information of objects that may not be seen due to viewing angle limitations or other visual obstacles is supplemented in the driver's field of view, ensuring that the driver obtains complete visual information about the surrounding environment, thereby significantly reducing visual blind spots and improving driving intuitiveness and comfort.

[0054] In some embodiments, the environmental information includes a distance between an obstacle and a vehicle, and determining whether there is a driving risk based on the environmental information includes:

[0055] In response to determining that the distance between at least one obstacle and the vehicle is less than or equal to a first preset safety distance, determining that there is a driving risk;

[0056] In specific implementation, when it is detected that the distance between at least one obstacle and the vehicle is less than or equal to a first preset safety distance (for example, the first preset safety distance is 50 cm), a collision will occur between the obstacle and the vehicle, and it is determined that there is a driving risk.

[0057] In response to determining that the distances between all obstacles and the vehicle are greater than a first preset safety distance and less than or equal to a second preset safety distance, it is determined whether there is a driving risk based on spatial position information of the obstacles relative to the vehicle.

[0058] In specific implementation, when the distance between all obstacles and the vehicle is greater than the first preset safety distance and less than or equal to the second preset safety distance (for example, the first preset safety distance is 50 cm and the second preset safety distance is 100 cm), it indicates that the obstacles maintain a certain safety distance relative to the vehicle, but the driving risk cannot be eliminated. It is necessary to determine whether there is a driving risk based on the spatial position information of the obstacle relative to the vehicle to avoid potential collisions and reduce traffic accidents.

[0059] In this embodiment, by monitoring the distance between the obstacle and the vehicle in real time and comparing it with the preset safety distance, the potential collision risk can be identified in time, thereby warning the driver in advance to take risk avoidance measures. Setting multiple levels of safety distance (such as the first preset safety distance and the second preset safety distance) allows different response measures to be taken according to different levels of distance, making risk management more flexible and accurate, and being able to identify and respond to potential driving risks in real time, optimize driving decisions, reduce traffic accidents, and enhance the overall driving experience.

[0060] In some embodiments, determining whether there is a driving risk based on the spatial position information of the obstacle relative to the vehicle includes:

[0061] In response to determining that at least one obstacle is on the predicted driving trajectory of the vehicle, determining that there is a driving risk;

[0062] Wherein, the predicted driving trajectory is determined based on the tire steering angle.

[0063] In specific implementation, the tire steering angle refers to the deflection angle of the front wheels of the vehicle relative to the body. The tire steering angle directly affects the driving direction of the vehicle, so it is one of the key parameters for determining the predicted driving trajectory. By real-time monitoring and analyzing the steering angle, the vehicle's driving path in the future can be accurately predicted. The vehicle's predicted driving trajectory is determined by combining the vehicle's current position, speed, and tire steering angle data with the vehicle dynamics model and environmental information. The predicted driving trajectory represents the path that the vehicle may travel without changing the current driving input (such as steering angle, acceleration). The environment around the vehicle is continuously monitored through environmental sensors to identify obstacles on the road, such as pedestrians, other vehicles, roadblocks, etc. When an obstacle is detected on the predicted driving trajectory of the vehicle, it is determined that there is a driving risk; it can effectively identify and respond to various risks during driving, and improve the safety and efficiency of road traffic.

[0064] In this embodiment, the future driving path of the vehicle can be accurately predicted by real-time analysis of the tire steering angle. Obstacles on the track can be identified in advance through the predicted driving trajectory, so as to conduct a forward-looking driving risk assessment, provide the driver with sufficient reaction time, and help the driver better understand the vehicle's driving intentions and potential risks, thereby enhancing the practicality and effectiveness of the driving assistance system.

[0065] In some embodiments, determining whether there is a driving risk based on the spatial position information of the obstacle relative to the vehicle includes:

[0066] Determine the driver's real-time blind spots;

[0067] In specific implementation, the real-time blind spot can be detected based on the vehicle's environmental sensors and head-mounted AR device. For example, the head-mounted AR device can be used to collect spatial information under the driver's field of view, and the difference space is determined based on the spatial information in the environmental spatial information that is different from the spatial information under the driver's field of view, and the difference space is determined as the driver's real-time blind spot.

[0068] In response to determining that at least one obstacle is in a preset blind spot and / or a real-time blind spot of the driver, it is determined that there is a driving risk.

[0069] In specific implementation, when it is determined that at least one obstacle is in the driver's real-time blind spot or preset blind spot, the driver cannot observe the obstacle and there is a possibility of collision. Therefore, it is determined that there is a driving risk. The real-time blind spot can be detected by the vehicle's environmental sensors and head-mounted AR device, while the preset blind spot is mainly determined by factors such as vehicle design (such as the position and size of the vehicle pillars), the driver's body position (such as seat height and angle), and the driver's physical characteristics (such as height and field of vision). For example, the A-pillar (the pillars located on both sides of the vehicle's front windshield) and the C-pillar (the pillars located on both sides of the vehicle's rear window) are often the main parts that cause the blind spot. The vehicle can effectively identify the potential risks in the driver's blind spot, significantly improve driving safety, and reduce the occurrence of traffic accidents.

[0070] In this embodiment, when an obstacle is detected in the blind spot of the field of vision, it is determined that there is a driving risk, so the driver can be warned to provide the driver with sufficient time to react to avoid potential collisions, thereby significantly reducing the incidence of traffic accidents, especially in urban driving and complex road conditions.

[0071] In some embodiments, the environmental information includes environmental space information, and determining the real-time blind spot of the driver includes:

[0072] The spatial information under the driver's field of view is collected through a head-mounted AR device, and a difference space is determined according to the spatial information in the environmental spatial information that is different from the spatial information under the driver's field of view, and the difference space is determined as the driver's real-time field of view blind spot.

[0073] In specific implementation, environmental sensors (such as millimeter-wave radars and lidars) are used to collect environmental spatial information around the vehicle. The head-mounted AR device can capture and analyze the driver's line of sight and the spatial information seen in real time through sensors (such as lidars); the spatial information under the driver's field of view obtained from the head-mounted AR device is compared with the environmental spatial information collected by the environmental sensors. By identifying the difference between the two, the spatial areas that exist outside the driver's field of view can be determined. These areas are identified as "difference spaces", which are areas that the driver cannot see through direct sight and are defined as real-time blind spots. Real-time blind spots are potential risk areas because the driver cannot perceive obstacles or dynamic changes in these areas.

[0074] In this embodiment, by comparing the spatial information collected by the head-mounted AR device under the driver's field of view with the environmental spatial information of the environmental sensor, potential obstacles or dangers in the area that the driver cannot directly see can be identified, and a warning can be issued to the driver in time to avoid possible collisions or accidents. At the same time, the driver's spatial perception ability is also enhanced, improving the overall driving experience.

[0075] In some embodiments, the environmental information includes an environmental image and point cloud data of environmental objects; and determining whether there is a driving risk based on the environmental information includes:

[0076] Perform edge feature detection on the point cloud data of the environmental object to obtain an edge feature image of the environmental object; perform edge detection on the environmental image to obtain a second edge image of the object in the environmental image;

[0077] In specific implementations, point cloud data is generated by 3D scanning equipment (such as LiDAR), which records the three-dimensional position information of objects in the environment. By performing edge feature detection on these point cloud data, the precise boundaries and shapes of objects can be identified, which usually involves calculating the local curvature of each point in the point cloud data or using other edge detection algorithms to identify edges. Environmental images are usually captured by on-board cameras, providing a two-dimensional visual representation of the environment. Edge detection on these images means using image processing technology to identify the outlines of objects in the image, which helps to determine the position and shape of the object from a visual perspective.

[0078] In response to a difference between the second edge image and the edge feature image being greater than or equal to a preset difference, it is determined that there is a driving risk.

[0079] In a specific implementation, when an edge feature image extracted from the point cloud data and a second edge image obtained from the environmental image are obtained, the two are compared. If the difference between the second edge image and the edge feature image is greater than or equal to a preset difference (exemplarily, the difference between the second edge image and the edge feature image can be determined by the percentage of the number of pixels of the edge contour in the second edge image that are inconsistent with the edge feature image to the total number of pixels in the second edge image, then the preset difference can be 30%), it indicates that the environmental conditions may only appear in the point cloud data and are almost invisible in the environmental image. This may be due to factors such as lighting conditions, the color or reflective characteristics of the object. At this time, it is not easy for the driver to observe the actual situation of the environment, and it is determined that there is a driving risk.

[0080] In this embodiment, by comparing the edge features in the point cloud data and the environmental image, objects that are not clearly displayed in the environmental image due to visual limitations (such as insufficient light, color confusion, etc.) can be identified. The driver can be warned in time to pay attention to risks that may not be visually obvious. By identifying potential risks early and providing early warnings, it helps the driver take risk avoidance measures, such as slowing down or changing the driving route, thereby reducing the occurrence of accidents. In low light or bad weather, the environmental image may not provide enough information. The use of point cloud data can make up for this deficiency and ensure driving safety under various environmental conditions. It not only improves the accuracy and comprehensiveness of environmental perception, but also significantly improves the safety and comfort of driving, and can effectively reduce traffic accidents and improve road safety.

[0081] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0082] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a head-mounted AR device.

[0084] refer to Figure 4 , the head-mounted AR device comprises:

[0085] A camera module is included, which is configured to capture an image of the driver's field of view;

[0086] A spatial information acquisition module configured to acquire spatial information within the driver's field of view;

[0087] A display screen is configured to display a target image to a driver, wherein the target image is an image for assisting the driver in identifying driving risks.

[0088] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a vehicle assisted driving system based on a head-mounted AR device.

[0089] refer to Figure 5 , the vehicle assisted driving system based on the head-mounted AR device includes:

[0090] Vehicle and the above-mentioned head-mounted AR device;

[0091] The vehicle comprises:

[0092] An environmental sensor configured to collect environmental information;

[0093] A communication module, configured to connect the vehicle with a head-mounted AR device for information interaction;

[0094] A processor is configured to, in response to vehicle startup, collect environmental information through the vehicle's environmental sensors and determine whether there is a driving risk based on the environmental information; if it is determined that there is a driving risk, send a target image to a head-mounted AR device for display through a communication module, wherein the target image is an image determined based on the environmental information, the field of view image, and the spatial information to assist the driver in identifying the driving risk.

[0095] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a vehicle assisted driving device based on a head-mounted AR device.

[0096] refer to Figure 6 , the vehicle assisted driving device based on the head-mounted AR device includes:

[0097] A judgment module 701 is configured to collect environmental information through an environmental sensor of the vehicle in response to the vehicle starting and determine whether there is a driving risk based on the environmental information;

[0098] The display control module 702 is configured to display a target image through a head-mounted AR device in response to the existence of driving risks, where the target image is an image for assisting the driver in identifying driving risks.

[0099] Furthermore, the display control module 702 is specifically used for:

[0100] The driver's field of vision image is collected through a head-mounted AR device, and an image in the environment image that is different from the driver's field of vision image is determined as a target image;

[0101] The target image is displayed in the driver's field of view through a head-mounted AR device so that the image observed by the driver is consistent with the environmental image.

[0102] Furthermore, the display control module 702 is further specifically configured to:

[0103] Performing edge feature detection on the point cloud data of the environmental object to obtain an edge feature image of the environmental object; collecting the driver's field of view image through the head-mounted AR device, and performing edge detection on the field of view image to obtain a first edge image of the object in the field of view image;

[0104] Determine an image in the edge feature image that is different from the first edge image as a target image;

[0105] The target image is displayed in the driver's field of view through a head-mounted AR device so that the edge information of the picture observed by the driver is consistent with the edge information of the edge feature image.

[0106] Furthermore, the judging module 701 is specifically used for:

[0107] In response to determining that the distance between at least one obstacle and the vehicle is less than or equal to a first preset safety distance, determining that there is a driving risk;

[0108] In response to determining that the distances between all obstacles and the vehicle are greater than a first preset safety distance and less than or equal to a second preset safety distance, it is determined whether there is a driving risk based on spatial position information of the obstacles relative to the vehicle.

[0109] Furthermore, the judging module 701 is further configured to:

[0110] In response to determining that at least one obstacle is on the predicted driving trajectory of the vehicle, determining that there is a driving risk;

[0111] Wherein, the predicted driving trajectory is determined based on the tire steering angle.

[0112] Furthermore, the judging module 701 is further configured to:

[0113] Determine the driver's real-time blind spots;

[0114] In response to determining that at least one obstacle is in a preset blind spot and / or a real-time blind spot of the driver, it is determined that there is a driving risk.

[0115] Furthermore, the judging module 701 is further configured to:

[0116] The spatial information under the driver's field of view is collected through a head-mounted AR device, and a difference space is determined according to the spatial information in the environmental spatial information that is different from the spatial information under the driver's field of view, and the difference space is determined as the driver's real-time field of view blind spot.

[0117] Furthermore, the judging module 701 is further configured to:

[0118] Perform edge feature detection on the point cloud data of the environmental object to obtain an edge feature image of the environmental object; perform edge detection on the environmental image to obtain a second edge image of the object in the environmental image;

[0119] In response to a difference between the second edge image and the edge feature image being greater than or equal to a preset difference, it is determined that there is a driving risk.

[0120] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0121] The device of the above embodiment is used to implement the corresponding vehicle assisted driving method based on a head-mounted AR device in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0122] Figure 7 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0123] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0124] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0125] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0126] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0127] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0128] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0129] The electronic device of the above embodiment is used to implement the corresponding vehicle assisted driving method based on head-mounted AR device in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0130] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the vehicle assisted driving method based on a head-mounted AR device as described in any of the above embodiments.

[0131] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0132] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the vehicle assisted driving method based on a head-mounted AR device as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0133] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0134] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0135] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0136] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0137] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0138] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0139] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0140] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0141] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A vehicle assisted driving method based on a head-mounted AR device, characterized in that: The vehicle includes an environmental sensor, and the method includes: In response to the vehicle being started, collecting environmental information through an environmental sensor of the vehicle and determining whether there is a driving risk based on the environmental information; In response to the existence of driving risks, a target image is displayed through a head-mounted AR device, where the target image is an image that assists the driver in identifying driving risks.

2. The vehicle assisted driving method based on a head-mounted AR device according to claim 1, characterized in that: The environmental information includes an environmental image; and displaying the target image through a head-mounted AR device includes: The driver's field of view image is collected by the head-mounted AR device, and an image in the environment image that is different from the driver's field of view image is determined as a target image; The target image is displayed in the driver's field of view through a head-mounted AR device so that the picture observed by the driver is consistent with the environmental image.

3. The vehicle assisted driving method based on a head-mounted AR device according to claim 1, characterized in that: The environmental information includes point cloud data of environmental objects; and displaying the target image through a head-mounted AR device includes: Performing edge feature detection on the point cloud data of the environmental object to obtain an edge feature image of the environmental object; collecting a field of view image of the driver through the head-mounted AR device, and performing edge detection on the field of view image to obtain a first edge image of the object in the field of view image; Determine an image in the edge feature image that is different from the first edge image as a target image; The target image is displayed in the driver's field of vision through the head-mounted AR device so that edge information of the picture observed by the driver is consistent with edge information of the edge feature image.

4. The vehicle assisted driving method based on a head-mounted AR device according to claim 1, characterized in that: The environmental information includes the distance between the obstacle and the vehicle, and determining whether there is a driving risk based on the environmental information includes: In response to determining that the distance between at least one obstacle and the vehicle is less than or equal to a first preset safety distance, determining that there is a driving risk; In response to determining that the distances between all obstacles and the vehicle are greater than a first preset safety distance and less than or equal to a second preset safety distance, it is determined whether there is a driving risk based on spatial position information of the obstacles relative to the vehicle.

5. The vehicle assisted driving method based on a head-mounted AR device according to claim 4, characterized in that: The determining whether there is a driving risk based on the spatial position information of the obstacle relative to the vehicle includes: In response to determining that at least one obstacle is on the predicted driving trajectory of the vehicle, determining that there is a driving risk; Wherein, the predicted driving trajectory is determined based on the tire steering angle.

6. The vehicle assisted driving method based on a head-mounted AR device according to claim 4, characterized in that: The determining whether there is a driving risk based on the spatial position information of the obstacle relative to the vehicle includes: Determine the driver's real-time blind spots; In response to determining that at least one obstacle is in a preset blind spot and / or a real-time blind spot of the driver, it is determined that there is a driving risk.

7. The vehicle assisted driving method based on a head-mounted AR device according to claim 6, characterized in that: The environmental information includes environmental space information, and the determining of the real-time blind spot of the driver includes: The spatial information under the driver's field of view is collected by the head-mounted AR device, and a difference space is determined according to the spatial information in the environmental spatial information that is different from the spatial information under the driver's field of view, and the difference space is determined as the driver's real-time field of view blind spot.

8. The vehicle assisted driving method based on a head-mounted AR device according to claim 1, characterized in that: The environmental information includes an environmental image and point cloud data of environmental objects; and determining whether there is a driving risk based on the environmental information includes: Perform edge feature detection on the point cloud data of the environmental object to obtain an edge feature image of the environmental object; perform edge detection on the environmental image to obtain a second edge image of the object in the environmental image; In response to a difference between the second edge image and the edge feature image being greater than or equal to a preset difference, it is determined that there is a driving risk.

9. A head-mounted AR device, characterized in that: include: A camera module configured to capture an image of the driver's field of view; A spatial information acquisition module configured to acquire spatial information within the driver's field of view; A display screen is configured to display a target image to a driver, wherein the target image is an image for assisting the driver in identifying driving risks.

10. A vehicle assisted driving system based on a head-mounted AR device, characterized in that: include: A vehicle and a head-mounted AR device as claimed in claim 9; The vehicle comprises: An environmental sensor configured to collect environmental information; A communication module, configured to connect the vehicle with a head-mounted AR device for information interaction; A processor is configured to, in response to vehicle startup, collect environmental information through the vehicle's environmental sensors and determine whether there is a driving risk based on the environmental information; if it is determined that there is a driving risk, send a target image to a head-mounted AR device for display through a communication module, wherein the target image is an image determined based on the environmental information, the field of view image, and the spatial information to assist the driver in identifying the driving risk.

Citation Information

Cited By

  • Information display method and device and vehicle

    CN120439800A

  • Information display method, device and vehicle

    CN120439800B