An assisted driving method, apparatus and related device

Abstract

The embodiment of the application discloses a kind of auxiliary driving method, device and related equipment, the method includes: obtaining first information, first information is based on the external environment information that vehicle's first perception sensor set is collected;Second information is obtained, second information is based on the internal environment information that second perception sensor set is collected in vehicle, and second information includes the data under the perspective of driver, the eye data of driver and the head posture data of driver;Third information is obtained, third information is based on vehicle state;Based on first information, second information and third information generate the driving image data of vehicle under the perspective of driver, and driving image data is used to assist driving.By first perception sensor set sufficient collection external environment information to solve the problem of visual interference, by collecting second information generates the driving image data of driver and driving cabin interaction under the perspective of driver, to better assist driving, improve the driving safety of vehicle.

Description

Technical Field

[0001] This application relates to the field of vehicle driver assistance technology, and in particular to a driver assistance method, device and related equipment. Background Technology

[0002] Over the past century, automobiles have evolved from early horse-drawn carriage-like driving modes to modern enclosed intelligent driving cockpits, resulting in countless improvements in comfort and safety. Safety, a perpetual theme in automotive technology development, encompasses a wide range of aspects, from occupant safety and vehicle safety to pedestrian safety. From early safety devices such as windshields, rearview mirrors, enclosed driver's cockpits, seat belts, and airbags, to the continuous emergence of advanced driver assistance technologies like radar, 330-degree camera systems, and head-up displays (HUDs), driving safety has steadily improved. However, some driving safety issues remain, such as blind spots caused by vehicle obstruction, decreased or even temporary blindness due to strong sunlight, reduced vision at night or in low light, obstructed visibility due to weather, and drivers losing sight of the road, all of which threaten the safety of occupants and pedestrians. Therefore, improving vehicle driving safety is of paramount importance. Summary of the Invention

[0003] This application addresses a series of visual interference problems in automobile driving by providing an assisted driving method, device, and related equipment. It solves the visual interference problem by displaying enhanced driving image data from the driver's perspective using a head-mounted display device. At the same time, it displays image data of the interaction between the driver and the cockpit from the driver's perspective, eliminating the sense of disconnect caused by the inconsistency between visual and tactile feedback. It does not change the driver's driving habits based on traditional driving methods, thereby better assisting the driver in driving the vehicle and improving vehicle driving safety.

[0004] In a first aspect, embodiments of this application provide an assisted driving method, including:

[0005] First information is obtained, which is based on the external environment information around the vehicle collected by the first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one position on the vehicle body.

[0006] The second information is obtained based on the internal environment information of the vehicle collected by the second set of perception sensors installed in the vehicle. The second information includes data from the driver's perspective, driver's eye data, and driver's head posture data.

[0007] Obtain third information, which is derived based on the vehicle state;

[0008] Based on the first information, the second information, and the third information, driving image data of the vehicle from the driver's perspective is generated, and the driving image data is used to assist driving.

[0009] Secondly, embodiments of this application provide a driver assistance device, including:

[0010] The first acquisition unit is used to acquire first information, which is obtained based on the external environment information around the vehicle collected by the first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one position on the vehicle body.

[0011] The second acquisition unit is used to acquire second information, which is obtained based on the internal environment information of the vehicle collected by the second set of perception sensors installed in the vehicle. The second information includes data from the driver's perspective, driver's eye data, and driver's head posture data.

[0012] The third acquisition unit is used to acquire third information, which is obtained based on the vehicle state.

[0013] The generation unit is used to generate driving image data of the vehicle from the driver's perspective based on the first information, the second information and the third information, and the driving image data is used to assist driving.

[0014] Thirdly, embodiments of this application provide a computing device, the computing device comprising: a processor and a memory, the processor being configured to execute the method described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a vehicle that includes the computing device described in the third aspect above.

[0016] Fifthly, embodiments of this application provide a computer-readable storage medium storing program instructions that, when executed, implement the method described in the first aspect above.

[0017] In a sixth aspect, embodiments of this application provide a computer program product, characterized in that the computer program product includes program instructions, which, when executed by a processor, implement the method described in the first aspect above.

[0018] This application embodiment can acquire first information, which is obtained based on the external environment information around the vehicle collected by a set of first perception sensors of the vehicle. The first perception sensor set includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged in at least one position on the vehicle body. It can also acquire second information, which is obtained based on the internal environment information of the vehicle collected by a set of second perception sensors installed inside the vehicle. The second information includes data from the driver's perspective, driver's eye data, and driver's head posture data. Finally, it can acquire third information, which is obtained based on the vehicle's state. Based on the first, second, and third information, it generates driving image data of the vehicle from the driver's perspective, which is used to assist driving. By fully collecting the first information using multiple vehicle external perception sensors of different types arranged in different positions on the vehicle body, and enhancing or filtering the first information, it helps to solve the problem of visual interference. By collecting perspective data, driver's eye data, and head posture data, it helps to generate driving image data of the driver interacting with the cockpit from the driver's perspective, thereby better assisting the driver in driving the vehicle based on the driving image data displayed on the head-mounted display device and improving vehicle driving safety. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of an assisted driving system provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the structure of a central computing processing unit provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a data flow provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the layout of a vehicle external perception sensor provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of another layout of vehicle external perception sensors provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the layout of a sensing sensor inside a driver's cockpit provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the layout of another driver's cockpit interior sensing sensor provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the layout of sensing sensors on a head-mounted display device provided in an embodiment of this application;

[0028] Figure 9 This is a schematic flowchart of an assisted driving method provided in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram illustrating the establishment of a coordinate system based on the driver's perspective, provided in an embodiment of this application.

[0030] Figure 11 This is a schematic diagram illustrating the process of switching between assisted driving mode and autonomous driving mode according to an embodiment of this application;

[0031] Figure 12 This is a flowchart illustrating another assisted driving method provided in an embodiment of this application;

[0032] Figure 13 This is a schematic diagram of driving image data of a vehicle from a driver's perspective, provided in an embodiment of this application.

[0033] Figure 14 This is a schematic diagram of driving image data of a vehicle from another driver's perspective, provided in an embodiment of this application.

[0034] Figure 15 This is a schematic diagram of driving image data of a vehicle from the driver's perspective, provided in another embodiment of this application.

[0035] Figure 16 This is a schematic diagram of driving image data of a vehicle from the driver's perspective, provided in another embodiment of this application.

[0036] Figure 17 This is a schematic diagram of the structure of an assisted driving device provided in an embodiment of this application;

[0037] Figure 18 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0040] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0041] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0042] In specific implementations, the computing devices described in the embodiments of this application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the device is not a portable communication device, but a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0043] This application provides an assisted driving method, apparatus, and related equipment. The assisted driving method can be applied to an assisted driving device, which can be housed in a central computing processing unit (CCPU). The CCPU can be integrated into an assisted driving system. The assisted driving system includes a vehicle and a central computing unit. The vehicle includes an external vehicle perception sensor subsystem, a driver's cabin interior perception sensor subsystem, and a head-mounted display device. The external vehicle perception sensor subsystem is a first set of perception sensors, comprising multiple external vehicle perception sensors. The driver's cabin interior perception sensor subsystem and the sensors on the head-mounted display device form a second set of perception sensors. The driver's cabin interior perception sensor subsystem includes multiple driver's cabin interior perception sensors, and the sensors on the head-mounted display device include at least one external head-mounted display sensor and at least one internal head-mounted display sensor. The central computing processing unit can be housed in a computing device, which may include, but is not limited to, a terminal device or server connected to the vehicle. The computing device can also be a device installed on the vehicle, independent of the vehicle, or a component of the vehicle. This application provides an assisted driving method applicable to vehicle driving scenarios.

[0044] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of an assisted driving system provided in an embodiment of this application. The assisted driving system includes an external vehicle perception sensor subsystem 11, an internal driver's cabin perception sensor subsystem 12, a head-mounted display device 13, a central computing processing unit 14, and a high-speed communication system 15.

[0045] The system includes, but is not limited to, multiple external vehicle perception sensors such as visual perception sensors, infrared perception sensors, radar perception sensors, and lidar perception sensors; the driver's cabin internal perception sensor subsystem 12 includes, but is not limited to, visual perception sensors and infrared perception sensors; the head-mounted display device 13 includes, but is not limited to, visual perception sensors, infrared perception sensors, display screens, lens modules, feature identification devices, and gyroscopes, and the head-mounted display device 13 may include, but is not limited to, mixed reality (MR) head-mounted display devices; the central computing processing unit 14 includes, but is not limited to, a main processor, an image processor, an artificial intelligence processor, and a memory; and the high-speed communication system 15 is used to assist data communication between various modules of the entire system, and can adopt a low-latency high-speed bus method to ensure real-time performance.

[0046] Among them, the hardware modules of the central computing processing unit are as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a central computing processing unit provided in an embodiment of this application, such as... Figure 2 As shown, the central computing processing unit consists of a main processor 201, an image processor 202, an artificial intelligence processor 203, a memory 204, and a communication interface 205. The main processor 201 is responsible for the main process control of the overall auxiliary tasks, including mode control, data input / output, and process processing. The image processor 202 has dedicated image processing acceleration capabilities and is responsible for dedicated parallel acceleration processing of images. The artificial intelligence processor 203 is a dedicated processor with artificial intelligence acceleration functions (such as convolutional neural networks and recurrent neural networks), responsible for accelerating intelligent recognition, intelligent fusion, intelligent driving assistance, and autonomous driving modules. The memory 204 assists in storing data and managing software resources during system operation. The communication interface 205 is responsible for connecting to the high-speed communication system, receiving data from various sensor modules, data sent to the head-mounted display device, and control commands sent to external execution units.

[0047] The software portion of the central computing unit consists of a data preprocessing module, a fusion perception computing module, an intelligent (e.g., neural network) recognition module, an intelligent assisted driving module, a layer rendering management module, and (optionally) an autonomous driving module. The data preprocessing module of the central computing unit processes the raw data collected by each perception module. The data preprocessing module removes overlapping areas from the perception sensors' fields of view and removes noise from the acquired signals. The fusion perception computing module fuses data from multiple types of perception sensors to establish unified, high-confidence external environmental spatial data. The intelligent recognition module identifies vehicles, roads, pedestrians, buildings, signs, obstacles, etc., in the external environment for subsequent interference filtering or assisted driving prompts. The intelligent assisted driving module provides users with assisted driving information and driving decision suggestions. The layer rendering management module transforms and calculates images from different layers based on the driver's changing viewpoint, then fuses and renders them, ultimately outputting the result to the driver's head-mounted display screen. Layers can include the vehicle external environment layer, vehicle body reference contour layer, cockpit interior hand interaction layer, vehicle status data layer, and assisted driving information interaction layer, etc. Different layer data are processed by different layer managers, and finally rendered by the layer manager module. The autonomous driving module is optional and provides autonomous driving services to the driver.

[0048] The data flow of the driver assistance system provided in this embodiment of the invention during operation is as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of a data flow provided in an embodiment of this application, such as... Figure 3 As shown, the vehicle external perception sensor 31, the driver's cabin internal perception sensor 32, and the head-mounted display device perception sensor 33 (including the head-mounted display device external perception sensor and the head-mounted display device internal perception sensor) assist in collecting data and send it to the central computing processing unit 34. After receiving the vehicle status information 35 data from the vehicle, the central computing processing unit 34 processes the data and outputs image data or voice prompt data to the head-mounted display device 36.

[0049] In order to maintain the driver's driving habits and ensure that the driver's field of vision and angle remain consistent whether driving with or without a head-mounted display, thereby improving driving safety and comfort without altering the driver's long-established driving habits, this application proposes an assisted driving method. This method utilizes information collected by perception sensors installed both outside and inside the vehicle. After processing by a central computing unit and incorporating vehicle status information, the final rendered image information (i.e., driving image data) is sent to the display module of the head-mounted display to assist the driver in making driving decisions based on the images seen through the head-mounted display.

[0050] The central computing unit can acquire first information, which is based on the external environment information around the vehicle collected by the vehicle's first set of perception sensors; acquire second information, which is based on the internal environment information of the vehicle collected by the second set of perception sensors installed inside the vehicle, including data from the driver's perspective, driver's eye data, and driver's head posture data; and acquire third information, which is based on the vehicle's state. Based on the first, second, and third information, the central computing unit generates driving image data of the vehicle from the driver's perspective. The driving image data is used to assist driving. The driving image data is sent to a head-mounted display device so that the head-mounted display device can display the driving image data to assist the driver in driving the vehicle according to the driving image data displayed by the head-mounted display device.

[0051] The first set of sensing sensors may include, but is not limited to, multiple external vehicle sensing sensors, which together form a vehicle external sensing sensor subsystem. Each external vehicle sensing sensor is located on the exterior of the vehicle and consists of different types of sensors installed at different locations on the vehicle body, including, but not limited to, visual sensing sensors, infrared sensing sensors, radar sensing sensors, and lidar sensing sensors. The radar sensing sensors include common ultrasonic radar sensors and millimeter-wave radar sensors. Ultrasonic radar sensors are mainly used for detecting objects in the near range around the vehicle and providing warning information, while millimeter-wave radar sensors are mainly used for long-range detection and spatial point cloud data acquisition in the vehicle's driving environment. The locations of the various external vehicle sensing sensors include, but are not limited to, the front, middle, rear, sides, and roof of the vehicle body, to ensure that each individual sensor can acquire 360-degree environmental data around the vehicle. The visual and infrared sensing sensors collect image data, while the lidar and radar sensing sensors collect point cloud data.

[0052] Specifically, it can be combined with Figure 4 and Figure 5 The arrangement of the various external sensing sensors on the vehicle is illustrated schematically. Figure 4 This is a schematic diagram of the layout of a vehicle external perception sensor provided in an embodiment of this application. Figure 5 This is a schematic diagram of another layout of vehicle external perception sensors provided in an embodiment of this application.

[0053] like Figure 4As shown, perception sensors 401-403 are front-facing sensors, primarily responsible for collecting data on the front, left front, and right front of the vehicle; perception sensors 404 and 405 are left and right rear-view sensors, primarily responsible for simulating the rearview mirror angle in traditional driving methods and collecting left and right rear-view angle data; perception sensors 406 and 407 are responsible for collecting data on the left front and right front of the vehicle, especially data on blind spots obstructed by the A-pillar; perception sensor 408 is a roof-mounted perception sensor, primarily responsible for collecting data on the front of the vehicle with a relatively wide angle; perception sensors 409 and 410 are responsible for collecting data on the left rear and right rear of the vehicle, especially data on blind spots obstructed by the C-pillar; perception sensors 411-413 are rear-facing sensors, primarily responsible for collecting data on the directly rear, left rear, and right rear of the vehicle. Here, the A-pillar refers to the pillars on either side of the vehicle's windshield, and the C-pillar refers to the vehicle's roof and body. In other embodiments, the vehicle also includes a B-pillar, which refers to the pillar between the front and rear doors.

[0054] like Figure 5 As shown, the side view 501, top view 502, front view 503, and rear view 504 of the vehicle, which are marked with various external sensing sensors, further illustrate the layout of the vehicle's external sensing sensors.

[0055] In some embodiments, Figure 4 , Figure 5 The vehicle's external sensing sensors at each location are not limited to a single type of sensing sensor; they can be a combination of one or more sensing sensors.

[0056] In some embodiments, the second set of sensing sensors may include, but is not limited to, one or more of the following: external sensing sensors for the head-mounted display, internal sensing sensors for the head-mounted display, and internal sensing sensors for the cockpit.

[0057] The cockpit interior sensing sensor is a subsystem that includes, but is not limited to, multiple sensing sensors such as visual sensing sensors and infrared sensing sensors. The cockpit interior sensing sensor is located at the front and top of the cockpit to ensure that it can cover the entire cockpit space, so as to accurately and stably collect data inside the cockpit under various lighting conditions, especially the image data of the head-mounted display device worn by the driver.

[0058] Specifically, it can be combined with Figure 6 and Figure 7 The layout of the sensing sensors inside the cockpit is illustrated schematically. Figure 6 This is a schematic diagram of the layout of a sensing sensor inside the cockpit provided in an embodiment of this application, as shown below. Figure 6 As shown, 601 represents the sensing sensor and its sensing field of view on the front dashboard of the driver's cockpit; 602 represents the sensing sensor and its sensing field of view on the front top of the driver's cockpit; and 603 represents the sensing sensor and its sensing field of view on the rear top of the driver's cockpit. The sensing sensors within the driver's cockpit cover the entire cockpit, ensuring that the driver, while wearing the 604 head-mounted display, can be detected by the internal sensing sensors regardless of their head position or posture within the cockpit. Figure 7 This is a schematic diagram of another layout of the in-cabin sensing sensor provided in an embodiment of this application, as shown below. Figure 7 The diagram shows another perspective of the sensor arrangement inside the driver's cockpit. 701 is a sensor located on the upper left front of the driver's cockpit, 702 is a sensor located on the lower left front of the driver's cockpit, 703 is a sensor located on the upper front of the driver's cockpit, 704 is a sensor located on the upper right front of the driver's cockpit, and 705 is a sensor located on the lower right front of the driver's cockpit.

[0059] The head-mounted display device (HMD) consists of external and internal sensing sensors. The HMD comprises a visual sensing sensor, an infrared sensing sensor, a display screen, a lens assembly, a feature identification device, a gyroscope, a head-mounted fastening device, and a cooling and ventilation module. The visual and infrared sensing sensors are further categorized as external and internal sensing sensors based on their installation location and function.

[0060] Specifically Figure 8 The following is an illustrative explanation of a head-mounted display device. Figure 8 This is a schematic diagram of the layout of sensing sensors on a head-mounted display device provided in an embodiment of this application, as shown below. Figure 8As shown, 801 is the head-mounted display device, and 802 is a schematic diagram of the external sensing sensor of the head-mounted display device and its sensing angle. This sensor is responsible for acquiring images of the interior of the cockpit, simulating the driver's "first-person" (i.e., driver's perspective) images of interactive elements such as the steering wheel, hands, and dashboard. This prevents the driver from missing images of their hands interacting with the cockpit, thus avoiding a disconnect between visual and tactile perception. 803 is a schematic diagram of the internal sensing sensor of the head-mounted display device and its sensing range. The internal sensing sensor 803 is mainly responsible for locating the driver's eye position and orientation, as well as detecting driver fatigue. 804 is a feature marking device (i.e., a preset reference point) on the head-mounted display device. The feature marking device 804 is a combination of multiple circular feature points of known geometric dimensions. These circular feature points can actively emit visible or invisible light, and through different sizes or intervals, they form special image features. After calibration, the position and attitude information of the head-mounted display device can be located by acquiring images of these feature marking devices.

[0061] The identification device can be placed in different locations depending on the form of the head-mounted display device: if the head-mounted display device is like... Figure 8 The device shown has a helmet-like structure, and the identification device can be placed on the outside of the helmet; if the head-mounted display is in the form of VR glasses, the identification device can be placed on the outside of the VR glasses. In addition, the head-mounted display also includes a display screen and a lens module to provide the driver with "realistic" image information with a sense of distance and space. The display screen can be a high-definition, high-resolution, high-refresh-rate screen, and can be a flat or curved screen. The lens module includes lens groups and vision-correcting lenses. The display screen and lens module are consistent with the screens and lens groups in common VR devices, so they will not be discussed in detail.

[0062] The gyroscope provides additional attitude and acceleration information to improve the positioning accuracy of the head-mounted display and the driver's eye position. The head-mounted display's fixation device is used to comfortably and stably secure the display to the driver's head. It can be made of elastic fabric or a helmet-like design. Once secured, it creates a closed display space around the driver's eyes, preventing external light from entering and increasing immersion and realism while reducing distractions.

[0063] The cooling and ventilation module is responsible for cooling down the display screen and maintaining a moderate temperature and humidity in the enclosed display space around the driver's eyes, ensuring the driver's comfort.

[0064] Please see Figure 9 , Figure 9This is a flowchart illustrating an assisted driving method provided in an embodiment of this application, as shown below. Figure 9 The assisted driving method shown in this embodiment can be applied to an assisted driving device, which is located in a central computing processing unit, which is located in a computing device. The method may include at least the following steps.

[0065] S901: Obtain first information, which is obtained based on the external environment information around the vehicle collected by the vehicle's first set of perception sensors.

[0066] In this embodiment, the central computing processing unit can acquire first information, which is obtained based on the external environment information around the vehicle collected by a first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one location on the vehicle body. In some embodiments, each vehicle external perception sensor is a different type of sensor arranged at different locations on the vehicle body, as described above, and will not be repeated here.

[0067] Multiple different types of vehicle external perception sensors can fully collect 360-degree external environmental data around the vehicle body. This data can then be processed by the central computing unit to provide the driver with driving image data that does not have blind spots, including forward, side, and rear views, thus solving the problem of visual interference.

[0068] S902: Obtain second information, which is obtained based on the vehicle's internal environment information collected by a set of second sensing sensors installed in the vehicle.

[0069] In this embodiment of the application, the central computing processing unit can obtain second information. The second information is obtained based on the internal environment information of the vehicle collected by the second set of perception sensors installed in the vehicle. The second information includes data from the driver's perspective, driver's eye data, and driver's head posture data.

[0070] The second set of sensing sensors includes external sensing sensors for the head-mounted display device, internal sensing sensors for the head-mounted display device, and internal sensing sensors for the driver's cockpit. The second information includes data collected from the driver's perspective by the external sensing sensors for the head-mounted display device; eye data of the driver collected by the internal sensing sensors for the head-mounted display device; and head posture data of the driver collected by the internal sensing sensors for the driver's cockpit.

[0071] S903: Obtain third information, which is based on the vehicle status.

[0072] S904: Generates driving image data of the vehicle from the driver's perspective based on the first information, the second information and the third information. The driving image data is used to assist driving.

[0073] In this embodiment, the central computing processing unit can generate driving image data of the vehicle from the driver's perspective based on the first information, the second information, and the third information. The driving image data is used to assist driving. The third information includes, but is not limited to, information such as vehicle speed, remaining driving range, in-vehicle temperature and humidity, and tire pressure.

[0074] In one embodiment, the second information may include the viewpoint data, eye data, and head posture data; the central computing processing unit may determine the type of external environment in which the vehicle is currently located based on the first information, select multiple corresponding external environment data from the first information according to the external environment type, and perform fusion processing on the multiple external environment data to obtain target external environment data; extract interactive area image data from the viewpoint data, and perform segmentation processing on the interactive area image data to obtain segmented image data; calculate the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensor based on the eye data and head posture data, as well as the geometric positional relationship between the reference coordinate system of the driver's cockpit internal perception sensor and the reference coordinate system of the vehicle's external perception sensor; and determine the driving image data of the vehicle from the driver's perspective based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensor, and the third information.

[0075] In one embodiment, the central computing unit can determine the current external environment type of the vehicle based on the first information combined with information from weather forecasts, temperature sensors, humidity sensors, and barometric pressure sensors. This external environment type may include, but is not limited to, nighttime, daytime (sunny, cloudy), rainy, snowy, foggy, and dusty conditions. The central computing unit can then filter the first information collected by each of the vehicle's external sensing sensors based on the determined external environment type to select multiple external environment data points corresponding to that type. For example, if the external environment type is determined to be sunny, the infrared radiation from sunlight may interfere with the infrared sensor, leading to inaccurate data; therefore, it can be determined not to collect data from the infrared sensor. Similarly, if the external environment type is determined to be rainy, snowy, or foggy, the laser feedback from the lidar may interfere with the lidar sensor, resulting in inaccurate data; therefore, it can be determined not to collect data from the lidar sensor.

[0076] Different sensors have different characteristics. LiDAR (Light Detection and Ranging) has the advantages of high ranging accuracy, high point cloud density, and is unaffected by external light intensity. However, its disadvantages include a small ranging range, especially due to its relatively low resolution. When sensing at long distances, the points projected onto the target are very sparse, further shortening the effective range of the usable point cloud. Furthermore, LiDAR is significantly affected by rain, snow, water accumulation on roadsides, and glass walls. Visual sensors have the advantages of high frame rate, high resolution, color perception capabilities, and a long sensing range. Their disadvantages include insufficient depth perception, poor ranging accuracy, poor perception performance in harsh environments, and significant impact from strong light, low-light conditions at night, and rain, snow, or fog. Radar sensors are unaffected by the environment and have strong penetrating power through smoke, dust, rain, and snow. Their disadvantages include sparse point clouds and significant interference from signal noise. Infrared sensors have the advantages of strong perception capabilities in various scenarios such as nighttime, low-light environments, rain, fog, and sandstorms. Their disadvantage is significant impact from sunlight during the day.

[0077] By combining the characteristics of different sensors, different sensor types can be selected for different scenarios, and specific sensor signals can be prioritized while other sensor signals serve as supplementary signals. For example, during normal daytime conditions, a combination of visual sensors and LiDAR can be used as the primary sensor; at night or in low-light conditions, LiDAR and infrared sensors can be used as the primary sensor; and in rain, snow, fog, or dust storms, radar and infrared sensors can be used as the primary sensor. This method of selecting multiple sensors to collect vehicle external environmental data based on environmental type allows for the acquisition of sufficient external environmental data in various environments, including daytime, nighttime, rain, snow, fog, and dust storms. This enhances the vehicle's information acquisition capabilities and helps reduce driving safety accidents caused by excessive or insufficient light, glare, and severe weather. By selecting the most suitable sensor combination for different scenarios, external environmental data can be acquired to the maximum extent, filtering strong light and glare interference, enhancing low-light acquisition, and penetrating the visual interference of smoke, rain, snow, dust, and fog. This improves the driver's understanding of road information, increases the accuracy of driving decisions, reduces visual burden, and enhances driving safety and comfort.

[0078] Overall, by deploying various types of external vehicle perception sensors at different locations on the vehicle body, the collection range covers 360 degrees around the vehicle body, which can fully collect external environmental data of the vehicle. This makes the external environmental data obtained by the vehicle more accurate and stable in various environments, which helps to provide sufficient external environmental data in various weather conditions and helps to eliminate blind spots in traditional driving modes.

[0079] In one embodiment, when the central computing processing unit fuses multiple external environmental data to obtain target external environmental data, it can utilize the fusion perception computing module in the central computing processing unit to fuse multiple external environmental data using preset perception fusion algorithms, deep learning algorithms, etc., in order to obtain more accurate target external environmental data.

[0080] In one embodiment, when the central computing processing unit extracts interactive area image data from the viewpoint data and segments the interactive area image data to obtain segmented image data, it can acquire the driver's cockpit interior image data and the driver's cockpit exterior environment image data from the interactive area image data; it performs segmentation processing on the driver's cockpit interior image data and the driver's cockpit exterior environment image data, and determines that the driver's cockpit interior image data is the segmented image data. The driver's cockpit interior image data may include, but is not limited to, the driver's arm, steering wheel, instrument panel, etc., as seen from the driver's perspective, and the driver's cockpit exterior environment image data may include, but is not limited to, the exterior environment image data seen through the vehicle's windshield and windows from the driver's perspective.

[0081] In one embodiment, when the central computing processing unit calculates the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external sensing sensors based on eye data, head posture data, and the geometric positional relationship between the reference coordinate system of the driver's cockpit interior sensing sensors and the reference coordinate system of the vehicle's external sensing sensors, it can calculate the driver's eye position and orientation in the reference coordinate system of the head-mounted display device's interior sensing sensors based on eye data and a preset eye-tracking algorithm; identify the position of a preset reference point on the head-mounted display device from the head posture data; calculate the position and orientation of the head-mounted display device in the reference coordinate system of the driver's cockpit interior sensing sensors based on the head posture data; and calculate the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external sensing sensors based on the driver's eye position and orientation in the reference coordinate system of the head-mounted display device, the position of the preset reference point on the head-mounted display device, the position and orientation of the head-mounted display device in the reference coordinate system of the driver's cockpit interior sensing sensors, and the geometric positional relationship between the reference coordinate system of the driver's cockpit interior sensing sensors and the reference coordinate system of the vehicle's external sensing sensors. The preset reference point is a feature marker point on the head-mounted display device.

[0082] The methods for establishing various coordinate systems, such as the reference coordinate system for the internal sensing sensors of the head-mounted display device, the reference coordinate system for the internal sensing sensors of the driver's cockpit, and the reference coordinate system for the external sensing sensors of the vehicle, are as follows: Figure 10 As shown, Figure 10This is a schematic diagram of a coordinate system established based on the driver's perspective, provided in an embodiment of this application. In this diagram, 1001 is the reference coordinate system of the vehicle's external perception sensor, 1002 is the reference coordinate system of the driver's cabin internal perception sensor, 1003 is the reference coordinate system of the head-mounted display device's feature identifier, 1004 is the reference coordinate system of the head-mounted display device's internal perception sensor, 1005 is the head-mounted display device's internal perception sensor for driver's eye tracking, and 1006 is a schematic diagram of the driver's eye position and orientation.

[0083] Among them, the reference coordinate system 1001 for the vehicle external perception sensor is a unified reference coordinate system O1X1Y1Z1 established by the vehicle external perception sensor subsystem. Through calibration and calculation, the external environment data obtained by different sensors in the vehicle external perception sensor subsystem can be fused and unified under a single reference coordinate system. The reference coordinate system 1002 for the driver's cabin internal perception sensor is a unified reference coordinate system O2X2Y2Z2 established by the driver's cabin internal perception sensor subsystem. Through calibration and calculation, the data obtained by different sensors in the driver's cabin internal perception sensor subsystem can be unified under a single reference coordinate system. The reference coordinate system 1003 for the head-mounted display device feature markers is a feature coordinate system O3X3Y3Z3 established by the feature marker points on the head-mounted display device. This reference coordinate system is established based on the feature marker points. The reference coordinate system 1004 for the head-mounted display device internal perception sensor is a reference coordinate system O4X4Y4Z4 established by the head-mounted display device internal perception sensor. This reference coordinate system is used to observe the position and orientation of the driver's eyeballs.

[0084] Beforehand, the transformation relationship Trans1 between the 1001 coordinate system O1X1Y1Z1 and the 1002 coordinate system O2X2Y2Z2 is pre-calibrated, and the transformation relationship Trans2 between the 1003 coordinate system O3X3Y3Z3 and the 1004 coordinate system O4X4Y4Z4 is pre-calibrated. During actual operation, the sensor inside the cockpit acquires images of feature markers on the head-mounted display and identifies the positions of these markers in the 1002 coordinate system O2X2Y2Z2, thus obtaining the transformation relationship Trans_r1 between the 1002 coordinate system O2X2Y2Z2 and the 1003 coordinate system O3X3Y3Z3 at each acquisition time. The sensor inside the head-mounted display uses an eye-tracking algorithm to determine in real time the position P4 and orientation V4 of the driver's eyeball in the 1004 coordinate system O4X4Y4Z4. Then, through continuous coordinate transformation, the position P1 and orientation V1 of the driver's eyeball in the coordinate system O1X1Y1Z1 of the vehicle's external perception sensor can be obtained. The calculation formulas are shown in formulas (1) and (2) below:

[0085] [P1;1]=Trans1*Trans_r1*Trans2*[P4;1] (1)

[0086] [V1;0]=Trans1*Trans_r1*Trans2*[V4;0] (2)

[0087] By employing visual positioning methods, human eye tracking methods, and establishing multi-level reference coordinate systems, the position and orientation of the driver's eyeballs within the reference coordinate system of the vehicle's external perception sensors are determined.

[0088] In one embodiment, when the central computing processing unit determines the vehicle's driving image data from the driver's perspective based on target external environment data, segmented image data, the driver's eye position and orientation in the reference coordinate system of the vehicle's external perception sensors, and third-party information, it can determine an external environment image data layer and a cockpit interior interactive environment image data layer from the driver's perspective based on the target external environment data, segmented image data, and the driver's eye position and orientation in the reference coordinate system of the vehicle's external perception sensors; determine a vehicle driving data layer based on vehicle status data; and perform fusion processing on the external environment image data layer, the cockpit interior interactive environment image data layer, and the driving data layer to obtain the vehicle's driving image data from the driver's perspective. Each layer is composed of many pixels, and multiple layers are combined and superimposed to form the entire image.

[0089] The external environment image data layer may include, but is not limited to, the vehicle external environment layer and the vehicle body reference contour layer; the driver's cockpit interior interactive environment image data layer may include, but is not limited to, the cockpit interior hand interaction layer and the driver assistance information interaction layer; and the driving data layer may include, but is not limited to, the vehicle status data layer. Different layer data are processed by different layer managers, and finally, the layer manager module merges and renders them into driving image data.

[0090] Optionally, the central computing processing unit can render image data of different positions of the vehicle as driving image data according to different scenarios.

[0091] Optionally, the central computing processing device may not render the driver's cockpit in the preset focused driving mode, but only render the image information related to the steering wheel, hands and tactile interaction as driving image data, so as to reduce visual information input interference.

[0092] Optionally, the central computing unit can filter glare and stray light through different sensing sensors, and can also use artificial intelligence to identify and filter out information that is not useful for driving decisions, thereby further reducing information input interference to the driver, improving the driver's concentration, and reducing driving fatigue.

[0093] Optionally, the central computing and processing unit can convert the driving image data into three-dimensional driving image data and send it to the head-mounted display device for display. In this way, the driver sees virtual image information that is consistent with the actual driving environment, further reducing interference from useless information. Different simulation styles can also be rendered to bring a completely new driving experience. If the acquired image signals are stitched together before being sent to the head-mounted display device, inconsistent image quality will occur. For example, in existing 360-degree car imaging systems, the 360-degree image is captured and stitched from cameras in different directions of the vehicle. Because the light intensity varies in different directions, the image quality of the captured images will also be inconsistent. The stitched image will have obvious boundaries and a sense of fragmentation, causing users to be distracted by additional differences when viewing the image information, requiring them to expend effort to distinguish them. However, using a simulation mode to display external environment information through a head-mounted display device avoids these problems, ensuring that the image seen by the driver is uniform, consistent, smooth, and interference-free, improving the visual experience and driving safety.

[0094] Optionally, the central computing processing unit can further render the driving image data, changing the environmental information outside the road and switching to different environmental landscapes while ensuring the authenticity of the actual driving environment spatial information. The environmental landscapes include, but are not limited to, the seaside, snow-capped mountains, cities, sunsets, etc., providing a richer driving visual experience without affecting driving safety, increasing driving pleasure, and keeping the driver in a good mood, thereby further improving driving safety.

[0095] The above method establishes a geometric relationship between the driver's eye position and orientation and the external environmental spatial data. This, in turn, helps to obtain images (i.e., driving image data) from the driver's perspective at any position in the cockpit through real-time image transformation calculations.

[0096] In one embodiment, the central computing processing unit can send driving image data to a head-mounted display device, so that the head-mounted display device can display the driving image data to assist the driver in driving the vehicle based on the driving image data displayed by the head-mounted display device.

[0097] This application embodiment uses a central computing processing unit to determine the current external environment type of the vehicle based on first information collected by various external perception sensors on the vehicle. It then selects multiple corresponding external environment data from the first information based on the external environment type, fuses these multiple external environment data, and obtains target external environment data. Interactive area image data is extracted from the viewpoint data collected by various perception sensors on the head-mounted display device, and the interactive area image data is segmented to obtain the driver's cockpit interior image data. Based on eye data, head posture data, and the geometric position between the reference coordinate system of the driver's cockpit interior perception sensors and the reference coordinate system of the vehicle's external perception sensors... The system calculates the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external perception sensors. Based on target external environment data, cockpit interior image data, the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external perception sensors, and third-party information, it determines the vehicle's driving image data from the driver's perspective and sends this data to the head-mounted display device for display. This ensures that the driver's field of vision and angle remain consistent whether driving with or without the head-mounted display device is used, improving driving safety and comfort without altering the driver's long-established driving habits.

[0098] Please see Figure 11 As shown, Figure 11 This is a schematic diagram illustrating the process of switching between assisted driving mode and autonomous driving mode, as provided in an embodiment of this application.

[0099] S1101: Obtain first information, which is obtained based on the external environment information around the vehicle collected by the vehicle's first set of perception sensors.

[0100] S1102: When the start command of the assisted driving mode is received, determine whether the current driving mode of the vehicle meets the conditions for assisted driving based on the first information. If it is determined that the current driving mode of the vehicle meets the conditions for assisted driving, then execute step S1103. If it is determined that the current driving mode of the vehicle does not meet the conditions for assisted driving, then execute step S1104.

[0101] S1103: Activate the vehicle's vision-assisted driving system and proceed to step S1105.

[0102] S1104: Activate the vehicle's autonomous driving system and trigger the execution of preset autonomous driving decisions.

[0103] In this embodiment, the central computing processing unit can send a control command to the vehicle to enter the autonomous driving mode when it receives the start command for the autonomous driving mode, so as to start the vehicle's autonomous driving system, so that the vehicle enters the autonomous driving mode and triggers the execution of preset autonomous driving decisions.

[0104] Optionally, when the vehicle enters the autonomous driving mode and executes a preset autonomous driving decision, the central computing unit can send the driving image data collected by the vehicle in the autonomous driving mode from the driver's perspective to the head-mounted display device, so that the head-mounted display device can display the driving image data from the driver's perspective in the autonomous driving mode, so that the driver can observe the driving situation of the vehicle at any time and manually brake the vehicle when observing special situations, which helps to further improve the driving safety of the vehicle.

[0105] Optionally, the autonomous driving system can collect the driver's driving decisions in different scenarios and learn the driver's driving habits when the driver assistance system is working.

[0106] S1105: Obtain second information, which is based on the vehicle's internal environment information collected by a set of second sensing sensors installed in the vehicle.

[0107] In this embodiment, the central computing processing unit can determine whether the vehicle's current driving mode meets the conditions for assisted driving based on the first information. If it is determined that the vehicle's current driving mode meets the conditions for assisted driving, the vehicle's visual assisted driving system is activated, and the acquisition of the vehicle's second information collected by the vehicle's second perception sensor set is triggered. The description of the second perception sensor set has been described above and will not be repeated here.

[0108] S1106: Obtain third information, which is based on the vehicle status.

[0109] S1107: Based on the first information, the second information, and the third information, determine the driving image data of the vehicle from the driver's perspective.

[0110] S1108: Send driving image data to a head-mounted display device so that the head-mounted display device can display the driving image data to assist the driver in driving the vehicle based on the driving image data displayed by the head-mounted display device.

[0111] Optionally, in assisted driving mode, the driver can be prompted to make the correct driving decisions through visual images, voice, etc. At the same time, the autonomous driving system can take over vehicle control in dangerous situations and slow down to stop in a safe area.

[0112] This application embodiment switches between assisted driving mode and autonomous driving mode. If the current mode is assisted driving mode, the data collected by the vehicle's external perception sensors is processed and displayed to the driver, who then makes driving decisions. If the mode is switched to autonomous driving mode, the data collected by the vehicle's external perception sensors is processed and sent to the autonomous driving module, which then makes driving decisions. Optionally, the driving image data can be displayed to the driver via the vehicle's central control screen or a head-mounted display.

[0113] Please see Figure 12 , Figure 12 This is a flowchart illustrating another assisted driving method provided in an embodiment of this application, such as... Figure 12 The assisted driving method shown in this embodiment can be applied to an assisted driving device, which is located in a central computing processing unit, which is located in a computing device. The method may include at least the following steps.

[0114] S1201: Obtain first information, which is obtained based on the external environment information around the vehicle collected by the vehicle's first set of perception sensors.

[0115] S1202: Determine the type of external environment in which the vehicle is currently located based on the first information, select multiple corresponding external environment data from the first information based on the external environment type, and perform fusion processing on the multiple external environment data to obtain target external environment data.

[0116] S1203: Acquire data from the driver's perspective collected by the external sensing sensors of the head-mounted display device.

[0117] S1204: Extract interactive region image data from the viewpoint data, and segment the interactive region image data to obtain segmented image data.

[0118] S1205: Acquire driver head posture data collected by the sensor inside the cockpit.

[0119] In this embodiment of the application, the central computing processing unit can acquire the driver's head posture data collected by the sensing sensors inside the cockpit, wherein the head posture data may include, but is not limited to, the head rotation direction and rotation angle.

[0120] S1206: Identify the position of the preset reference point on the head-mounted display device from the head posture data, and calculate the position and attitude of the head-mounted display device in the reference coordinate system of the sensing sensors inside the cockpit based on the head posture data.

[0121] S1207: Acquire driver eye data collected by the sensing sensors inside the head-mounted display device.

[0122] In this embodiment of the application, the central computing processing unit can acquire the driver's eye data collected by the sensing sensors inside the head-mounted display device, wherein the eye data includes, but is not limited to, the rotation angle of the eyeball and the rotation direction of the eyeball.

[0123] S1208: Based on eye data and a preset human eye tracking algorithm, calculate the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display device.

[0124] S1209: Based on the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display, the position of the preset reference point of the head-mounted display, the position and attitude of the head-mounted display in the reference coordinate system of the sensor inside the driver's cabin, and the geometric positional relationship between the reference coordinate system of the sensor inside the driver's cabin and the reference coordinate system of the sensor outside the vehicle, calculate the position and orientation of the driver's eyeball in the reference coordinate system of the sensor outside the vehicle.

[0125] S1210: Obtain third information, which is based on the vehicle status.

[0126] S1211: Based on the target external environment data, segmented image data, and the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the external environment image data layer and the driver's cockpit internal interactive environment image data layer from the driver's perspective; determine the vehicle's driving data layer based on the vehicle status data; and perform fusion processing on the external environment image data layer, the driver's cockpit internal interactive environment image data layer, and the driving data layer to obtain the vehicle's driving image data from the driver's perspective.

[0127] S1212: Send driving image data to a head-mounted display device so that the head-mounted display device can display the driving image data to assist the driver in driving the vehicle based on the driving image data displayed by the head-mounted display device.

[0128] In some embodiments, the vehicle's outline in the driving image data can be displayed on the head-mounted display device with a preset transparency or a preset line type. The preset line type can include, but is not limited to, solid lines and dashed lines of varying thicknesses, and the preset transparency can be any pre-defined transparency. Displaying the vehicle's driving image data on the head-mounted display device with preset transparency or a preset line type can provide the driver with a vehicle position reference while minimizing differences in driving habits and, to some extent, eliminating blind spots.

[0129] like Figure 13 As shown, Figure 13This is a schematic diagram of driving image data of a vehicle from a driver's perspective, provided in an embodiment of this application. 1301, 1302, and 1303 represent the front rearview view, left rearview view, and right rearview view, respectively. Figure 13 This is an example of a rearview view layout; 1304 shows a pedestrian seen through "perspective" (e.g., 100% transparency to indicate full transparency) in the original left A-pillar blind spot; 1305 shows a road sign seen through "perspective" in the original right A-pillar blind spot; 1309 shows road potholes seen through "perspective" in the original hood blind spot; 1306 shows the outline of the "transparent" effect of the hood, used to provide the driver with a reference for the vehicle's position; 1307 is a diagram of the steering wheel; 1310 is a diagram of the driver's hands; 1308 is a diagram of external vehicle images perceived by the vehicle's external sensing sensors in rain, snow, or foggy weather. Figure 13 The diagram illustrates how the driver assistance system of this application enhances the driver's road perception capabilities without altering the driver's driving habits or driving vision habits.

[0130] For example Figure 14 As shown, Figure 14 This is a schematic diagram of driving image data of a vehicle from another driver's perspective provided in an embodiment of this application, wherein 1401, 1402, and 1403 are the front rear view, left rear view, and right rear view, respectively. Figure 13 The difference is Figure 14 The arrangement of 1401, 1402, and 1403 displayed on the head-mounted display device is similar to... Figure 13 The positions of 1301, 1302, and 1303 displayed on the head-mounted display device are different.

[0131] Optionally, the driving image data can be a rear view image obtained from the vehicle's rearview mirrors from the driver's perspective, such as... Figure 15 As shown, Figure 15 This is a schematic diagram of driving image data of a vehicle from the driver's perspective, provided in another embodiment of this application, wherein 1501 is a schematic diagram of the left electronic rearview mirror and 1502 is a schematic diagram of the right electronic rearview mirror.

[0132] In some embodiments, the central computing processing unit may acquire the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor; determine the sharpness and / or transparency of the image region corresponding to the position and orientation of the eyeball in the reference coordinate system of the vehicle's external perception sensor based on the position and orientation of the driver's eyeball in the reference coordinate system; and send the sharpness and / or transparency of the image region to the head-mounted display device so that the head-mounted display device displays the image region in the driving image data according to the sharpness and / or transparency.

[0133] When the central computing processing unit acquires the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensors, it can obtain the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensors from a preset database. This preset database is used to store data that has been acquired or calculated in advance. The central computing unit can also collect the driver's eye data through the sensing sensors inside the head-mounted display, and calculate the driver's eye position and orientation in the reference coordinate system of the sensing sensors inside the head-mounted display based on the eye data and a preset eye-tracking algorithm; identify the position of a preset reference point on the head-mounted display from the driver's head posture data collected by the sensing sensors inside the cockpit; calculate the position and orientation of the head-mounted display in the reference coordinate system of the sensing sensors inside the cockpit based on the head posture data; and calculate the position and orientation of the driver's eyeballs in the reference coordinate system of the external sensing sensors based on the driver's eye position and orientation in the reference coordinate system of the sensing sensors inside the head-mounted display, the position of the preset reference point of the head-mounted display, the position and orientation of the head-mounted display in the reference coordinate system of the sensing sensors inside the cockpit, and the geometric positional relationship between the reference coordinate system of the sensing sensors inside the cockpit and the reference coordinate system of the external sensing sensors.

[0134] In one example, the rearview view in the driving image data can be adjusted based on eye-tracking data. When the driver's eye gaze direction (i.e., the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external perception sensors) is detected to be towards the rearview view area, or when the eye gaze direction shows a trend of movement towards the rearview view area, the rearview view displays a high-resolution (e.g., 4320P) and opaque (e.g., 0% transparency) image. When the driver's eye gaze direction (i.e., the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external perception sensors) is detected to be towards the frontview view area, the rearview view can increase transparency (e.g., 80% transparency) or decrease resolution (e.g., 1080P). By determining the resolution and / or transparency of the image area corresponding to the position and orientation of the driver's eyes in the reference coordinate system of the vehicle's external perception sensors, it helps to reduce interference with the driver's information input. In some embodiments, to avoid altering the driver's driving habits, the rearview view can be obtained directly from a sensing sensor installed in the traditional rearview mirror location, or it can be synthesized by combining images acquired by sensing sensors installed in other locations on the vehicle body after changing the viewing angle. Additionally, the rearview view retains the vehicle body outline image for the driver's reference; however, the vehicle body outline can be displayed with preset transparency or preset line type (such as dashed lines) to eliminate blind spots caused by vehicle body obstruction. The position of the rearview view can be positioned at the driver's eye level, while ensuring safety, by controlling the angle of head and eye rotation required for the driver to view the rearview view; alternatively, it can be positioned on either side of the A-pillar, following traditional driving habits.

[0135] In one embodiment, the central computing processing unit can identify the road information on which the vehicle is traveling based on the first information; when it is determined that the road information on which the vehicle is traveling meets the driving prompt conditions, it generates corresponding driving prompt information and sends the driving prompt information to the head-mounted display device so that the head-mounted display device displays the driving prompt information.

[0136] When the central computing unit determines that there are obstacles on the road the vehicle is traveling on based on road information, it can determine that the road information meets the driving prompt conditions; generate obstacle marker image data and / or voice prompt information, and send the obstacle marker image data and / or voice prompt information to the head-mounted display device, so that the head-mounted display device can display the obstacle marker image data and / or output the voice prompt information. For example, information such as potholes and road obstacles can be marked with preset lines or preset transparent colors to provide a warning.

[0137] When the central computing unit determines that road traffic signs exist on the road the vehicle is traveling on, and that the road information meets the driving prompt conditions, it generates road traffic sign image data and / or voice prompt information, and sends this data to a head-mounted display device. The head-mounted display device then displays the road traffic sign image data and / or outputs the voice prompt information. The road traffic sign information may include, but is not limited to, information on permitted driving directions at intersections and road speed limits.

[0138] When the central computing unit determines that the road surface is slippery (such as in rainy weather) based on road information, it can determine that the road information meets the driving warning conditions, generate safety warning image data and / or voice prompts, and send the road traffic sign image data and / or voice prompts to the head-mounted display device, so that the head-mounted display device can display the safety warning image data and / or output the voice prompts. The safety warnings and voice prompts may include, but are not limited to, current braking distance, safe following distance, and vehicle passability warnings. Vehicle passability warnings can be provided by measuring and displaying information on the depth of water on the road surface.

[0139] The central computing and processing unit can also generate defensive driving prompts based on road information and send them to the head-mounted display device for display. These prompts can be text or voice messages, and their content may include, but is not limited to: intersection slow-down reminders, slow-down reminders for pedestrians appearing unexpectedly, oncoming vehicle reminders at turns, and reminders for vehicles overtaking from behind.

[0140] The driver assistance information prompt function displays driver assistance information in an area that is at the driver's eye level and does not obstruct road information, including but not limited to displaying vehicle status information: driving speed, remaining driving range, in-vehicle temperature and humidity, tire pressure status, etc.

[0141] In some embodiments, the central computing processing unit can render driver assistance information in an area that is directly in the driver's line of sight and does not obstruct road information, and send the driver assistance information to a head-mounted display device so that the head-mounted display device can display the driver assistance information. The driver assistance information may include, but is not limited to, vehicle status information such as vehicle speed, remaining driving range, in-vehicle temperature and humidity, tire pressure, weather information, and time information.

[0142] like Figure 16 As shown, Figure 16This is a schematic diagram of driving image data of a vehicle from a driver's perspective, provided in another embodiment of this application. 1601 displays driving status and navigation information at a fixed location, such as remaining mileage, current speed, road speed limit information, and navigation information. The navigation information, in addition to being displayed at a fixed location, can interact with the actual road image to generate dynamic road navigation information, such as lane selection and turn prompts. 1602 displays information such as weather, vehicle interior and exterior temperature, and time. 1603 displays road vehicle interaction data, showing the distance and speed of other vehicles on the road. In addition, it can also annotate road obstacle information, display road surface slipperiness, indicate braking distance and safe following distance, display overtaking prompts for vehicles behind, provide defensive driving prompts (speed reduction at intersections, speed reduction prompts in areas with pedestrians suddenly appearing), and indicate vehicle passability after measuring the depth of roadside water.

[0143] In some embodiments, the driving image data can be a fixed forward-viewing view from the driver's forward-looking perspective. This forward-viewing view can be a view from the perspective where the driver's head is facing the front of the vehicle and their eyes are focused in a target direction (such as directly in front of the vehicle). This target direction can be a pre-defined direction. By sending the fixed forward-viewing view from the driver's forward-looking perspective to the head-mounted display device, the image displayed to the driver by the head-mounted display device is always a forward-looking view from the optimal driving position, and the view does not change with the movement of the driver's head, thus achieving a view-keeping function. By fixing the driving image data to a forward-looking view from the driver's perspective, it is possible to still clearly see the vehicle's forward view from a fixed perspective even when the driver's head needs to deviate from the road's line of sight, thereby further improving driving safety.

[0144] For example, after a long drive, the driver's head remains in a fixed position, causing neck and shoulder stiffness and soreness. By fixing the driving image data to the driver's forward-facing view, the driver's field of vision remains the road ahead even when moving their head. Or, when the driver needs to pick something up, their body posture deviates from the normal driving posture, causing their gaze to leave the road, thus posing a driving safety hazard. By fixing the driving image data to the driver's forward-facing view, the driver's field of vision remains the road ahead in these situations, helping to reduce the occurrence of safety accidents and further improving driving safety.

[0145] In some embodiments, the central computing processing unit may acquire the driver's eye data; determine the driver's fatigue level based on the eye data; when the fatigue level is determined to be greater than a preset threshold, generate fatigue warning information and send the fatigue warning information to a head-mounted display device so that the head-mounted display device displays the fatigue warning information; and / or, when the fatigue level is determined to be greater than a preset threshold, generate fatigue voice warning information and send the fatigue voice warning information to the head-mounted display device so that the head-mounted display device plays the fatigue warning information.

[0146] In some embodiments, the central computing processing unit can acquire the driver's eye data, determine the driver's fatigue level based on the eye data, and when the fatigue level is determined to be greater than a preset threshold, generate an automatic stop control command and send the automatic stop control command to the vehicle so that the vehicle enters the automatic driving mode and responds to the automatic stop control command to control the vehicle to brake and stop.

[0147] In some embodiments, when a command to activate the assisted driving mode is received, the central computing unit can determine whether the vehicle's current driving mode meets the conditions for assisted driving based on first information. If it determines that the vehicle's current driving mode meets the conditions for assisted driving, it sends a control command to the vehicle to enter the assisted driving mode, thereby activating the vehicle's visual assisted driving system, causing the vehicle to enter the assisted driving mode, and triggering the step of acquiring the first information. In some embodiments, the command to activate the assisted driving mode can be generated by the driver touching a assisted driving mode activation switch. After entering the assisted driving mode, the vehicle can be driven with the assistance of the driver's head-mounted display.

[0148] Optionally, the external vehicle perception sensors, the driver's cabin perception sensors, and the perception sensors on the head-mounted display can be set to automatically turn off when the vehicle is not running, or can be turned off by the driver to protect the privacy of the driver and passengers. When entering the assisted driving mode or the autonomous driving mode, the perception sensors are triggered to turn on and cannot be turned off.

[0149] Optionally, when the central computing processing unit detects that the vehicle is in a parked state, it can generate a full 360-degree image of the vehicle as driving image data and send the full 360-degree image of the vehicle to the head-mounted display device so that the full 360-degree image of the vehicle is displayed on the head-mounted display device.

[0150] Optionally, the central computing unit can send a rear-view image of the vehicle from the driver's perspective to the head-mounted display when it detects that the vehicle is reversing. By mirroring the image left and right or changing the steering wheel signal, the driver can perceive the vehicle as if it were driving forward normally. That is, when turning the steering wheel to the left, the rear of the vehicle in the image moves to the left front, and when turning the steering wheel to the right, the rear of the vehicle moves to the right front. This method allows the driver to have the same tactile experience when reversing as when driving forward, improving control and accuracy during reversing.

[0151] This application embodiment displays driving image data from the driver's perspective using different transparency, clarity, and display positions in different scenarios. This provides the driver with a vehicle position reference while reducing differences in driving habits and eliminating blind spots. Furthermore, by outputting various corresponding prompts or voice prompts on the head-mounted display device when prompts are needed, it helps to further improve vehicle driving safety.

[0152] Please see Figure 17 , Figure 17 This is a schematic diagram of the structure of an assisted driving device provided in an embodiment of this application. Specifically, the assisted driving device is disposed in a computing device, and the assisted driving device includes: a first acquisition unit 1701, a second acquisition unit 1702, a third acquisition unit 1703, and a generation unit 1704;

[0153] The first acquisition unit 1701 is used to acquire first information, which is obtained based on the external environment information around the vehicle collected by the first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one position on the vehicle body.

[0154] The second acquisition unit 1702 is used to acquire second information, which is obtained based on the internal environment information of the vehicle collected by the second set of perception sensors installed in the vehicle. The second information includes data from the driver's perspective, driver's eye data, and driver's head posture data.

[0155] The third acquisition unit 1703 is used to acquire third information, which is obtained based on the vehicle state.

[0156] The generation unit 1704 is used to generate driving image data of the vehicle from the driver's perspective based on the first information, the second information and the third information, and the driving image data is used to assist driving.

[0157] Furthermore, the driver assistance device also includes: a transmitting unit 1705,

[0158] The transmitting unit 1705 is used to transmit the driving image data to the head-mounted display device, so that the head-mounted display device displays the driving image data to assist the driver in driving the vehicle based on the driving image data displayed by the head-mounted display device.

[0159] Furthermore, the second set of sensing sensors includes: external sensing sensors for the head-mounted display device, internal sensing sensors for the head-mounted display device, and internal sensing sensors for the cockpit; the second information includes:

[0160] Data collected from the driver's perspective by external sensing sensors of the head-mounted display device;

[0161] The driver's eye data is collected by the sensing sensors inside the head-mounted display device;

[0162] The driver's head posture data is collected by the sensing sensors inside the cockpit.

[0163] Furthermore, the generation unit 1704 generates driving image data of the vehicle from the driver's perspective based on the first information, the second information, and the third information. When used to assist driving, the driving image data is specifically used for:

[0164] The vehicle's current external environment type is determined based on the first information, and multiple corresponding external environment data are selected from the first information based on the external environment type. The multiple external environment data are then fused to obtain target external environment data.

[0165] Interactive region image data is extracted from the viewpoint data, and the interactive region image data is segmented to obtain segmented image data.

[0166] Based on the eye data and head posture data, as well as the geometric positional relationship between the reference coordinate system of the driver's cockpit interior perception sensor and the reference coordinate system of the vehicle exterior perception sensor, the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle exterior perception sensor are calculated.

[0167] Based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, and the third information, the driving image data of the vehicle from the driver's perspective is determined.

[0168] Furthermore, when the generation unit 1704 performs segmentation processing on the interactive region image data to obtain segmented image data, it is specifically used for:

[0169] Acquire the cockpit interior image data and cockpit exterior environment image data from the image data of the interactive area;

[0170] The image data inside the cockpit and the image data of the external environment of the cockpit are segmented, and the image data inside the cockpit is determined to be the segmented image data.

[0171] Furthermore, when the generation unit 1704 calculates the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensors based on the eye data and the head posture data, it is specifically used for:

[0172] Based on the eye data and a preset human eye tracking algorithm, the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display device are calculated.

[0173] The position of a preset reference point on the head-mounted display device is identified from the head posture data;

[0174] The position and orientation of the head-mounted display device in the reference coordinate system of the sensing sensors inside the cockpit are calculated based on the head posture data.

[0175] Based on the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display, the position of the preset reference point of the head-mounted display, the position and attitude of the head-mounted display in the reference coordinate system of the sensor inside the driver's cockpit, and the geometric positional relationship between the reference coordinate system of the sensor inside the driver's cockpit and the reference coordinate system of the external vehicle sensor, the position and orientation of the driver's eyeball in the reference coordinate system of the external vehicle sensor are calculated.

[0176] Further, when the generation unit 1704 determines the driving image data of the vehicle from the driver's perspective based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, and the third information, it is specifically used for:

[0177] Based on the target external environment data, the segmented image data, and the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the external environment image data layer and the driver's cockpit internal interactive environment image data layer from the driver's perspective.

[0178] The vehicle's driving data layer is determined based on the vehicle status data;

[0179] The external environment image data layer, the driver's cockpit interior interactive environment image data layer, and the driving data layer are fused to obtain driving image data of the vehicle from the driver's perspective.

[0180] Furthermore, the transmitting unit 1705 is used for:

[0181] When a command to activate the assisted driving mode is received, the system determines whether the current driving mode of the vehicle meets the conditions for assisted driving based on the first information.

[0182] When it is determined that the current driving mode of the vehicle meets the conditions for assisted driving, a control command to enter the assisted driving mode is sent to the vehicle to activate the vehicle's visual assisted driving system, so that the vehicle enters the assisted driving mode and triggers the execution of the step of acquiring the first information about the vehicle's surroundings collected by the vehicle's first set of perception sensors.

[0183] Furthermore, the transmitting unit 1705 is also used for:

[0184] The position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensors are obtained;

[0185] Based on the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the sharpness and / or transparency of the image area corresponding to the position and orientation of the eyeball in the reference coordinate system of the vehicle's external perception sensor.

[0186] The image region's sharpness and / or transparency are sent to the head-mounted display device so that the head-mounted display device displays the image region in the driving image data according to the sharpness and / or transparency.

[0187] Furthermore, the vehicle's outline in the driving image data is displayed on the head-mounted display device with a preset transparency or preset line shape.

[0188] Furthermore, the driving image data refers to a forward view with the driver's head facing the front of the vehicle.

[0189] Furthermore, the transmitting unit 1705 is also used for:

[0190] Acquire the driver's eye data;

[0191] The driver's fatigue level is determined based on the eye data;

[0192] When the fatigue level is determined to be greater than a preset threshold, a fatigue warning message is generated and sent to the head-mounted display device, so that the head-mounted display device displays the fatigue warning message; and / or,

[0193] When the fatigue level is determined to be greater than a preset threshold, a fatigue voice prompt is generated and sent to the head-mounted display device so that the head-mounted display device can play the fatigue prompt.

[0194] Furthermore, the transmitting unit 1705 is also used for:

[0195] Identify the road information on which the vehicle is traveling based on the first information;

[0196] When it is determined that the road information on which the vehicle is traveling meets the driving prompt conditions, corresponding driving prompt information is generated and sent to the head-mounted display device so that the head-mounted display device can display the driving prompt information.

[0197] Furthermore, when the sending unit 1705 determines that the road information of the vehicle meets the driving prompt conditions, and generates the corresponding driving prompt information, it is specifically used for:

[0198] When it is determined from the road information that there is an obstacle on the road the vehicle is traveling on, it is determined that the road information meets the driving prompt conditions;

[0199] Obstacle marker image data and / or voice prompt information are generated and sent to the head-mounted display device so that the head-mounted display device displays the obstacle marker image data and / or outputs the voice prompt information.

[0200] This application embodiment acquires first information about the vehicle's surroundings collected by a first set of vehicle perception sensors, including multiple external vehicle perception sensors, each of different types and positioned at different locations on the vehicle body. It then acquires second information about the vehicle collected by a second set of vehicle perception sensors, including data from the driver's perspective, driver's eye data, and driver's head posture data. Finally, it acquires third information based on the vehicle's state and, based on the first, second, and third information, determines driving image data from the driver's perspective. This driving image data is then sent to a head-mounted display device for display. By utilizing multiple external vehicle perception sensors of different types positioned at different locations on the vehicle body to fully collect external environmental information, visual interference problems are effectively addressed. The acquisition of perspective data, driver's eye data, and head posture data helps generate driving image data from the driver's perspective, allowing for better driver assistance in driving the vehicle based on the driving image data displayed on the head-mounted display device, thereby improving driving safety.

[0201] See Figure 18 , Figure 18 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. The computing device includes: a memory 1801 and a processor 1802.

[0202] In one embodiment, the computing device further includes a data interface 1803 for transmitting data information between the computing device and other devices.

[0203] The memory 1801 may include volatile memory; the memory 1801 may also include non-volatile memory; the memory 1801 may also include a combination of the above types of memory. The processor 1802 may be a central processing unit (CPU). The processor 1802 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), or any combination thereof.

[0204] The memory 1801 is used to store programs, and the processor 1802 can call the programs stored in the memory 1801 to perform the following steps:

[0205] First information is obtained, which is based on the external environment information around the vehicle collected by the first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one position on the vehicle body.

[0206] The second information is obtained based on the internal environment information of the vehicle collected by the second set of perception sensors installed in the vehicle. The second information includes data from the driver's perspective, driver's eye data, and driver's head posture data.

[0207] Obtain third information, which is derived based on the vehicle state;

[0208] Based on the first information, the second information, and the third information, driving image data of the vehicle from the driver's perspective is generated, and the driving image data is used to assist driving.

[0209] Furthermore, the processor 1802 is also used for:

[0210] The driving image data is sent to a head-mounted display device so that the head-mounted display device can display the driving image data to assist the driver in driving the vehicle based on the driving image data displayed by the head-mounted display device.

[0211] Furthermore, the second set of sensing sensors includes: external sensing sensors for the head-mounted display device, internal sensing sensors for the head-mounted display device, and internal sensing sensors for the cockpit; the second information includes:

[0212] Data collected from the driver's perspective by external sensing sensors of the head-mounted display device;

[0213] The driver's eye data is collected by the sensing sensors inside the head-mounted display device;

[0214] The driver's head posture data is collected by the sensing sensors inside the cockpit.

[0215] Furthermore, the processor 1802 generates driving image data of the vehicle from the driver's perspective based on the first information, the second information, and the third information. When used to assist driving, the driving image data is specifically used for:

[0216] The vehicle's current external environment type is determined based on the first information, and multiple corresponding external environment data are selected from the first information based on the external environment type. The multiple external environment data are then fused to obtain target external environment data.

[0217] Interactive region image data is extracted from the viewpoint data, and the interactive region image data is segmented to obtain segmented image data.

[0218] Based on the eye data and head posture data, as well as the geometric positional relationship between the reference coordinate system of the driver's cockpit interior perception sensor and the reference coordinate system of the vehicle exterior perception sensor, the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle exterior perception sensor are calculated.

[0219] Based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, and the third information, the driving image data of the vehicle from the driver's perspective is determined.

[0220] Furthermore, when the processor 1802 performs segmentation processing on the interactive region image data to obtain segmented image data, it is specifically used for:

[0221] Acquire the cockpit interior image data and cockpit exterior environment image data from the image data of the interactive area;

[0222] The image data inside the cockpit and the image data of the external environment of the cockpit are segmented, and the image data inside the cockpit is determined to be the segmented image data.

[0223] Furthermore, when the processor 1802 calculates the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensors based on the eye data and the head posture data, it is specifically used for:

[0224] Based on the eye data and a preset human eye tracking algorithm, the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display device are calculated.

[0225] The position of a preset reference point on the head-mounted display device is identified from the head posture data;

[0226] The position and orientation of the head-mounted display device in the reference coordinate system of the sensing sensors inside the cockpit are calculated based on the head posture data.

[0227] Based on the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display, the position of the preset reference point of the head-mounted display, the position and attitude of the head-mounted display in the reference coordinate system of the sensor inside the driver's cockpit, and the geometric positional relationship between the reference coordinate system of the sensor inside the driver's cockpit and the reference coordinate system of the external vehicle sensor, the position and orientation of the driver's eyeball in the reference coordinate system of the external vehicle sensor are calculated.

[0228] Further, when the processor 1802 determines the driving image data of the vehicle from the driver's perspective based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, and the third information, it is specifically used for:

[0229] Based on the target external environment data, the segmented image data, and the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the external environment image data layer and the driver's cockpit internal interactive environment image data layer from the driver's perspective.

[0230] The vehicle's driving data layer is determined based on the vehicle status data;

[0231] The external environment image data layer, the driver's cockpit interior interactive environment image data layer, and the driving data layer are fused to obtain driving image data of the vehicle from the driver's perspective.

[0232] Furthermore, the processor 1802 is also used for:

[0233] When a command to activate the assisted driving mode is received, the system determines whether the current driving mode of the vehicle meets the conditions for assisted driving based on the first information.

[0234] When it is determined that the current driving mode of the vehicle meets the conditions for assisted driving, a control command to enter the assisted driving mode is sent to the vehicle to activate the vehicle's visual assisted driving system, so that the vehicle enters the assisted driving mode and triggers the execution of the step of acquiring the first information about the vehicle's surroundings collected by the vehicle's first set of perception sensors.

[0235] Furthermore, the processor 1802 is also used for:

[0236] The position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensors are obtained;

[0237] Based on the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the sharpness and / or transparency of the image area corresponding to the position and orientation of the eyeball in the reference coordinate system of the vehicle's external perception sensor.

[0238] The image region's sharpness and / or transparency are sent to the head-mounted display device so that the head-mounted display device displays the image region in the driving image data according to the sharpness and / or transparency.

[0239] Furthermore, the vehicle's outline in the driving image data is displayed on the head-mounted display device with a preset transparency or preset line shape.

[0240] Furthermore, the driving image data refers to a forward view with the driver's head facing the front of the vehicle.

[0241] Furthermore, the processor 1802 is also used for:

[0242] Acquire the driver's eye data;

[0243] The driver's fatigue level is determined based on the eye data;

[0244] When the fatigue level is determined to be greater than a preset threshold, a fatigue warning message is generated and sent to the head-mounted display device, so that the head-mounted display device displays the fatigue warning message; and / or,

[0245] When the fatigue level is determined to be greater than a preset threshold, a fatigue voice prompt is generated and sent to the head-mounted display device so that the head-mounted display device can play the fatigue prompt.

[0246] Furthermore, the processor 1802 is also used for:

[0247] Identify the road information on which the vehicle is traveling based on the first information;

[0248] When it is determined that the road information on which the vehicle is traveling meets the driving prompt conditions, corresponding driving prompt information is generated and sent to the head-mounted display device so that the head-mounted display device can display the driving prompt information.

[0249] Furthermore, when the processor 1802 determines that the road information on which the vehicle is traveling meets the driving prompt conditions, and generates the corresponding driving prompt information, it is specifically used for:

[0250] When it is determined from the road information that there is an obstacle on the road the vehicle is traveling on, it is determined that the road information meets the driving prompt conditions;

[0251] Obstacle marker image data and / or voice prompt information are generated and sent to the head-mounted display device so that the head-mounted display device displays the obstacle marker image data and / or outputs the voice prompt information.

[0252] In this embodiment, the computing device can acquire first information about the vehicle's surroundings collected by a first set of vehicle perception sensors, which includes multiple external vehicle perception sensors, each of which is a different type of sensor arranged at different locations on the vehicle body; acquire second information about the vehicle collected by a second set of vehicle perception sensors, including data from the driver's perspective, driver's eye data, and driver's head posture data; acquire third information based on the vehicle's state, and determine driving image data of the vehicle from the driver's perspective based on the first, second, and third information; and send the driving image data to a head-mounted display device so that the head-mounted display device can display the driving image data. By fully collecting external environmental information through multiple external vehicle perception sensors of different types arranged at different locations on the vehicle body, the problem of visual interference can be solved. By collecting perspective data, driver's eye data, and head posture data, driving image data of the driver's perspective and the interaction between the driver and the cockpit can be generated, thereby better assisting the driver in driving the vehicle based on the driving image data displayed on the head-mounted display device and improving vehicle driving safety.

[0253] It should be understood that, in the embodiments of this application, the processor 1802 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0254] The memory 1801 may include read-only memory and random access memory, and provides instructions and data to the processor 1802. A portion of the memory 1801 may also include non-volatile random access memory. For example, the memory 1801 may also store device type information.

[0255] In specific implementations, the processor 1802 described in this application embodiment can execute the implementation method described in the assisted driving method provided in this application embodiment, or it can execute the implementation method of the assisted driving device described in this application embodiment, which will not be repeated here.

[0256] This application also provides a vehicle that includes the computing device described above.

[0257] This application also provides a computer-readable storage medium storing program instructions that, when executed, implement the assisted driving method described above.

[0258] The computer-readable storage medium can be an internal storage unit of the device described in any of the foregoing embodiments, such as the device's hard drive or memory. The computer-readable storage medium can also be an external storage device of the device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device. Further, the computer-readable storage medium may include both internal and external storage units of the device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computing device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0259] Embodiments of this application also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to perform the methods provided in the various embodiments described above.

[0260] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0261] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0262] In the several embodiments provided in this application, it should be understood that the disclosed computing devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0263] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0264] The units in the terminal of this application embodiment can be merged, divided, and deleted according to actual needs.

[0265] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0266] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0267] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0268] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A driving assistance method, characterized in that, include: First information is obtained, which is based on the external environment information around the vehicle collected by the first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one position on the vehicle body. The second information is obtained based on the internal environment information of the vehicle collected by a second set of sensing sensors installed in the vehicle. The second set of sensing sensors includes an external sensing sensor of the head-mounted display device, an internal sensing sensor of the head-mounted display device, and an internal sensing sensor of the driver's cockpit. The second information includes perspective data from the driver's perspective collected by the external sensing sensor of the head-mounted display device, eye data of the driver collected by the internal sensing sensor of the head-mounted display device, and head posture data of the driver collected by the internal sensing sensor of the driver's cockpit. The second information includes image data of the cockpit interior and image data of the cockpit exterior environment. The methods for obtaining the cockpit interior and exterior image data included in the second information include: Extract interactive area image data from the perspective data; Acquire the cockpit interior image data and cockpit exterior environment image data from the image data of the interactive area; Obtain third information, which is derived based on the vehicle state; Based on the first information, the second information, and the third information, driving image data of the vehicle from the driver's perspective is generated, and the driving image data is used to assist driving. The process of generating driving image data of the vehicle from the driver's perspective based on the first information, the second information, and the third information, wherein the driving image data is used to assist driving, includes: The vehicle's current external environment type is determined based on the first information, and multiple corresponding external environment data are selected from the first information based on the external environment type. The multiple external environment data are then fused to obtain target external environment data. Interactive region image data is extracted from the viewpoint data, and the interactive region image data is segmented to obtain segmented image data. Based on the eye data and head posture data, as well as the positional relationship between the reference coordinate system of the driver's cockpit interior perception sensor and the reference coordinate system of the vehicle exterior perception sensor, the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle exterior perception sensor are calculated. Based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, and the third information, the driving image data of the vehicle from the driver's perspective is determined. The step of segmenting the image data of the interactive region to obtain segmented image data includes: Acquire the cockpit interior image data and cockpit exterior environment image data from the image data of the interactive area; The image data inside the cockpit and the image data of the external environment of the cockpit are segmented, and the image data inside the cockpit is determined to be the segmented image data.

2. The method according to claim 1, further comprising: The driving image data is sent to a head-mounted display device so that the head-mounted display device can display the driving image data to assist the driver in driving the vehicle based on the driving image data displayed by the head-mounted display device.

3. The method according to claim 1, characterized in that, The step of calculating the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external sensing sensors based on the eye data, the head posture data, and the positional relationship between the reference coordinate system of the driver's cockpit interior sensing sensor and the reference coordinate system of the vehicle exterior sensing sensor includes: Based on the eye data and a preset human eye tracking algorithm, the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display device are calculated. The position of a preset reference point on the head-mounted display device is identified from the head posture data; The position and orientation of the head-mounted display device in the reference coordinate system of the sensing sensors inside the cockpit are calculated based on the head posture data. Based on the driver's eye position and orientation in the reference coordinate system of the sensor inside the head-mounted display, the position of the preset reference point of the head-mounted display, the position and attitude of the head-mounted display in the reference coordinate system of the sensor inside the driver's cockpit, and the geometric positional relationship between the reference coordinate system of the sensor inside the driver's cockpit and the reference coordinate system of the external vehicle sensor, the position and orientation of the driver's eyeball in the reference coordinate system of the external vehicle sensor are calculated.

4. The method according to claim 1, characterized in that, The step of determining the driving image data of the vehicle from the driver's perspective based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, and the third information includes: Based on the target external environment data, the segmented image data, and the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the external environment image data layer and the driver's cockpit internal interactive environment image data layer from the driver's perspective. The vehicle's driving data layer is determined based on the vehicle status data; The external environment image data layer, the driver's cockpit interior interactive environment image data layer, and the driving data layer are fused to obtain driving image data of the vehicle from the driver's perspective.

5. The method according to claim 2, characterized in that, The method further includes: When a command to activate the assisted driving mode is received, the system determines whether the current driving mode of the vehicle meets the conditions for assisted driving based on the first information. When it is determined that the current driving mode of the vehicle meets the conditions for assisted driving, a control command to enter the assisted driving mode is sent to the vehicle to activate the vehicle's visual assisted driving system, so that the vehicle enters the assisted driving mode and triggers the execution of the step of obtaining the first information.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensors are obtained; Based on the position and orientation of the driver's eyeball in the reference coordinate system of the vehicle's external perception sensor, determine the sharpness and / or transparency of the image area corresponding to the position and orientation of the eyeball in the reference coordinate system of the vehicle's external perception sensor. The image region's sharpness and / or transparency are sent to the head-mounted display device so that the head-mounted display device displays the image region in the driving image data according to the sharpness and / or transparency.

7. The method according to any one of claims 1-5, characterized in that, The vehicle's outline in the driving image data is displayed on the head-mounted display device with a preset transparency or preset line shape.

8. The method according to any one of claims 1-5, characterized in that, The driving image data refers to the forward view of the driver's head facing the front of the vehicle.

9. The method according to any one of claims 1-5, characterized in that, The method further includes: Acquire the driver's eye data; The driver's fatigue level is determined based on the eye data; When the fatigue level is determined to be greater than a preset threshold, a fatigue warning message is generated and sent to the head-mounted display device, so that the head-mounted display device displays the fatigue warning message; and / or, When the fatigue level is determined to be greater than a preset threshold, a fatigue voice prompt is generated and sent to the head-mounted display device so that the head-mounted display device can play the fatigue prompt.

10. The method according to any one of claims 1-5, characterized in that, The method further includes: Identify the road information on which the vehicle is traveling based on the first information; When it is determined that the road information on which the vehicle is traveling meets the driving prompt conditions, corresponding driving prompt information is generated and sent to the head-mounted display device so that the head-mounted display device can display the driving prompt information.

11. The method according to claim 10, characterized in that, When it is determined that the road information on which the vehicle is traveling meets the driving prompt conditions, corresponding driving prompt information is generated, including: When it is determined from the road information that there is an obstacle on the road the vehicle is traveling on, it is determined that the road information meets the driving prompt conditions; Obstacle marker image data and / or voice prompt information are generated and sent to the head-mounted display device so that the head-mounted display device displays the obstacle marker image data and / or outputs the voice prompt information.

12. A driver assistance device, characterized in that, include: The first acquisition unit is used to acquire first information, which is obtained based on the external environment information around the vehicle collected by the first set of vehicle perception sensors. The first set of perception sensors includes at least one vehicle external perception sensor, and each vehicle external perception sensor is a sensor arranged at at least one position on the vehicle body. The second acquisition unit is used to acquire second information, which is obtained based on the internal environment information of the vehicle collected by the second set of perception sensors installed in the vehicle. The second set of perception sensors includes an external perception sensor of the head-mounted display device, an internal perception sensor of the head-mounted display device, and an internal perception sensor of the driver's cockpit. The second information includes perspective data from the driver's perspective collected by the external perception sensor of the head-mounted display device, eye data of the driver collected by the internal perception sensor of the head-mounted display device, and head posture data of the driver collected by the internal perception sensor of the driver's cockpit. The second information includes cockpit interior image data and cockpit exterior environment image data. The second acquisition unit acquires the cockpit interior image data and cockpit exterior environment image data included in the second information by: extracting interactive area image data from the viewpoint data; and acquiring the cockpit interior image data and cockpit exterior environment image data from the interactive area image data. The third acquisition unit is used to acquire third information, which is obtained based on the vehicle state. The generation unit is used to generate driving image data of the vehicle from the driver's perspective based on the first information, the second information and the third information, and the driving image data is used to assist driving. The generation unit generates driving image data of the vehicle from the driver's perspective based on the first information, the second information, and the third information. When used to assist driving, this driving image data specifically involves: determining the current external environment type of the vehicle based on the first information; selecting multiple corresponding external environment data from the first information based on the external environment type; fusing the multiple external environment data to obtain target external environment data; extracting interactive area image data from the perspective data; segmenting the interactive area image data to obtain segmented image data; calculating the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensor based on the eye data, the head posture data, and the positional relationship between the reference coordinate system of the driver's cockpit interior perception sensor and the reference coordinate system of the vehicle's external perception sensor; and determining the driving image data of the vehicle from the driver's perspective based on the target external environment data, the segmented image data, the position and orientation of the driver's eyeballs in the reference coordinate system of the vehicle's external perception sensor, and the third information. The generation unit performs segmentation processing on the interactive area image data to obtain segmented image data. Specifically, it acquires the cockpit interior image data and the cockpit exterior environment image data from the interactive area image data; it performs segmentation processing on the cockpit interior image data and the cockpit exterior environment image data, and determines the cockpit interior image data as the segmented image data.

13. A computing device, characterized in that, The device includes a processor and a memory interconnected, wherein the memory is used to store a computer program, and the processor is configured to invoke the computer program to perform the method as described in any one of claims 1-11.

14. A vehicle, characterized in that, The vehicle includes the computing device as described in claim 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed, implement the method as described in any one of claims 1-11.

16. A computer program product, characterized in that, The computer program product includes program instructions that, when executed by a processor, implement the method described in any one of claims 1-11.

Images