Image display method and related device, vehicle, storage medium and program

Abstract

Embodiments of the present application provide an image display method and related device, vehicle, storage medium and program. The method comprises: obtaining comprehensive state information of the vehicle; predicting a target region which needs to be observed by a driver's eye and is in an occluded state according to the comprehensive state information; and controlling a left screen of a left A-pillar and / or a right screen of a right A-pillar of the vehicle to display images according to the target region. In the A-pillar image display scene of the intelligent cabin of the vehicle, compared with the existing single and fixed display fixed framing range solution, the embodiments of the present application are beneficial to improve the flexibility, accuracy and comprehensiveness of the image display of the automatic driving domain controller, improve the driving safety of the vehicle, and improve the driving experience of the user.

Description

Technical Field

[0001] This application relates to the field of automotive safe driving technology, and in particular to image display methods and related devices, vehicles, storage media and programs based on the driver's field of vision in driving scenarios. Background Technology

[0002] In recent years, traffic accidents involving collisions between cars and pedestrians have been increasing, causing significant harm to pedestrians. The A-pillar blind spot is a major contributing factor to these accidents. The A-pillar, located at the front left and right, connects the roof and the front cabin, primarily supporting the windshield and roof. During actual driving, the A-pillar inevitably obstructs the driver's view to some extent, creating a blind spot, especially during turns. Drivers must then sway their bodies and heads to adjust their vision and overcome this blind spot, which is both inconvenient and unsafe.

[0003] Currently, most solutions to the blind spot problem in cars use rearview mirrors or lenses to obtain information about objects in the blind spot through the refraction or reflection of light, or use cameras to capture images blocked by the A-pillar and display them on the A-pillar.

[0004] However, when using products like rearview mirrors or lenses, the image obtained by the driver is relatively small due to light refraction and reflection, and sometimes it is even a mirror image. This undoubtedly increases the driver's reaction time. Moreover, when the light is strong, the light reflected into the driver's eyes can easily cause glare and affect driving safety. Furthermore, using a camera to capture the image obscured by the A-pillar and project it onto the corresponding A-pillar is only useful in specific situations such as when the vehicle is turning or there are obstacles around the vehicle. If it is kept in working condition at other times, it has no practical significance and may even affect driving safety. It also cannot eliminate the safety hazards caused by the A-pillar blind spot during driving.

[0005] A relatively intelligent way to solve the problem of blind spots in cars is usually to obtain external images through devices such as external cameras and project them onto a screen. However, in general, the image displayed on the screen is largely the image obtained by the camera, without taking into account comfort and accuracy. Furthermore, there will be differences between the image captured by the camera and the image captured by the driver's eyes, which may cause the driver to be distracted by the screen and cause problems.

[0006] Therefore, how to safely eliminate the blind spots caused by the A-pillars on both sides of a car and the resulting safety hazards is an urgent problem that needs to be solved. Summary of the Invention

[0007] This application provides an image display method and device based on the driver's field of vision in a driving scenario. In the A-pillar image display scenario of a vehicle's smart cockpit, compared with the existing single, fixed display with a fixed field of view, it is beneficial to improve the flexibility, accuracy and comprehensiveness of the autonomous driving domain controller in image display, improve vehicle driving safety and enhance the user's driving experience.

[0008] In a first aspect, embodiments of this application provide an image display method based on the driver's field of vision in a driving scenario. The method is applied to an autonomous driving domain controller of a vehicle's domain controller system. The domain controller system includes the autonomous driving domain controller and a body domain controller, and the autonomous driving domain controller is communicatively connected to the body domain controller. The method includes:

[0009] The vehicle's comprehensive status information is obtained, including at least one of the following: steering wheel angle, driver's head posture, and the posture of the passenger in the front passenger seat.

[0010] Based on the comprehensive state information, the target area that the driver's eyes need to observe and is in a blocked state is predicted. The target area includes at least one of the following: the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and the scene area corresponding to the image displayed by the right rearview mirror of the vehicle.

[0011] The system controls the left screen on the left A-pillar and / or the right screen on the right A-pillar of the vehicle to display images according to the target area. The left screen is a screen set on the left A-pillar, and the right screen is a screen set on the right A-pillar.

[0012] In most common car models on the market today, there is an A-pillar connecting the front engine compartment and the roof, a B-pillar located between the front and rear doors, and a C-pillar connecting the trunk and the roof. The A-pillar, B-pillar, and C-pillar all obstruct part of the driver's view while driving. However, since most vehicles are equipped with rearview mirrors and reversing cameras, the B-pillar and C-pillar generally do not pose a safety hazard to the driver while driving. But because the A-pillar is located at the front of the vehicle, it may pose a certain safety hazard to the driver while driving.

[0013] The vehicle is generally the subject to which the method provided in this application embodiment is applied. Specifically, the method is applied to the autonomous driving domain controller of the vehicle's domain controller system. The autonomous driving domain controller is the subject to which the method is executed. Since the domain controller system includes the autonomous driving domain controller and the body domain controller, the body domain controller is generally used to control the devices in the vehicle. The devices in the vehicle include external devices. The two controllers work together to enable the driver of the vehicle to observe the obscured area in a timely manner.

[0014] It should be emphasized in advance that this method determines whether the image is displayed on the screen of the left A-pillar, the right A-pillar, or both A-pillars based on the driver's line of sight. Therefore, if the driver's line of sight is not obstructed while driving the vehicle, the screen is always off.

[0015] The method applied in the first aspect above first obtains the comprehensive status information of the vehicle. The comprehensive status information may be obtained through the vehicle body domain controller, but the main body that processes the obtained information is the autonomous driving domain controller.

[0016] Secondly, the comprehensive state information can be used to predict the target area that the driver's eyes need to observe and that is obstructed. The target area is divided into three cases: Case 1, the left A-pillar of the vehicle obstructs the driver's field of vision, then the target area is the area obstructed by the left A-pillar of the vehicle and the farthest distance between the boundary of the area and the vehicle is less than a preset distance; Case 2, the right A-pillar of the vehicle obstructs the driver's field of vision, then the target area is the area obstructed by the right A-pillar of the vehicle and the farthest distance between the boundary of the area and the vehicle is less than a preset distance; Case 3, the passenger in the front passenger seat obstructs the driver's field of vision to observe the right rearview mirror, then the target area is the scene area corresponding to the image displayed in the right rearview mirror of the vehicle.

[0017] Based on the above three situations, the key to this method is how to determine the occurrence of the above three situations and what images to project onto the screen accordingly so that the driver can drive safely without being bothered by obstructed vision.

[0018] Specifically, the autonomous driving domain controller makes predictions and judgments based on the comprehensive state information acquired from the vehicle domain controller. This means that the acquired comprehensive state information can be used to determine the occurrence of the three situations mentioned above. Therefore, the autonomous driving domain controller analyzes the comprehensive state information to determine what is obstructing the driver's view.

[0019] Once the obstruction to the driver's view is determined, the target area can be identified. This target area is closely related to the image displayed on the screen. For example, if the target area is determined to be the region obstructed by the left A-pillar of the vehicle, and the furthest distance between the region's boundary and the vehicle is less than a preset distance, the vehicle domain controller acquires an image of the region obstructed by the left A-pillar, processes the image, and displays it on the screen on the left A-pillar. In this method, the autonomous driving domain controller is the entity that processes the image.

[0020] It is evident that by cooperating with the autonomous driving domain controller and the vehicle domain controller, the autonomous driving domain controller becomes more intelligent. It can identify the areas that the driver needs to observe but are obstructed at the appropriate time, and display the corresponding area on the control screen. At other times, the control screen remains off, which not only ensures that the driver's line of sight is not obstructed, but also ensures energy conservation and environmental protection.

[0021] In another possible implementation of the first aspect, the comprehensive state information of the vehicle includes the driver's head posture and the posture of the passenger in the front seat; predicting the target area that the driver's eyes need to observe and that is in an occluded state based on the comprehensive state information includes:

[0022] Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right side rearview mirror of the vehicle;

[0023] Based on the driver's head posture and the posture of the passenger in the front seat, determine whether the driver's line of sight to the right rearview mirror of the vehicle is blocked by the passenger in the front seat.

[0024] If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

[0025] Specifically, the screen can display not only the view obscured by the A-pillar, but also other views. Since drivers frequently need to check their side mirrors to observe oncoming traffic, passengers in the front passenger seat may obstruct the driver's view of the passenger-side mirror due to their actions. Even if the obstruction is brief, the high speed of the vehicle, especially on highways, means that even a short period of misoperation can endanger the driver or passenger. Therefore, in this embodiment, the other... The display includes the view shown in the rearview mirror on the passenger side. The screen on the right A-pillar is mounted on the passenger side A-pillar and faces the driver. When the driver's view of the passenger side rearview mirror is obstructed, the screen on the right A-pillar displays the side and rear view information on the passenger side, replacing the passenger side rearview mirror, so that the driver can observe the view that the rearview mirror should display. A second prompt message is output so that when the driver ignores the view displayed on the right A-pillar screen, he can promptly realize that he needs to observe the side or rear.

[0026] In scenario three of the above three situations, where the passenger in the front passenger seat obstructs the driver's view of the right-side rearview mirror, and the target area is the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle, the acquired comprehensive state information includes at least the driver's head posture and the posture of the passenger in the front passenger seat.

[0027] The system determines whether the driver's view of the vehicle's right-side rearview mirror is obstructed by the passenger based on the driver's head posture and the passenger's posture. If obstructed, it means the driver needs to observe, but the obstructed area is the scene area corresponding to the image displayed in the vehicle's right-side rearview mirror. The information analyzed by the autonomous driving domain controller varies depending on the situation.

[0028] In another possible implementation of the first aspect, if the target area is the scene area corresponding to the image displayed by the right rearview mirror of the vehicle, then the comprehensive state information of the vehicle includes the in-vehicle state information and the out-of-vehicle state information, and the in-vehicle state information includes the driver's head posture and the posture of the passenger in the front seat; the step of controlling the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area includes:

[0029] Acquire image data from at least one of the vehicle's multiple external cameras whose field of view includes the scene area corresponding to the image displayed by the right rearview mirror.

[0030] Based on the image data from the at least one external camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror, a target image adapted to the right screen of the right A-pillar is generated.

[0031] The target image is displayed on the right-side screen of the right-side A-pillar.

[0032] It should be noted that, in the above process, generating a target image that fits the screen of the right A-pillar requires the autonomous driving domain controller to adjust parameters such as frame rate and field of view of the image data acquired by the external camera in order to generate a target image that fits the right screen of the right A-pillar.

[0033] Optionally, during the process of generating the target image of the right screen adapted to the right A-pillar, a standard that conforms to the driver's observation habits is added to generate an image that matches the driver's observation habits. The standard can be obtained through a model that conforms to the current driver's habits, or it can be set automatically during generation.

[0034] Compared to existing solutions that use a single, fixed display with a fixed field of view, this setup improves the flexibility, accuracy, and comprehensiveness of image display by the autonomous driving domain controller.

[0035] In another possible implementation of the first aspect, generating a target image adapted to the right screen of the right A-pillar based on image data from the at least one exterior camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror includes:

[0036] If the at least one exterior camera is a single camera, a target image adapted to the right screen of the right A-pillar is generated based on the image data acquired by the exterior camera, including the image data displayed by the right rearview mirror, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0037] If there are multiple external cameras, then target image data corresponding to the image displayed by the right rearview mirror is generated by fusing the multiple image data acquired by the external cameras; and a target image adapted to the right screen of the right A-pillar is generated based on the target image data, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0038] Considering the differences in the devices controlled by the vehicle domain controllers of different vehicle models, and in order to adapt the target image to the screen and the driver's viewing habits, the method of generating the target image varies depending on the number of external cameras, further improving the comprehensiveness of the image display by the autonomous driving domain controller.

[0039] In another possible implementation of the first aspect, the integrated state includes at least one of a steering wheel angle or a driver's head posture, the driver's head posture including the direction the driver's head is facing; predicting the target area that the driver's eyes need to observe and that is in an obstructed state based on the integrated state information includes:

[0040] Based on the comprehensive status information, the driver's pre-observation direction is determined;

[0041] If the driver's expected observation direction is the left side of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle, whose viewing area includes the scene area blocked by the left A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0042] If the driver's pre-observation direction is the right side of the vehicle and the driver's head is facing the right A-pillar of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle whose viewing area includes the scene area blocked by the right A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0043] The above process is key to demonstrating the intelligence of the autonomous driving domain controller. For the sake of driver comfort, the screens located on the left and right A-pillars are generally always off. However, after the target area is determined, the corresponding image will be displayed on the screen. But to determine the target area, the driver's pre-observation direction needs to be determined. The driver's pre-observation direction is determined based on the steering wheel angle and / or the driver's head posture. For example, if the driver's steering wheel angle changes, it means that the vehicle's trajectory will change. At this time, the driver will naturally observe, which means that the steering wheel angle can represent the driver's observation direction.

[0044] In another possible implementation of the first aspect, if the target area is an area obscured by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance, or an area obscured by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance, then the comprehensive state information of the vehicle includes the in-vehicle state information and the out-of-vehicle state information, and the in-vehicle state information includes the driver's head posture and the steering wheel angle; the step of controlling the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area includes:

[0045] The driving status of the vehicle is determined based on the comprehensive status information of the vehicle, and the driving status includes at least going straight or turning;

[0046] The status information of the target object is determined based on the external status information, which is image data of the outside of the vehicle obtained by at least one external camera. The target object includes pedestrians and / or other vehicles. The status information of the target object includes the distance between the target object and the vehicle.

[0047] If the target object is a pedestrian, when the vehicle is traveling straight and the distance between the pedestrian and the vehicle is less than or equal to a first distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area.

[0048] If the target object is a pedestrian, when the vehicle is turning and the distance between the pedestrian and the vehicle is less than or equal to a second distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output, wherein the second distance is less than the first distance;

[0049] If the target object is another vehicle, when the vehicle is traveling straight and the distance between the other vehicle and the vehicle is less than or equal to the third distance, the screen on the same side as the other vehicle displays the image of the target area corresponding to the other vehicle and outputs the first prompt information.

[0050] If the target object is another vehicle, when the vehicle is turning and the distance between the other vehicle and the vehicle is less than or equal to a fourth distance, the target area corresponding to the other vehicle is displayed on the screen on the same side as the other vehicle, and a first prompt message is output, wherein the fourth distance is less than the third distance.

[0051] Furthermore, since the time given to the driver to observe, think, and operate differs depending on whether the car is going straight or turning, the safe distance between the car and the target object will also be different. Therefore, the driving state of the vehicle must first be determined. The driving state includes going straight or turning. In other embodiments, the driving state may also include other situations, such as turning around or braking.

[0052] Furthermore, if the target object approaching the vehicle is a pedestrian, different safety distances will be set depending on the vehicle's driving status. When the vehicle is traveling straight, its speed is generally higher, and when turning, its speed is generally lower. Therefore, to give the driver more time to control the vehicle and avoid danger, the second distance is set smaller than the first distance; similarly, the fourth distance is smaller than the third distance. In summary, setting different safety distances to cope with complex situations during vehicle driving is more reasonable and humane.

[0053] In another possible implementation of the first aspect, before determining the state information of the target object based on the vehicle exterior state information, the method further includes:

[0054] Obtain a vehicle dataset, which includes the vehicle's historical driving status and first speed information, wherein the driving status includes at least going straight or turning;

[0055] The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information.

[0056] The target object dataset and the vehicle dataset are input into the safe distance prediction model to obtain the safe distance between the target object and the vehicle. The safe distance includes the first distance, the second distance, the third distance, or the fourth distance. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data.

[0057] Specifically, the aforementioned safety distance can be dynamic, determined based on the actual conditions during vehicle operation. In this embodiment, the safety distance includes the first distance, the second distance, the third distance, and the fourth distance. Therefore, before determining the vehicle's driving state, the safety distance for each target object is determined for subsequent operations.

[0058] Furthermore, the safe distance is related to the actual driving / walking state of the vehicle and the target object. Therefore, firstly, relevant data of the vehicle is acquired, i.e., a vehicle dataset is acquired. The vehicle dataset includes the vehicle's historical driving state and first speed information, used to characterize the vehicle's operating state from a certain point in time to the present. The driving state includes at least going straight or turning. Secondly, the target object dataset is generated based on the aforementioned video information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information, used to characterize the pedestrian's movement state at the current point in time or time period. That is, the pedestrian morphology information includes height and orientation. The vehicle dataset includes: vehicle driving state information, vehicle trajectory information, and third speed information, used to characterize the operating state of the other vehicles at the current point in time or time period.

[0059] Furthermore, the target object dataset and the vehicle dataset are input into a safe distance prediction model. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data. The model and logic applied by the safe distance prediction model are diverse. For example, by integrating the relevant data of the vehicles and the target objects into driving trajectories, fitting the driving trajectories of the vehicles and the target objects to obtain collision judgment results, and determining appropriate safe distances based on the collision judgment results and the relevant data of the target objects.

[0060] Optionally, the pedestrian dataset further includes: a first weight and a second weight; the first weight is used to constrain the influence of the pedestrian morphology information on the corresponding safe distance result; the second weight is used to constrain the influence of the first speed information on the corresponding safe distance result.

[0061] Optionally, the vehicle dataset further includes: a third weight, a fourth weight, and a fifth weight; the third weight is used to constrain the influence of the vehicle driving status information on the corresponding safe distance result; the fourth weight is used to constrain the influence of the vehicle trajectory information on the corresponding safe distance result; and the fifth weight is used to constrain the influence of the third speed information on the corresponding safe distance result.

[0062] By using the above methods, the safety distance can be adjusted more flexibly, making it more effective in dealing with more complex road conditions.

[0063] In another possible implementation of the first aspect, after determining the state information of the target object based on the vehicle exterior state information, the method further includes:

[0064] If the target objects include pedestrians and other vehicles, when the vehicle is traveling straight, and the distance between the pedestrian and any of the other vehicles and the vehicle is less than or equal to the larger of the first distance and the third distance, the target image of the target area corresponding to the pedestrian is displayed on the screen, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area.

[0065] If the target objects include pedestrians and other vehicles, when the vehicle is turning, and the distance between the pedestrian and any of the other vehicles and the vehicle is less than or equal to the larger of the second distance and the fourth distance, the target image of the target area corresponding to the pedestrian is displayed on the screen, and the first prompt information is output.

[0066] Specifically, on some roads where vehicles and pedestrians are not separated or on narrow roads, it is easy to encounter situations where the target object includes both pedestrians and other vehicles. In such complex situations, frequent prompts can be distracting to the driver. Therefore, when the vehicle is traveling straight, and the distance between the vehicle and either the pedestrian or other vehicle is less than or equal to the larger of the first distance and the third distance, the screen on the A-pillar near the driver's seat displays the area obscured by the A-pillar and outputs a first prompt message. When the vehicle is turning, and the distance between the vehicle and either the pedestrian or other vehicle is less than or equal to the larger of the second distance and the fourth distance, the screen on the A-pillar near the driver's seat displays the area obscured by the A-pillar and outputs the first prompt message. Optionally, if any of the above situations occur again during the output of the first prompt message, and a prompt message needs to be output again, the prompt message can be omitted, and only the display of the area obscured by the A-pillar on the screen can be extended to avoid frequent prompts interfering with the driver and potentially causing an accident.

[0067] In another possible implementation of the first aspect, the values ​​of the third distance and the fourth distance are determined based on the type of the other vehicles, which include at least bicycles, electric bicycles, cars, trucks, buses, trailers, incomplete vehicles, motorcycles, tractor-trailers, or special vehicles, including ambulances, police cars, or fire trucks.

[0068] Because there are many types of vehicles on the road, and the types of vehicles commonly seen vary from city to city and road to road, some cities have more electric bicycles, some have more cars, and some have more trucks, the third and fourth distances related to other vehicles in the target group will vary depending on the type of vehicle. It is worth noting that the safe distance between other vehicles that are special vehicles is the largest. Optionally, when other vehicles are identified as special vehicles, special prompts can be output to the driver, which not only complies with traffic regulations but also reflects humanistic considerations.

[0069] In another possible implementation of the first aspect, the screen includes a screen on the right A-pillar located on the passenger side, with the screen on the right A-pillar facing the driver's seat. When the distance between the target object and the vehicle is less than or equal to a preset distance, the screen on the side of the A-pillar closest to the driver's seat displays an image of the area obscured by the A-pillar and outputs a first prompt message. The method further includes:

[0070] If it is detected that the driver's view of the passenger-side rearview mirror is obstructed, the screen on the right A-pillar is controlled to display the side and rear view information on the passenger side, and a second prompt message is output. The second prompt message is used to instruct the driver to observe the road conditions on the side and rear of the passenger side of the vehicle.

[0071] In another possible implementation of the first aspect, after displaying an image of the area obscured by the A-pillar on the screen and outputting a first prompt message when the distance between the target object and the vehicle is less than or equal to a preset distance, the method further includes:

[0072] A third prompt message is output, which is used to indicate the distance value between the vehicle and the target object.

[0073] Specifically, the third prompt information can be displayed on the screen or output via voice broadcast, so that the driver can take appropriate actions based on the actual distance.

[0074] In another possible implementation of the first aspect, the vehicle includes a dashcam positioned to record the view directly in front of the vehicle. After displaying the view of the area obscured by the A-pillar on the screen of the A-pillar and outputting a first prompt message when the distance between the target object and the vehicle is less than or equal to a preset distance, the method further includes:

[0075] Adjust the shooting direction of the dashcam so that it captures images related to the target object.

[0076] Generally, dashcams can only capture the view directly in front of the vehicle. In real-world scenarios, accidents often occur because drivers cannot observe the A-pillar blind spot. However, sometimes dashcams fail to record the actual situation of such accidents due to their angle. Therefore, to address situations where there is no video recording in the event of such a safety accident, the dashcam adjusts its shooting direction when the distance between the target object and the vehicle is less than or equal to a preset distance. This allows the dashcam to capture images related to the target object. Optionally, the angle of adjustment is 10°, ensuring that the dashcam captures both images related to the target object and most of the view directly in front of the vehicle.

[0077] In one optional implementation, the shooting direction of the dashcam is adjusted according to the collision prediction result between the target object and the vehicle, so that the dashcam can capture images related to the target object.

[0078] Specifically, a vehicle dataset is obtained, which includes the vehicle's historical driving status and first speed information, and the driving status includes at least going straight or turning;

[0079] The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information.

[0080] The target object dataset and the vehicle dataset are input into the collision prediction model to obtain the collision prediction results between the target object and the vehicle. The collision prediction results include complete collision, possible collision, or impossible collision. A complete collision means that the vehicle and the target object will definitely collide. A possible collision means that the vehicle and the target object will collide if the driver does not take appropriate action. A no-collision means that the vehicle and the target object will definitely not collide if the driver does not take any action to change its driving state and speed. The collision prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding collision prediction results. The target object dataset and the vehicle dataset are feature data, and the collision prediction results are label data.

[0081] If the collision prediction result is a complete collision or a possible collision, the shooting direction of the dashcam is adjusted so that the dashcam can capture images related to the target object.

[0082] In this way, the dashcam can make adaptive adjustments according to the methods provided in one or more implementations, thereby improving the driver's experience.

[0083] Secondly, embodiments of this application provide an image display device based on the driver's field of vision in a driving scenario. The device includes at least an acquisition unit, a prediction unit, and a display unit. This image display device based on the driver's field of vision in a driving scenario is used to implement the method described in any embodiment of the first aspect, wherein the acquisition unit, prediction unit, and display unit are described below:

[0084] The acquisition unit is used to acquire the comprehensive status information of the vehicle, which includes at least one of the following: steering wheel angle, driver's head posture, and the posture of the passenger in the front passenger seat.

[0085] The prediction unit is used to predict, based on the comprehensive state information, the target area that the driver's eyes need to observe and that is in an obstructed state. The target area includes at least one of the following: the area obstructed by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; the area obstructed by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and the scene area corresponding to the image displayed by the right rearview mirror of the vehicle.

[0086] The display unit is used to control the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area, wherein the left screen is a screen set on the left A-pillar and the right screen is a screen set on the right A-pillar.

[0087] In most common car models on the market today, there is an A-pillar connecting the front engine compartment and the roof, a B-pillar located between the front and rear doors, and a C-pillar connecting the trunk and the roof. The A-pillar, B-pillar, and C-pillar all obstruct part of the driver's view while driving. However, since most vehicles are equipped with rearview mirrors and reversing cameras, the B-pillar and C-pillar generally do not pose a safety hazard to the driver while driving. But because the A-pillar is located at the front of the vehicle, it may pose a certain safety hazard to the driver while driving.

[0088] The vehicle is generally the subject to which the method provided in this application embodiment is applied. Specifically, the method is applied to the autonomous driving domain controller of the vehicle's domain controller system. The autonomous driving domain controller is the subject to which the method is executed. Since the domain controller system includes the autonomous driving domain controller and the body domain controller, the body domain controller is generally used to control the devices in the vehicle. The devices in the vehicle include external devices. The two controllers work together to enable the driver of the vehicle to observe the obscured area in a timely manner.

[0089] It should be emphasized in advance that this method determines whether the image is displayed on the screen of the left A-pillar, the right A-pillar, or both A-pillars based on the driver's line of sight. Therefore, if the driver's line of sight is not obstructed while driving the vehicle, the screen is always off.

[0090] The method applied in the first aspect above first obtains the comprehensive status information of the vehicle. The comprehensive status information may be obtained through the vehicle body domain controller, but the main body that processes the obtained information is the autonomous driving domain controller.

[0091] Secondly, the comprehensive state information can be used to predict the target area that the driver's eyes need to observe and that is obstructed. The target area is divided into three cases: Case 1, the left A-pillar of the vehicle obstructs the driver's field of vision, then the target area is the area obstructed by the left A-pillar of the vehicle and the farthest distance between the boundary of the area and the vehicle is less than a preset distance; Case 2, the right A-pillar of the vehicle obstructs the driver's field of vision, then the target area is the area obstructed by the right A-pillar of the vehicle and the farthest distance between the boundary of the area and the vehicle is less than a preset distance; Case 3, the passenger in the front passenger seat obstructs the driver's field of vision to observe the right rearview mirror, then the target area is the scene area corresponding to the image displayed in the right rearview mirror of the vehicle.

[0092] Based on the above three situations, the key to this method is how to determine the occurrence of the above three situations and what images to project onto the screen accordingly so that the driver can drive safely without being bothered by obstructed vision.

[0093] Specifically, the autonomous driving domain controller makes predictions and judgments based on the comprehensive state information acquired from the vehicle domain controller. This means that the acquired comprehensive state information can be used to determine the occurrence of the three situations mentioned above. Therefore, the autonomous driving domain controller analyzes the comprehensive state information to determine what is obstructing the driver's view.

[0094] Once the obstruction to the driver's view is determined, the target area can be identified. This target area is closely related to the image displayed on the screen. For example, if the target area is determined to be the region obstructed by the left A-pillar of the vehicle, and the furthest distance between the region's boundary and the vehicle is less than a preset distance, the vehicle domain controller acquires an image of the region obstructed by the left A-pillar, processes the image, and displays it on the screen on the left A-pillar. In this method, the autonomous driving domain controller is the entity that processes the image.

[0095] It is evident that by cooperating with the autonomous driving domain controller and the vehicle domain controller, the autonomous driving domain controller becomes more intelligent. It can identify the areas that the driver needs to observe but are obstructed at the appropriate time, and display the corresponding area on the control screen. At other times, the control screen remains off, which not only ensures that the driver's line of sight is not obstructed, but also ensures energy conservation and environmental protection.

[0096] In another possible implementation of the second aspect, the prediction unit is specifically used for:

[0097] Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right side rearview mirror of the vehicle;

[0098] Based on the driver's head posture and the posture of the passenger in the front seat, determine whether the driver's line of sight to the right rearview mirror of the vehicle is blocked by the passenger in the front seat.

[0099] If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

[0100] Specifically, the screen can display not only the view obscured by the A-pillar, but also other views. Since drivers frequently need to check their side mirrors to observe oncoming traffic, passengers in the front passenger seat may obstruct the driver's view of the passenger-side mirror due to their actions. Even if the obstruction is brief, the high speed of the vehicle, especially on highways, means that even a short period of misoperation can endanger the driver or passenger. Therefore, in this embodiment, the other... The display includes the view shown in the rearview mirror on the passenger side. The screen on the right A-pillar is mounted on the passenger side A-pillar and faces the driver. When the driver's view of the passenger side rearview mirror is obstructed, the screen on the right A-pillar displays the side and rear view information on the passenger side, replacing the passenger side rearview mirror, so that the driver can observe the view that the rearview mirror should display. A second prompt message is output so that when the driver ignores the view displayed on the right A-pillar screen, he can promptly realize that he needs to observe the side or rear.

[0101] In scenario three of the above three situations, where the passenger in the front passenger seat obstructs the driver's view of the right-side rearview mirror, and the target area is the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle, the acquired comprehensive state information includes at least the driver's head posture and the posture of the passenger in the front passenger seat.

[0102] The system determines whether the driver's view of the vehicle's right-side rearview mirror is obstructed by the passenger based on the driver's head posture and the passenger's posture. If obstructed, it means the driver needs to observe, but the obstructed area is the scene area corresponding to the image displayed in the vehicle's right-side rearview mirror. The information analyzed by the autonomous driving domain controller varies depending on the situation.

[0103] In another possible implementation of the second aspect, the display unit is specifically used for:

[0104] Acquire image data from at least one of the vehicle's multiple external cameras whose field of view includes the scene area corresponding to the image displayed by the right rearview mirror.

[0105] Based on the image data from the at least one external camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror, a target image adapted to the right screen of the right A-pillar is generated.

[0106] The target image is displayed on the right-side screen of the right-side A-pillar.

[0107] It should be noted that, in the above process, generating a target image that fits the screen of the right A-pillar requires the autonomous driving domain controller to adjust parameters such as frame rate and field of view of the image data acquired by the external camera in order to generate a target image that fits the right screen of the right A-pillar.

[0108] Optionally, during the process of generating the target image of the right screen adapted to the right A-pillar, a standard that conforms to the driver's observation habits is added to generate an image that matches the driver's observation habits. The standard can be obtained through a model that conforms to the current driver's habits, or it can be set automatically during generation.

[0109] Compared to existing solutions that use a single, fixed display with a fixed field of view, this setup improves the flexibility, accuracy, and comprehensiveness of image display by the autonomous driving domain controller.

[0110] In another possible implementation of the second aspect, in generating a target image adapted to the right screen of the right A-pillar based on image data from the at least one exterior camera, the size of the right-side screen of the right A-pillar, and the imaging characteristics of the right-side rearview mirror, the display unit is further configured to:

[0111] If the at least one exterior camera is a single camera, a target image adapted to the right screen of the right A-pillar is generated based on the image data acquired by the exterior camera, including the image data displayed by the right rearview mirror, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0112] If there are multiple external cameras, then target image data corresponding to the image displayed by the right rearview mirror is generated by fusing the multiple image data acquired by the external cameras; and a target image adapted to the right screen of the right A-pillar is generated based on the target image data, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0113] Considering the differences in the devices controlled by the vehicle domain controllers of different vehicle models, and in order to adapt the target image to the screen and the driver's viewing habits, the method of generating the target image varies depending on the number of external cameras, further improving the comprehensiveness of the image display by the autonomous driving domain controller.

[0114] In another possible implementation of the second aspect, the prediction unit is specifically used for:

[0115] Based on the comprehensive status information, the driver's pre-observation direction is determined;

[0116] If the driver's expected observation direction is the left side of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle, whose viewing area includes the scene area blocked by the left A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0117] If the driver's pre-observation direction is the right side of the vehicle and the driver's head is facing the right A-pillar of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle whose viewing area includes the scene area blocked by the right A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0118] The above process is key to demonstrating the intelligence of the autonomous driving domain controller. For the sake of driver comfort, the screens located on the left and right A-pillars are generally always off. However, after the target area is determined, the corresponding image will be displayed on the screen. But to determine the target area, the driver's pre-observation direction needs to be determined. The driver's pre-observation direction is determined based on the steering wheel angle and / or the driver's head posture. For example, if the driver's steering wheel angle changes, it means that the vehicle's trajectory will change. At this time, the driver will naturally observe, which means that the steering wheel angle can represent the driver's observation direction.

[0119] In another possible implementation of the second aspect, the display unit is specifically used for:

[0120] The driving status of the vehicle is determined based on the comprehensive status information of the vehicle, and the driving status includes at least going straight or turning;

[0121] The status information of the target object is determined based on the external status information, which is image data of the outside of the vehicle obtained by at least one external camera. The target object includes pedestrians and / or other vehicles. The status information of the target object includes the distance between the target object and the vehicle.

[0122] If the target object is a pedestrian, when the vehicle is traveling straight and the distance between the pedestrian and the vehicle is less than or equal to a first distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area.

[0123] If the target object is a pedestrian, when the vehicle is turning and the distance between the pedestrian and the vehicle is less than or equal to a second distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output, wherein the second distance is less than the first distance;

[0124] If the target object is another vehicle, when the vehicle is traveling straight and the distance between the other vehicle and the vehicle is less than or equal to the third distance, the screen on the same side as the other vehicle displays the image of the target area corresponding to the other vehicle and outputs the first prompt information.

[0125] If the target object is another vehicle, when the vehicle is turning and the distance between the other vehicle and the vehicle is less than or equal to a fourth distance, the target area corresponding to the other vehicle is displayed on the screen on the same side as the other vehicle, and a first prompt message is output, wherein the fourth distance is less than the third distance.

[0126] Furthermore, since the time given to the driver to observe, think, and operate differs depending on whether the car is going straight or turning, the safe distance between the car and the target object will also be different. Therefore, the driving state of the vehicle must first be determined. The driving state includes going straight or turning. In other embodiments, the driving state may also include other situations, such as turning around or braking.

[0127] Furthermore, if the target object approaching the vehicle is a pedestrian, different safety distances will be set depending on the vehicle's driving status. When the vehicle is traveling straight, its speed is generally higher, and when turning, its speed is generally lower. Therefore, to give the driver more time to control the vehicle and avoid danger, the second distance is set smaller than the first distance; similarly, the fourth distance is smaller than the third distance. In summary, setting different safety distances to cope with complex situations during vehicle driving is more reasonable and humane.

[0128] In yet another possible implementation of the second aspect, the display unit is further configured to:

[0129] Obtain a vehicle dataset, which includes the vehicle's historical driving status and first speed information, wherein the driving status includes at least going straight or turning;

[0130] The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information.

[0131] The target object dataset and the vehicle dataset are input into the safe distance prediction model to obtain the safe distance between the target object and the vehicle. The safe distance includes the first distance, the second distance, the third distance, or the fourth distance. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data.

[0132] Specifically, the aforementioned safety distance can be dynamic, determined based on the actual conditions during vehicle operation. In this embodiment, the safety distance includes the first distance, the second distance, the third distance, and the fourth distance. Therefore, before determining the vehicle's driving state, the safety distance for each target object is determined for subsequent operations.

[0133] Furthermore, the safe distance is related to the actual driving / walking state of the vehicle and the target object. Therefore, firstly, relevant data of the vehicle is acquired, i.e., a vehicle dataset is acquired. The vehicle dataset includes the vehicle's historical driving state and first speed information, used to characterize the vehicle's operating state from a certain point in time to the present. The driving state includes at least going straight or turning. Secondly, the target object dataset is generated based on the aforementioned video information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information, used to characterize the pedestrian's movement state at the current point in time or time period. That is, the pedestrian morphology information includes height and orientation. The vehicle dataset includes: vehicle driving state information, vehicle trajectory information, and third speed information, used to characterize the operating state of the other vehicles at the current point in time or time period.

[0134] Furthermore, the target object dataset and the vehicle dataset are input into a safe distance prediction model. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data. The model and logic applied by the safe distance prediction model are diverse. For example, by integrating the relevant data of the vehicles and the target objects into driving trajectories, fitting the driving trajectories of the vehicles and the target objects to obtain collision judgment results, and determining appropriate safe distances based on the collision judgment results and the relevant data of the target objects.

[0135] Optionally, the pedestrian dataset further includes: a first weight and a second weight; the first weight is used to constrain the influence of the pedestrian morphology information on the corresponding safe distance result; the second weight is used to constrain the influence of the first speed information on the corresponding safe distance result.

[0136] Optionally, the vehicle dataset further includes: a third weight, a fourth weight, and a fifth weight; the third weight is used to constrain the influence of the vehicle driving status information on the corresponding safe distance result; the fourth weight is used to constrain the influence of the vehicle trajectory information on the corresponding safe distance result; and the fifth weight is used to constrain the influence of the third speed information on the corresponding safe distance result.

[0137] By using the above methods, the safety distance can be adjusted more flexibly, making it more effective in dealing with more complex road conditions.

[0138] Thirdly, embodiments of this application provide a vehicle that includes an autonomous driving domain controller, a memory, and a computer program stored in the memory and capable of running on the autonomous driving domain controller; when the autonomous driving domain controller executes the computer program, the vehicle can execute the method described in the first aspect or any possible implementation of the first aspect.

[0139] It should be noted that the autonomous driving domain controller included in the vehicle described in the third aspect above can be an autonomous driving domain controller specifically designed to execute these methods (referred to as a dedicated autonomous driving domain controller for distinction), or an autonomous driving domain controller that executes these methods by invoking a computer program, such as a general-purpose autonomous driving domain controller. Optionally, at least one autonomous driving domain controller may include both dedicated and general-purpose autonomous driving domain controllers.

[0140] Optionally, the aforementioned computer program can be stored in memory. For example, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the autonomous driving domain controller on the same device or disposed on different devices. This application does not limit the type of memory or the way the memory and the autonomous driving domain controller are disposed.

[0141] In one possible implementation, at least one of the aforementioned memory devices is located outside the aforementioned vehicle.

[0142] In yet another possible implementation, at least one of the aforementioned memory devices is located within the aforementioned vehicle.

[0143] In another possible implementation, a portion of the memory of the at least one memory is located inside the vehicle, while another portion of the memory is located outside the vehicle.

[0144] In this application, the autonomous driving domain controller and the memory may also be integrated into a single device, that is, the autonomous driving domain controller and the memory can be integrated together.

[0145] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when the instructions are executed on at least one autonomous driving domain controller, implements the method described in the first aspect or any of the optional solutions of the first aspect.

[0146] Fifthly, this application provides a computer program product comprising a computer program that, when run on at least one autonomous driving domain controller, implements the method described in the first aspect or any of the optional solutions of the first aspect.

[0147] Optionally, the computer program product can be a software installation package, which can be downloaded and executed on a computing device when the aforementioned method is required.

[0148] The beneficial effects of the technical solutions provided in the third to fifth aspects of this application can be referred to the beneficial effects of the technical solutions in the first and second aspects, and will not be repeated here. Attached Figure Description

[0149] The accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0150] Figure 1 This is a schematic diagram of a vehicle scenario provided in an embodiment of this application;

[0151] Figure 2 This is a schematic diagram of the structure of a vehicle domain controller system provided in an embodiment of this application;

[0152] Figure 3 This is a flowchart illustrating an image display method based on the driver's field of vision in a driving scenario, as provided in an embodiment of this application.

[0153] Figure 4 This is a schematic diagram of the structure of an image display device based on the driver's field of vision in a driving scenario, provided in an embodiment of this application;

[0154] Figure 5 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application. Detailed Implementation

[0155] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0156] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0157] The system architecture used in the embodiments of this application is described below. It should be noted that the system architecture and business scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0158] Please see Figure 1 , Figure 1 This is a schematic diagram of a vehicle scenario provided in an embodiment of this application, wherein:

[0159] Vehicle 10 can be an on-board device or the vehicle itself, such as a car, truck, bus, trailer, or incomplete vehicle. Vehicle 10 may be equipped with image acquisition devices such as cameras or industrial cameras. Optionally, the image acquisition devices are mounted on the A-pillars on both sides of vehicle 10, facing outwards. Vehicle 10 acquires image information of areas obscured by the A-pillars through these devices. Accordingly, vehicle 10 includes at least a left-side screen 101 on the left A-pillar and a right-side screen 102 on the right A-pillar. Vehicle 10 displays the image of the area obscured by the A-pillars through the left-side screen 101 and the right-side screen 102 on the right A-pillar. The left-side screen 101 is located on the A-pillar on the driver's side, and the right-side screen 102 is located on the A-pillar on the passenger side. Optionally, the right-side screen 102 on the right A-pillar can display images from other areas. The image acquisition device can also be mounted on the rearview mirror, allowing the right-side screen 102 on the right A-pillar to display images from the side and rear of the vehicle 10, i.e., the corresponding images displayed in the right rearview mirror. The vehicle 10 is equipped with information capture devices such as vehicle radar, infrared sensors, speed sensors, and distance sensors. Through these information capture devices, the vehicle 10 can obtain relevant information about the target object / vehicle 10 itself, including pedestrians or other vehicles. Optionally, the vehicle 10 is equipped with an audio device, which can be installed inside the left-side screen 101 on the left A-pillar and the right-side screen 102 on the right A-pillar, or it can be installed outside the left-side screen 101 on the left A-pillar and the right-side screen 102 on the right A-pillar, but inside the vehicle 10, and used to play prompts in certain scenarios.

[0160] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a vehicle domain controller system provided in an embodiment of this application. The vehicle domain controller system includes an autonomous driving domain controller 201 and a body domain controller 202. The execution subject of the question-and-answer recall method provided in this embodiment of the application is the autonomous driving domain controller 201, which has information processing capabilities. The autonomous driving domain controller 201 establishes a communication connection with the body domain controller 202. The body domain controller 202 is used to control devices in the vehicle, such as in-vehicle cameras, external cameras, or screens.

[0161] The autonomous driving domain controller 201 includes a first interaction layer 2011, a processing layer 2012, and a second interaction layer 2013. The first interaction layer 2011 is mainly used for information interaction with the vehicle domain controller 202, such as sending and receiving necessary information requests, including comprehensive vehicle status information and image data acquired by cameras. The processing layer 2012 is mainly used for processing the comprehensive status information and image data; for example, the processing layer 2012 predicts the target area by processing the comprehensive status information, and obtains the target image by processing the image data and other data. The second interaction layer 2013 is mainly used for sending screen control requests to the vehicle domain controller 202 to control the screen to display the target image.

[0162] The aforementioned autonomous driving domain controller 201 and body domain controller 202 can be integrated into the vehicle or outside the vehicle, or one can be installed in the vehicle and the other outside the vehicle. This application embodiment does not limit this.

[0163] Please see Figure 3 , Figure 3 This is a flowchart illustrating an image display method based on the driver's field of vision in a driving scenario, provided in an embodiment of this application. It is applied to the autonomous driving domain controller of a vehicle's domain controller system. The domain controller system includes the autonomous driving domain controller and a body domain controller, which are communicatively connected. This method can be based on... Figure 2 The system architecture diagram shown can also be implemented based on other architectures. This method includes, but is not limited to, the following steps:

[0164] Step S301: Obtain comprehensive status information of the vehicle.

[0165] The comprehensive status information is obtained through the vehicle domain controller. For example, the autonomous driving domain controller sends a request to the vehicle domain controller to obtain the comprehensive status information, and then receives the comprehensive status information sent by the vehicle domain controller. The comprehensive status information is used to predict the target area that the driver's eyes need to observe and that is currently obscured.

[0166] The comprehensive status information includes at least one of the following: steering wheel angle, driver's head posture, and the posture of the passenger in the front passenger seat.

[0167] In one alternative implementation, while the vehicle is traveling normally on the road, the comprehensive status information includes only the steering wheel angle. The steering wheel angle is used to determine whether the vehicle is turning or making a U-turn, thereby predicting the area that the driver's eyes need to observe.

[0168] In another optional implementation, the integrated state information includes only the driver's head posture, which is used to determine the direction or area that the driver's eyes need to observe, and whether the area that the driver needs to observe is obstructed.

[0169] In another optional implementation, the integrated status information includes the driver's head posture and the posture of the passenger in the front seat. The driver's head posture and the posture of the passenger in the front seat are used to determine whether the passenger's view of the right-side rearview mirror is obstructed by the passenger in the front seat.

[0170] In another optional implementation, since the above situation is unpredictable, the acquired comprehensive state information includes the steering wheel angle, the driver's head posture, and the posture of the passenger in the front seat. First, the steering wheel angle is used to determine whether the vehicle's driving route has deviated. If it has deviated, the direction of the deviation is used to determine whether the area to be observed is obstructed. If it has not deviated, the driver's head posture is used to determine whether the driver is preparing to observe both sides of the vehicle. If the driver's head posture is found to be deviating, it is determined that the driver is preparing to observe both sides of the vehicle, and the direction of observation can be determined. If the driver's head posture is deviating to the right, the posture of the passenger in the front seat is used to determine whether the target area that the driver's eyes need to observe and is obstructed is the vehicle's right-side rearview mirror.

[0171] Step S302: Based on the comprehensive state information, predict the target area that the driver's eyes need to observe and that is currently obstructed.

[0172] The target area includes at least one of the following: the area obscured by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; the area obscured by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and the scene area corresponding to the image displayed by the right rearview mirror of the vehicle.

[0173] In this embodiment, the target area includes the following three cases:

[0174] Scenario 1: The target area is the scene area corresponding to the image displayed in the right rearview mirror of the vehicle.

[0175] In scenario one, the comprehensive status information includes at least the driver's head posture and the posture of the passenger in the front passenger seat.

[0176] Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right-side rearview mirror of the vehicle; the driver's head posture can be obtained from a camera inside the vehicle.

[0177] Based on the driver's head posture and the posture of the passenger in the front seat, it is determined whether the driver's line of sight to the right rearview mirror is obstructed by the passenger in the front seat; the posture of the passenger in the front seat can be obtained from a camera inside the vehicle; specifically, the driver's line of sight is determined based on the driver's head posture, and if the driver's line of sight is in the direction of observing the right rearview mirror, then it is determined whether the posture of the passenger in the front seat obstructs the driver's view of the right rearview mirror.

[0178] If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

[0179] In scenario one, the corresponding scene corresponding to the image displayed in the right-side rearview mirror of the obscured vehicle is displayed on the right-side screen.

[0180] Scenario 2: The target area is the area obscured by the left A-pillar of the vehicle, and the furthest distance between the boundary of the area and the vehicle is less than a preset distance.

[0181] The overall state includes at least one of the steering wheel angle or the driver's head posture, wherein the driver's head posture includes the direction the driver's head is facing.

[0182] Based on the comprehensive status information, the driver's pre-observation direction is determined. The pre-observation direction is the direction the driver is prepared to observe. For example, if the driver's head is turning to the right, then the driver's pre-observation direction is to the right.

[0183] If the driver's expected observation direction is the left side of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle, whose view area includes the view area blocked by the left A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0184] It should be noted that the preset distance is determined based on the image data. Optionally, the preset distance is determined based on other vehicles or pedestrians in the image data. Alternatively, the preset distance is determined based on the driving posture and / or speed of other vehicles or pedestrians in the image data.

[0185] Scenario 3: The target area is the area obscured by the right A-pillar of the vehicle, and the furthest distance between the area boundary and the vehicle is less than a preset distance.

[0186] If the driver's pre-observation direction is the right side of the vehicle and the driver's head is facing the right A-pillar of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle whose viewing area includes the scene area blocked by the right A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0187] In one optional implementation, if the driver's pre-observation direction is the left / right side of the vehicle, the target area is directly determined to be the area obscured by the left / right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance, where the preset distance is pre-set.

[0188] Step S303: Control the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area.

[0189] The left screen is a screen mounted on the left A-pillar, and the right screen is a screen mounted on the right A-pillar.

[0190] Before displaying the image, image data of the corresponding target area is acquired through the vehicle's external cameras. Taking the above situation as an example, in an optional implementation, image data of at least one of the vehicle's multiple external cameras is acquired, whose framing area includes the scene area corresponding to the image displayed by the right rearview mirror.

[0191] Based on the image data from at least one external camera, the size of the right-side screen on the right A-pillar, and the imaging characteristics of the right-side rearview mirror, a target image adapted to the right-side screen of the right A-pillar is generated. Optionally, if the right-side screen is vertically elongated, then, combining the imaging characteristics of the right-side rearview mirror's side length and width, an image based on the imaging characteristics of the right-side rearview mirror's side length and width is displayed on the right-side screen. In this case, there will be some black borders at the top and / or bottom of the right-side screen. To present the driver with a larger view of the right rear of the vehicle, optionally, a target image adapted to the vertically elongated shape of the right-side screen is directly generated based on the image data. In this case, there will be no black borders on the right-side screen. Optionally, a second prompt message is output, which instructs the driver of the vehicle to observe the road conditions on the side and rear of the passenger side of the vehicle.

[0192] If the at least one exterior camera is a single camera, a target image adapted to the right screen of the right A-pillar is generated based on the image data acquired by the exterior camera, including the image data displayed by the right rearview mirror, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror; optionally, the exterior camera is mounted on the right rearview mirror.

[0193] If there are multiple exterior cameras, then target image data corresponding to the image displayed on the right-side rearview mirror is generated by fusing the multiple image data acquired by the exterior cameras. Based on the target image data, the size of the right-side screen of the right A-pillar, and the imaging characteristics of the right-side rearview mirror, a target image adapted to the right-side screen of the right A-pillar is generated. Optionally, the exterior cameras include cameras mounted on the outside of the A-pillar, with the image acquisition direction of the camera mounted on the outside of the A-pillar aligned with the driver's line of sight when observing the A-pillar, so that the acquired image of the obstructed area can precisely fill the driver's blind spot.

[0194] In one optional implementation, if there are pedestrians or other vehicles in the area obscured by the left or right A-pillar of the vehicle, then even if the driver does not observe in that direction, the corresponding image will be displayed on the screen on the same side as the pedestrian or other vehicle, as described below:

[0195] The driving state of the vehicle is determined based on the comprehensive status information of the vehicle, and the driving state includes at least going straight or turning; then the comprehensive status information of the vehicle includes the in-vehicle status information and the out-of-vehicle status information, and the in-vehicle status information includes the driver's head posture and the steering wheel angle.

[0196] The status information of the target object is determined based on the external status information, which is image data of the outside of the vehicle obtained by at least one external camera. The target object includes pedestrians and / or other vehicles. The status information of the target object includes the distance between the target object and the vehicle.

[0197] If the target object is a pedestrian, when the vehicle is traveling straight and the distance between the pedestrian and the vehicle is less than or equal to a first distance, a target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area. In an optional embodiment, a third prompt message is output, which is used to indicate the distance value between the vehicle and the target object. In this embodiment, the real-time distance value between the target object and the vehicle is displayed on the screen so that the driver of the vehicle can make appropriate operations based on the actual distance.

[0198] Furthermore, the preset distance is a preset safe distance, which means that when the distance between the target object and the vehicle reaches this distance value, the driver of the vehicle needs to take timely action.

[0199] If the target object is a pedestrian, when the vehicle is turning and the distance between the pedestrian and the vehicle is less than or equal to a second distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output, wherein the second distance is less than the first distance;

[0200] If the target object is another vehicle, when the vehicle is traveling straight and the distance between the other vehicle and the vehicle is less than or equal to the third distance, the screen on the same side as the other vehicle displays the image of the target area corresponding to the other vehicle and outputs the first prompt information.

[0201] If the target object is another vehicle, when the vehicle is turning and the distance between the other vehicle and the vehicle is less than or equal to a fourth distance, the target area corresponding to the other vehicle is displayed on the screen on the same side as the other vehicle, and a first prompt message is output, wherein the fourth distance is less than the third distance.

[0202] Inevitably, in some road sections with complex road conditions, the target object may include both pedestrians and other vehicles. Therefore, in an optional implementation, if the target object includes pedestrians and other vehicles, when the vehicle is traveling straight and the distance between the pedestrian and any of the other vehicles and the vehicle is less than or equal to the larger of the first distance and the third distance, the image of the area obscured by the A-pillar is displayed on the corresponding screen, and a first prompt message is output.

[0203] If the target objects include pedestrians and other vehicles, when the vehicle is turning, and the distance between the pedestrian and any of the other vehicles and the vehicle is less than or equal to the larger of the second distance and the fourth distance, the image of the area obscured by the A-pillar is displayed on the corresponding screen, and a first prompt message is output.

[0204] In one optional implementation, the preset distance between the target object and the vehicle is generated in real time based on the state of the target object, and the preset distance is the safety distance, as described below:

[0205] Obtain a vehicle dataset, which includes the vehicle's historical driving status and first speed information, wherein the driving status includes at least going straight or turning;

[0206] The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information. The pedestrian morphology information is used to analyze the pedestrian's trajectory. The pedestrian's orientation can be obtained based on the pedestrian's facial orientation and body characteristics. The vehicle driving status information corresponds to the driving status information of other vehicles in the target object, including going straight or turning. Optionally, the driving status information also includes turning around or stopping. The vehicle trajectory information can be calculated based on the vehicle driving status information and the third speed information, and is used to simulate the travel routes of other vehicles to determine whether the other vehicles and the vehicle in the target object overlap on the predicted trajectory. Optionally, if there is overlap, it indicates a possibility of collision between the other vehicles and the vehicle in the target object, and the corresponding safety distance will be relatively increased.

[0207] Optionally, the pedestrian dataset further includes: a first weight and a second weight; the first weight is used to constrain the influence of the pedestrian morphology information on the corresponding safe distance result; the second weight is used to constrain the influence of the first speed information on the corresponding safe distance result.

[0208] Optionally, the vehicle dataset further includes: a third weight, a fourth weight, and a fifth weight; the third weight is used to constrain the influence of the vehicle driving status information on the corresponding safe distance result; the fourth weight is used to constrain the influence of the vehicle trajectory information on the corresponding safe distance result; and the fifth weight is used to constrain the influence of the third speed information on the corresponding safe distance result.

[0209] The target object dataset and the vehicle dataset are input into the safe distance prediction model to obtain the safe distance between the target object and the vehicle. The safe distance includes the first distance, the second distance, the third distance, or the fourth distance. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data.

[0210] It should be noted that the safe distance prediction model can be pre-installed inside the vehicle, and the process of obtaining the safe distance occurs inside the vehicle. Alternatively, the safe distance prediction model can be installed on a server outside the vehicle. The vehicle establishes a communication connection with the server and sends the target object dataset and the vehicle dataset to the server. The server inputs the target object dataset and the vehicle dataset into the pre-trained safe distance prediction model to obtain the safe distance between the target object and the vehicle. The vehicle then receives the safe distance sent from the server.

[0211] Once the safe distance to the target object is determined, the system determines whether to display a corresponding image on the screen and provide prompts to the driver based on the vehicle's driving status.

[0212] The third and fourth distances, besides being derived from analysis of relevant datasets, may also be derived using other algorithms. In one optional implementation, the values ​​of the third and fourth distances are determined based on the vehicle types of the other vehicles. These other vehicle types include at least bicycles, electric bicycles, cars, trucks, buses, trailers, incomplete vehicles, motorcycles, tractor-trailers, or special vehicles. The special vehicles include ambulances, police cars, or fire trucks. It is worth noting that the safe distance between other vehicles that are special vehicles is the greatest. Optionally, when a special vehicle is identified, a special prompt message is output to the driver so that the driver can avoid the special vehicle in time.

[0213] The above process effectively avoids safety accidents caused by the A-pillar obstructing the driver's view. However, other components of the vehicle can also be adapted according to this method. In one optional implementation, the shooting direction of the dashcam is adjusted so that the dashcam can capture images related to the target object. This is to address situations where there is no video recording when a safety accident related to the target object occurs. Specifically, after the distance between the target object and the vehicle is less than or equal to a preset distance, the shooting angle of the dashcam is adjusted by 10° towards the target object, so that the dashcam can capture images related to the target object as well as most of the image in front of the vehicle.

[0214] In summary, the method provided in this embodiment is beneficial to improving the flexibility, accuracy, and comprehensiveness of image display by the autonomous driving domain controller compared to existing single, fixed display and fixed framing solutions, thereby improving vehicle driving safety and enhancing the user's driving experience.

[0215] Furthermore, by setting a safe distance to give the driver sufficient reaction time, the system displays the corresponding area obscured by the A-pillar on the screen, informing the driver of the actual situation of the obscured area and outputting prompts. This prevents the driver from ignoring the area obscured by the A-pillar due to driving habits or other factors, thus forming a complete logic to inform the driver that the area obscured by the A-pillar poses a safety hazard and prevents accidents caused by the A-pillar obstructing the driver's view.

[0216] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0217] Please see Figure 4 , Figure 4 This application provides a schematic diagram of the structure of an image display device 40 based on the driver's field of vision in a driving scenario. The image display device 40 based on the driver's field of vision in a driving scenario can be a device in a vehicle as mentioned above. The image display device 40 based on the driver's field of vision in a driving scenario may include an acquisition unit 401, a prediction unit 402, and a display unit 403. The detailed description of each unit is as follows.

[0218] The acquisition unit 401 is used to acquire the comprehensive status information of the vehicle, the comprehensive status information including at least one of the following: steering wheel angle, driver's head posture, and the posture of the passenger in the front passenger seat.

[0219] The prediction unit 402 is used to predict, based on the comprehensive state information, a target area that the driver's eyes need to observe and that is in an obstructed state. The target area includes at least one of the following: an area obstructed by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; an area obstructed by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and a scene area corresponding to the image displayed by the right rearview mirror of the vehicle.

[0220] Display unit 403 is used to control the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area, wherein the left screen is a screen set on the left A-pillar and the right screen is a screen set on the right A-pillar.

[0221] In one possible implementation, the prediction unit 402 is specifically used for:

[0222] Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right side rearview mirror of the vehicle;

[0223] Based on the driver's head posture and the posture of the passenger in the front seat, determine whether the driver's line of sight to the right rearview mirror of the vehicle is blocked by the passenger in the front seat.

[0224] If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

[0225] In yet another possible implementation, the display unit 403 is specifically used for:

[0226] Acquire image data from at least one of the vehicle's multiple external cameras whose field of view includes the scene area corresponding to the image displayed by the right rearview mirror.

[0227] Based on the image data from the at least one external camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror, a target image adapted to the right screen of the right A-pillar is generated.

[0228] The target image is displayed on the right-side screen of the right-side A-pillar.

[0229] In yet another possible implementation, in generating a target image adapted to the right screen of the right A-pillar based on image data from the at least one exterior camera, the size of the right-side screen of the right A-pillar, and the imaging characteristics of the right-side rearview mirror, the display unit 403 is further configured to:

[0230] If the at least one exterior camera is a single camera, a target image adapted to the right screen of the right A-pillar is generated based on the image data acquired by the exterior camera, including the image data displayed by the right rearview mirror, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0231] If there are multiple external cameras, then target image data corresponding to the image displayed by the right rearview mirror is generated by fusing the multiple image data acquired by the external cameras; and a target image adapted to the right screen of the right A-pillar is generated based on the target image data, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0232] In yet another possible implementation, the prediction unit 402 is specifically used for:

[0233] Based on the comprehensive status information, the driver's pre-observation direction is determined;

[0234] If the driver's expected observation direction is the left side of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle, whose viewing area includes the scene area blocked by the left A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0235] If the driver's pre-observation direction is the right side of the vehicle and the driver's head is facing the right A-pillar of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle whose viewing area includes the scene area blocked by the right A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0236] In yet another possible implementation, the display unit 403 is specifically used for:

[0237] The driving status of the vehicle is determined based on the comprehensive status information of the vehicle, and the driving status includes at least going straight or turning;

[0238] The status information of the target object is determined based on the external status information, which is image data of the outside of the vehicle obtained by at least one external camera. The target object includes pedestrians and / or other vehicles. The status information of the target object includes the distance between the target object and the vehicle.

[0239] If the target object is a pedestrian, when the vehicle is traveling straight and the distance between the pedestrian and the vehicle is less than or equal to a first distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area.

[0240] If the target object is a pedestrian, when the vehicle is turning and the distance between the pedestrian and the vehicle is less than or equal to a second distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output, wherein the second distance is less than the first distance;

[0241] If the target object is another vehicle, when the vehicle is traveling straight and the distance between the other vehicle and the vehicle is less than or equal to the third distance, the screen on the same side as the other vehicle displays the image of the target area corresponding to the other vehicle and outputs the first prompt information.

[0242] If the target object is another vehicle, when the vehicle is turning and the distance between the other vehicle and the vehicle is less than or equal to a fourth distance, the target area corresponding to the other vehicle is displayed on the screen on the same side as the other vehicle, and a first prompt message is output, wherein the fourth distance is less than the third distance.

[0243] In yet another possible implementation, the display unit 403 is further configured to:

[0244] Obtain a vehicle dataset, which includes the vehicle's historical driving status and first speed information, wherein the driving status includes at least going straight or turning;

[0245] The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information.

[0246] The target object dataset and the vehicle dataset are input into the safe distance prediction model to obtain the safe distance between the target object and the vehicle. The safe distance includes the first distance, the second distance, the third distance, or the fourth distance. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data.

[0247] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a vehicle 50 provided in an embodiment of this application. The vehicle 50 includes an autonomous driving domain controller 501 and a memory 502. The autonomous driving domain controller 501 and the memory 502 can be connected via a bus or other means. This embodiment of the application takes a bus connection as an example.

[0248] The autonomous driving domain controller 501 is the computing and control core of the vehicle 50. It can parse various instructions and data within the vehicle 50. For example, the autonomous driving domain controller 501 can be a central autonomous driving domain controller (CPU), which can transmit various interactive data between internal structures of the vehicle 50, and so on. The memory 502 is a memory device in the vehicle 50 used to store programs and data. It is understood that the memory 502 here can include the vehicle 50's built-in memory, or it can include extended memory supported by the vehicle 50. The memory 502 provides storage space, which stores the vehicle 50's operating system. This storage space also stores the program code or instructions required for the autonomous driving domain controller to perform corresponding operations. Optionally, this storage space can also store relevant data generated after the autonomous driving domain controller performs the corresponding operation.

[0249] In this embodiment of the application, the vehicle 50 further includes a body domain controller, which is used to control devices inside and / or outside the vehicle.

[0250] In this embodiment, the autonomous driving domain controller 501 runs the executable program code in the memory 502 to perform the following operations:

[0251] The vehicle's comprehensive status information is obtained, including at least one of the following: steering wheel angle, driver's head posture, and the posture of the passenger in the front passenger seat.

[0252] Based on the comprehensive state information, the target area that the driver's eyes need to observe and is in a blocked state is predicted. The target area includes at least one of the following: the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and the scene area corresponding to the image displayed by the right rearview mirror of the vehicle.

[0253] The system controls the left screen on the left A-pillar and / or the right screen on the right A-pillar of the vehicle to display images according to the target area. The left screen is a screen set on the left A-pillar, and the right screen is a screen set on the right A-pillar.

[0254] In one alternative embodiment, regarding the prediction of the target area that the driver's eyes need to observe and that is obscured based on the comprehensive state information, the autonomous driving domain controller 501 is specifically configured to:

[0255] Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right side rearview mirror of the vehicle;

[0256] Based on the driver's head posture and the posture of the passenger in the front seat, determine whether the driver's line of sight to the right rearview mirror of the vehicle is blocked by the passenger in the front seat.

[0257] If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

[0258] In one alternative embodiment, regarding the display of images on the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle according to the target area, the autonomous driving domain controller 501 is specifically configured to:

[0259] Acquire image data from at least one of the vehicle's multiple external cameras whose field of view includes the scene area corresponding to the image displayed by the right rearview mirror.

[0260] Based on the image data from the at least one external camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror, a target image adapted to the right screen of the right A-pillar is generated.

[0261] The target image is displayed on the right-side screen of the right-side A-pillar.

[0262] In one alternative embodiment, in generating a target image adapted to the right screen of the right A-pillar based on image data from the at least one exterior camera, the size of the right-side screen of the right A-pillar, and the imaging characteristics of the right-side rearview mirror, the autonomous driving domain controller 501 is specifically configured to:

[0263] If the at least one exterior camera is a single camera, a target image adapted to the right screen of the right A-pillar is generated based on the image data acquired by the exterior camera, including the image data displayed by the right rearview mirror, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0264] If there are multiple external cameras, then target image data corresponding to the image displayed by the right rearview mirror is generated by fusing the multiple image data acquired by the external cameras; and a target image adapted to the right screen of the right A-pillar is generated based on the target image data, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

[0265] In one alternative embodiment, regarding the prediction of the target area that the driver's eyes need to observe and that is obscured based on the comprehensive state information, the autonomous driving domain controller 501 is specifically configured to:

[0266] Based on the comprehensive status information, the driver's pre-observation direction is determined;

[0267] If the driver's expected observation direction is the left side of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle, whose viewing area includes the scene area blocked by the left A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0268] If the driver's pre-observation direction is the right side of the vehicle and the driver's head is facing the right A-pillar of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle whose viewing area includes the scene area blocked by the right A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

[0269] In one alternative embodiment, regarding the display of images on the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle according to the target area, the autonomous driving domain controller 501 is specifically configured to:

[0270] The driving status of the vehicle is determined based on the comprehensive status information of the vehicle, and the driving status includes at least going straight or turning;

[0271] The status information of the target object is determined based on the external status information, which is image data of the outside of the vehicle obtained by at least one external camera. The target object includes pedestrians and / or other vehicles. The status information of the target object includes the distance between the target object and the vehicle.

[0272] If the target object is a pedestrian, when the vehicle is traveling straight and the distance between the pedestrian and the vehicle is less than or equal to a first distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area.

[0273] If the target object is a pedestrian, when the vehicle is turning and the distance between the pedestrian and the vehicle is less than or equal to a second distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output, wherein the second distance is less than the first distance;

[0274] If the target object is another vehicle, when the vehicle is traveling straight and the distance between the other vehicle and the vehicle is less than or equal to the third distance, the screen on the same side as the other vehicle displays the image of the target area corresponding to the other vehicle and outputs the first prompt information.

[0275] If the target object is another vehicle, when the vehicle is turning and the distance between the other vehicle and the vehicle is less than or equal to a fourth distance, the target area corresponding to the other vehicle is displayed on the screen on the same side as the other vehicle, and a first prompt message is output, wherein the fourth distance is less than the third distance.

[0276] In one alternative embodiment, before determining the state information of the target object based on the external state information, the autonomous driving domain controller 501 is further configured to:

[0277] Obtain a vehicle dataset, which includes the vehicle's historical driving status and first speed information, wherein the driving status includes at least going straight or turning;

[0278] The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information.

[0279] The target object dataset and the vehicle dataset are input into the safe distance prediction model to obtain the safe distance between the target object and the vehicle. The safe distance includes the first distance, the second distance, the third distance, or the fourth distance. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data.

[0280] It should be noted that the implementation of each operation can also be referred to accordingly. Figure 3 The corresponding description of the method embodiments shown.

[0281] This application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by an autonomous driving domain controller, causes the autonomous driving domain controller to perform the operations described in the embodiment.

[0282] This application also provides a computer program product that, when running on an autonomous driving domain controller, implements the operations performed in the embodiments described above.

[0283] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for displaying images based on the driver's field of vision in a driving scenario, characterized in that, The method is applied to the autonomous driving domain controller of a vehicle's domain controller system. The autonomous driving domain controller includes: a first interaction layer, a processing layer, and a second interaction layer. The first interaction layer is used to interact with the vehicle's domain controller; the processing layer is used to process comprehensive status information or image data; and the second interaction layer is used to send screen control requests to the vehicle's domain controller to control the screen to display target images. The method includes: acquiring comprehensive status information of the vehicle, wherein the comprehensive status information includes at least one of the following: driver's head posture and the posture of the passenger in the front passenger seat; Based on the comprehensive state information, the target area that the driver's eyes need to observe and is in a blocked state is predicted. The target area includes at least one of the following: the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; the area blocked by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and the scene area corresponding to the image displayed by the right rearview mirror of the vehicle. The target area is controlled to display images on the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle. The left screen is a screen set on the left A-pillar and the right screen is a screen set on the right A-pillar. Wherein, when the comprehensive state information of the vehicle includes the driver's head posture and the posture of the passenger in the front seat; the step of predicting the target area that the driver's eyes need to observe and that is in an obstructed state based on the comprehensive state information includes: Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right side rearview mirror of the vehicle; Based on the driver's head posture and the posture of the passenger in the front seat, determine whether the driver's line of sight to the right rearview mirror of the vehicle is blocked by the passenger in the front seat. If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

2. The method according to claim 1, characterized in that, If the target area is the scene area corresponding to the image displayed by the right rearview mirror of the vehicle, then the comprehensive status information of the vehicle includes in-vehicle status information and out-of-vehicle status information, and the in-vehicle status information includes the driver's head posture and the posture of the passenger in the front seat. The step of controlling the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area includes: Acquire image data from at least one of the vehicle's multiple external cameras whose field of view includes the scene area corresponding to the image displayed by the right rearview mirror. Based on the image data from the at least one external camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror, a target image adapted to the right screen of the right A-pillar is generated. The target image is displayed on the right-side screen of the right-side A-pillar.

3. The method according to claim 2, characterized in that, The step of generating a target image adapted to the right screen of the right A-pillar based on image data from at least one external camera, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror includes: If the at least one exterior camera is a single camera, a target image adapted to the right screen of the right A-pillar is generated based on the image data acquired by the exterior camera, including the image data displayed by the right rearview mirror, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror. If there are multiple external cameras, then target image data corresponding to the image displayed by the right rearview mirror is generated by fusing the multiple image data acquired by the external cameras; and a target image adapted to the right screen of the right A-pillar is generated based on the target image data, the size of the right screen of the right A-pillar, and the imaging characteristics of the right rearview mirror.

4. The method according to any one of claims 1 to 3, characterized in that, When the comprehensive state information includes at least the driver's head posture, the driver's head posture includes the direction the driver's head is facing; The step of predicting the target area that the driver's eyes need to observe and that is currently obstructed based on the comprehensive state information includes: Based on the comprehensive status information, the driver's pre-observation direction is determined; If the driver's expected observation direction is the left side of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle, whose viewing area includes the scene area blocked by the left A-pillar of the vehicle, a target area is determined. The target area is the area blocked by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance. If the driver's pre-observation direction is the right side of the vehicle and the driver's head is facing the right A-pillar of the vehicle, then based on the image data of at least one of the multiple external cameras of the vehicle whose view area includes the view area obscured by the right A-pillar of the vehicle, a target area is determined. The target area is the area obscured by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance.

5. The method according to any one of claims 2-3, characterized in that, If the target area is an area obscured by the left A-pillar of the vehicle and the furthest distance between the area boundary and the vehicle is less than a preset distance, or an area obscured by the right A-pillar of the vehicle and the furthest distance between the area boundary and the vehicle is less than a preset distance, then the comprehensive status information of the vehicle includes the in-vehicle status information and the out-of-vehicle status information, and the in-vehicle status information includes the driver's head posture; the step of controlling the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area includes: The driving status of the vehicle is determined based on the comprehensive status information of the vehicle, and the driving status includes at least going straight or turning; The status information of the target object is determined based on the external status information, which is image data of the outside of the vehicle obtained by at least one external camera. The target object includes pedestrians and / or other vehicles. The status information of the target object includes the distance between the target object and the vehicle. If the target object is a pedestrian, when the vehicle is traveling straight and the distance between the pedestrian and the vehicle is less than or equal to a first distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output. The first prompt message is used to prompt the driver to observe the screen corresponding to the target area. If the target object is a pedestrian, when the vehicle is turning and the distance between the pedestrian and the vehicle is less than or equal to a second distance, the target image of the target area corresponding to the pedestrian is displayed on the screen on the same side as the pedestrian, and a first prompt message is output, wherein the second distance is less than the first distance; If the target object is another vehicle, when the vehicle is traveling straight and the distance between the other vehicle and the vehicle is less than or equal to the third distance, the screen on the same side as the other vehicle displays the image of the target area corresponding to the other vehicle and outputs the first prompt information. If the target object is another vehicle, when the vehicle is turning and the distance between the other vehicle and the vehicle is less than or equal to a fourth distance, the target area corresponding to the other vehicle is displayed on the screen on the same side as the other vehicle, and a first prompt message is output, wherein the fourth distance is less than the third distance.

6. The method according to claim 5, characterized in that, Before determining the state information of the target object based on the external state information, the method further includes: Obtain a vehicle dataset, which includes the vehicle's historical driving status and first speed information, wherein the driving status includes at least going straight or turning; The target object dataset is generated based on the vehicle exterior status information. The target object dataset includes a pedestrian dataset and / or a vehicle dataset. The pedestrian dataset includes pedestrian morphology information and second speed information. The pedestrian morphology information includes height and orientation. The vehicle dataset includes vehicle driving status information, vehicle trajectory information, and third speed information. The target object dataset and the vehicle dataset are input into the safe distance prediction model to obtain the safe distance between the target object and the vehicle. The safe distance includes the first distance, the second distance, the third distance, or the fourth distance. The safe distance prediction model is a model trained based on multiple target object dataset samples, corresponding vehicle dataset samples, and corresponding safe distances. The target object dataset and the vehicle dataset are feature data, and the safe distance is label data.

7. An image display device based on the driver's field of vision in a driving scenario, characterized in that, The device is applied to the autonomous driving domain controller of a vehicle's domain controller system. The autonomous driving domain controller includes: a first interaction layer, a processing layer, and a second interaction layer. The first interaction layer is used for information interaction with the vehicle's domain controller; the processing layer is used for processing comprehensive status information or image data; and the second interaction layer is used for sending screen control requests to the vehicle's domain controller to control the screen to display target images. The device includes: The acquisition unit is used to acquire comprehensive status information of the vehicle, the comprehensive status information including at least one of the following: driver's head posture and the posture of the passenger in the front passenger seat; The prediction unit is used to predict, based on the comprehensive state information, the target area that the driver's eyes need to observe and that is in an obstructed state. The target area includes at least one of the following: the area obstructed by the left A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; the area obstructed by the right A-pillar of the vehicle and the farthest distance between the area boundary and the vehicle is less than a preset distance; and the scene area corresponding to the image displayed by the right rearview mirror of the vehicle. The display unit is used to control the left screen of the left A-pillar and / or the right screen of the right A-pillar of the vehicle to display images according to the target area. The left screen is a screen set on the left A-pillar and the right screen is a screen set on the right A-pillar. Wherein, when the comprehensive state information of the vehicle includes the driver's head posture and the posture of the passenger in the front seat; the step of predicting the target area that the driver's eyes need to observe and that is in an obstructed state based on the comprehensive state information includes: Based on the driver's head posture, the driver's pre-observation direction is determined to be the right side of the vehicle, and the driver's head is facing the right side rearview mirror of the vehicle; Based on the driver's head posture and the posture of the passenger in the front seat, determine whether the driver's line of sight to the right rearview mirror of the vehicle is blocked by the passenger in the front seat. If the driver's line of sight to the right-side rearview mirror of the vehicle is blocked by the passenger in the front seat, then the target area that the driver's eyes need to observe and that is blocked is determined to be the scene area corresponding to the image displayed in the right-side rearview mirror of the vehicle.

8. A vehicle, characterized in that, include: An autonomous driving domain controller, a memory, and a computer program stored in the memory and capable of running on the autonomous driving domain controller, wherein the computer program, when executed by the autonomous driving domain controller, implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on an autonomous driving domain controller, implements the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, When the computer program product includes a computer program that is run on an autonomous driving domain controller, it implements the method as described in any one of claims 1-6.

Images