Display control method and device of head-up display and electronic equipment

Abstract

The invention relates to the field of auxiliary driving, and provides a display control method and device for a head-up display and electronic equipment, and the method comprises the steps: obtaining a display icon outputted by the head-up display in a display region on a front windshield, at least one environment object in a front road environment, and a sight line unit vector of a driver; according to the icon position of the display icon, the object position of the environment object and the sight line unit vector, monitoring whether the display icon and the environment object are matched with the sight line unit vector at the same time; and in response to the existence that the display icon and the environment object are matched with the sight line unit vector at the same time, adjusting display parameters of the display icon.

Description

Technical Field

[0001] The present application relates to the field of automotive electronics technology, and particularly to a display control method, device, and electronic device for a head-up display. Background Art

[0002] The head-up display of a vehicle projects driving prompt information onto the front windshield to prompt the driver. However, the driving prompt information may affect the driver's observation of the road environment ahead, or the road environment ahead may affect the driver's observation of the driving prompt information, thereby affecting the driving experience. Summary of the Invention

[0003] In view of the above problems, the present application provides a display control method, device, and electronic device for a head-up display to improve the driving experience. The specific solutions are as follows:

[0004] In a first aspect of the present application, a display control method for a head-up display is provided. The method includes:

[0005] Obtaining a display icon output in a display area of the head-up display on the front windshield, at least one environmental object in the road environment ahead, and a line-of-sight unit vector of the driver;

[0006] Monitoring whether there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector according to the icon position of the display icon, the object position of the environmental object, and the line-of-sight unit vector;

[0007] In response to there being a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector, adjusting the display parameters of the display icon.

[0008] In a possible implementation, the situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector includes:

[0009] The icon area of the display icon and the object area of the environmental object are matched in regional position; both the icon area and the object area are two-dimensional areas in the vehicle coordinate system;

[0010] And, the line-of-sight vector corresponding to the line-of-sight unit vector has an intersection point with the icon area that matches the object area, and the line-of-sight unit vector is a vector in the vehicle coordinate system.

[0011] In a possible implementation, the icon area of the display icon and the object area of the environmental object being matched in regional position includes:

[0012] The coincidence parameter of the icon area of the display icon and the object area of the environmental object in regional position is greater than or equal to a first threshold;

[0013] Wherein, the coincidence parameter is the intersection over union of the icon area and the object area; alternatively, the coincidence parameter is determined based on the intersection over union of the icon area and the object area and an influence parameter;

[0014] The influence parameter includes at least one of the following:

[0015] The distance between the icon area and the object area at the center point of the area, and the diagonal length of the minimum bounding rectangle of the icon area and the object area.

[0016] In a possible implementation, the line-of-sight vector is: a vector determined with the center position of the driver's pupil as the starting point based on the line-of-sight direction corresponding to the line-of-sight unit vector;

[0017] Wherein, the line-of-sight vector corresponding to the line-of-sight unit vector has an intersection with the icon area matching the object area, including:

[0018] The intersection coordinates of the line-of-sight vector on the display area are within the icon area matching the object area.

[0019] In a possible implementation, the display parameters of the display icon include at least one of the following:

[0020] The icon position of the display icon, the icon size of the display icon, the icon color lightness of the display icon, the icon brightness of the display icon, and the color lightness of the background area of the display icon.

[0021] In a possible implementation, adjusting the display parameters of the display icon includes:

[0022] Adjusting the display icon from a first icon position to a second icon position, where the display icon at the second icon position does not match the object area of any of the environmental objects in terms of regional position.

[0023] In a possible implementation, adjusting the display parameters of the display icon includes:

[0024] Adjusting the display icon from a first icon size to a second icon size, where the display icon with the second icon size does not match the object area of any of the environmental objects in terms of regional position.

[0025] In a possible implementation, adjusting the display parameters of the display icon includes:

[0026] Adjusting the display icon from a first icon brightness to a second icon brightness according to the environmental brightness of the background environmental area corresponding to the display icon in the front road environment;

[0027] Wherein, the brightness of the second icon is determined based on a brightness adjustment coefficient, the maximum brightness of the display icon, the minimum brightness of the display icon, the maximum ambient brightness and the minimum ambient brightness of the background ambient area corresponding to the display icon in the front road environment; the brightness adjustment coefficient is determined based on the ambient brightness corresponding to the display icon in the front road environment.

[0028] The second aspect of the present application provides a display control device for a head-up display, the device includes:

[0029] An obtaining unit, configured to obtain a display icon output in a display area of the head-up display on the front windshield, at least one ambient object in the front road environment, and a line-of-sight unit vector of the driver;

[0030] A monitoring unit, configured to monitor whether there is a situation where the display icon and the ambient object simultaneously match the line-of-sight unit vector according to the icon position of the display icon, the object position of the ambient object, and the line-of-sight unit vector;

[0031] An adjustment unit, configured to adjust the display parameters of the display icon in response to the situation where the display icon and the ambient object simultaneously match the line-of-sight unit vector.

[0032] The third aspect of the present application provides a computer program product, including computer-readable instructions, when the computer-readable instructions run on an electronic device, enabling the electronic device to implement the display control method of the head-up display in the first aspect or any implementation manner of the first aspect.

[0033] The fourth aspect of the present application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0034] The memory is used to store a computer program;

[0035] The processor is configured to execute the computer program so that the electronic device can implement the display control method of the head-up display in the first aspect or any implementation manner of the first aspect.

[0036] The fifth aspect of the present application provides a computer storage medium, the storage medium carrying one or more computer programs, when the one or more computer programs are executed by an electronic device, enabling the electronic device to implement the display control method of the head-up display in the first aspect or any implementation manner of the first aspect.

[0037] With the above technical solution, in a display control method, device, and electronic device of a head-up display provided by this application, by obtaining display icons output in the display area of the head-up display on the front windshield, at least one environmental object in the front road environment, and the line-of-sight unit vector of the driver, and accordingly monitoring whether there is a situation where a display icon and an environmental object simultaneously match the line-of-sight unit vector. If there is a situation where a display icon and an environmental object simultaneously match the line-of-sight unit vector, it indicates that there is interference between the display icon and the environmental object from the driver's perspective, which will further affect the driver's viewing of the display icon or the environmental object. At this time, in this application, when it is monitored that there is a situation where a display icon and an environmental object simultaneously match the line-of-sight unit vector, the display parameters of the display icon can be adjusted so that the display icon and the environmental object do not simultaneously match the line-of-sight unit vector, thus not affecting the driver's viewing of the display icon or the environmental object that the line-of-sight unit vector matches, thereby improving the driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original components and elements are not necessarily drawn to scale.

[0039] Figure 1 Schematic diagrams of a head-up display, a front-view camera, and a driver monitoring camera in an advanced driver assistance system for a vehicle;

[0040] Figure 2 Flowchart of a display control method of a head-up display provided by an embodiment of this application;

[0041] Figure 3 Flowchart of training a first model in an embodiment of this application;

[0042] Figure 4 Flowchart of training a second model in an embodiment of this application;

[0043] Figure 5 Example diagram of the eye box area of the driver's field of view in a vehicle;

[0044] Figure 6 Example diagram of the intersection over union between an object area and an icon area in an embodiment of this application;

[0045] Figure 7 Example diagram of the distance between the center points and the diagonal of the minimum bounding rectangle between an object area and an icon area in an embodiment of this application;

[0046] Figure 8 Example diagram of the display area partition in an embodiment of this application;

[0047] Figure 9 It is an exemplary diagram for adjusting the icon position of the display icon in the embodiment of the present application;

[0048] Figure 10 It is an exemplary diagram for adjusting the icon size of the display icon in the embodiment of the present application;

[0049] Figure 11 It is a schematic diagram of the display icon of the real-time vehicle speed;

[0050] Figure 12 It is a schematic diagram of the display icon of the speed limit reminder;

[0051] Figure 13 It is a schematic diagram of the display icon of the navigation information;

[0052] Figure 14 It is an exemplary diagram for adjusting the icon position and icon size of the display icon in the embodiment of the present application;

[0053] Figure 15 It is a schematic diagram of the 2D icon projected by the head-up display when the road environment brightness is relatively bright;

[0054] Figure 16 It is a schematic diagram of the 2D icon projected by the head-up display after adaptive brightness adjustment;

[0055] Figure 17 It is a schematic diagram of the 2D icon projected by the head-up display when the road environment brightness is relatively low;

[0056] Figure 18 It is a schematic diagram of the 2D icon projected by the head-up display after adaptive brightness adjustment;

[0057] Figure 19 It is an exemplary diagram of the light window effect of the 2D icon;

[0058] Figure 20 It is an exemplary diagram when the road environment color is light and single and the 2D icon projected by the head-up display is also a lighter color;

[0059] Figure 21 It is an exemplary diagram for darkening the color of the 2D icon in the embodiment of the present application;

[0060] Figure 22 It is an exemplary diagram of the lighter 2D icon projected by the head-up display when the road environment color is relatively complex;

[0061] Figure 23 It is an exemplary diagram for changing the color of the lighter 2D icon to a darker 2D icon in the embodiment of the present application;

[0062] Figure 24This is an example diagram in the embodiment of the present application where a darker base color is added around the periphery of a lighter 2D icon background;

[0063] Figure 25 This is an example diagram of the light band in the embodiment of the present application;

[0064] Figure 26 This is a schematic structural diagram of a display control device for a head-up display provided in the embodiment of the present application;

[0065] Figure 27 This is a schematic structural diagram of an electronic device provided in the embodiment of the present application. Detailed implementation manners

[0066] The embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms used in the embodiments of the present application are only for explaining the specific embodiments of the present application, and are not intended to limit the present application.

[0067] The embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0068] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these process, method, product or device.

[0069] The present application can be applied to the field of assisted driving. Taking a vehicle as an example of a transportation tool, the application scenarios of the present application implemented in products will be introduced below.

[0070] First, the application scenarios of the present application will be introduced.

[0071] Such as Figure 1As shown in , the vehicle is traveling on a road, and the vehicle is deployed with at least an image acquisition device and a head-up display. The image acquisition device refers to a device capable of acquiring images, such as the forward-looking camera and driver monitoring camera of the advanced driving assistance system ADAS (Advanced Driving Assistance System). The forward-looking camera is used to acquire environmental images of the road environment in front of the vehicle, and the driver monitoring camera is used to acquire facial images of the driver. A head-up display refers to a device capable of outputting light to achieve image projection, such as an augmented reality head-up display device AR-HUD. The head-up display is used to output any type of display icons such as 2D and 3D, such as a vehicle speed icon, on the corresponding display area (i.e., the HUD display surface) on the front windshield.

[0072] Among them, the head-up display includes at least an eye core module and a brightness module. The eye core module is used to obtain the area where the eyeball and the pupil are located, that is, the position information of the human eye, based on the facial image collected by the driver monitoring camera. The eye core position is used to determine the position of the display area on the front windshield. For example, the position of the display area on the front windshield is: the intersection area of ​​the field of view corresponding to the eye core position and the front windshield of the vehicle. The brightness module is used to provide the projection brightness of the display area, so that the head-up display projects content in the display area according to the projection brightness, such as displaying icons and other image content. Furthermore, the head-up display also includes a light source, which projects the image content onto the front windshield and reflects it into the human eye through multiple internal light path foldbacks, so that the driver can see a long-distance and enlarged virtual image in front.

[0073] The image acquisition device at least includes an image signal processing module ISP (Image Signal Processing), which is used to process the image acquired by the image acquisition device to obtain relevant statistical data of the image. For example, the H3A (Histogram 3A) statistical data of the image includes the pixel height, pixel width, etc. of the image, which includes pixel statistical data, such as the average pixel value of the image.

[0074] Based on this, the image acquisition device acquires an environmental image of the road environment ahead within the vehicle's driving field of view and a facial image of the driver, processes the image, and sends the image processing result to a head-up display. The head-up display projects the image content on the front windshield of the vehicle based on the image processing result of the image acquisition device. The image at least includes navigation prompt information such as a vehicle speed icon to assist vehicle driving.

[0075] In the above-mentioned assisted driving scenarios, although the head-up display can reduce the number of times the driver looks down at the driving information on the dashboard, the various 2D and 3D display icons projected on the front windshield will also have a certain impact on the driver's observation of the road environment ahead. For example, when the speed icon coincides with the position of a pedestrian, it may interfere with the driver's observation of the pedestrian. When the navigation information is the same color as the road background, it may interfere with the driver's observation of the navigation information. When the road environment brightness is too bright, it may interfere with the light-colored display icons, making it impossible for the driver to obtain the information projected by the head-up display in a timely manner.

[0076] In view of this, in the present application, the information of the road environment ahead can be obtained through the Advanced Driving Assistance System (ADAS) equipped in the vehicle. Further processing and analysis can obtain the three-dimensional coordinates of pedestrians and vehicles, the color and brightness of the road environment background, etc. Specifically, the results obtained by processing the forward-looking road environment information obtained by ADAS can be combined with the various 2D and 3D icons projected by the head-up display for analysis, and the parameters such as the position, color, brightness, and change speed of the various 2D and 3D icons projected can be adaptively adjusted to achieve the effect of conforming to the complex and changeable road environment and reducing interference to the driver.

[0077] To achieve the above object, an embodiment of the present application provides a method for controlling the display of a head-up display. The method for controlling the display of the head-up display according to the embodiment of the present application will be introduced in detail below with reference to the accompanying drawings.

[0078] Refer to Figure 2 , which is a flowchart for implementing a method for controlling the display of a head-up display provided by an embodiment of the present application. This method can be applied to an electronic device capable of data processing. The electronic device can be a head-up display deployed on a transportation vehicle such as a vehicle, or an in-vehicle terminal connected to the head-up display. The technical solution in this embodiment is mainly used to improve the driving experience of transportation vehicles.

[0079] Specifically, the method in this embodiment may include the following steps:

[0080] Step 201: Obtain the display icons output in the display area of the head-up display on the front windshield, at least one environmental object in the forward road environment, and the line-of-sight unit vector of the driver.

[0081] In one implementation, in this embodiment, a front camera can be used to collect environmental images of the road environment in front of the vehicle; the environmental images are images based on the front view coordinate system corresponding to the front camera; then, a first model is used to process the environmental images to obtain the three-dimensional coordinates of at least one environmental object in the road environment in front, and the three-dimensional coordinates are coordinates based on the front view coordinate system.

[0082] Among them, the three-dimensional coordinates of the environmental object include the two-dimensional coordinates of the environmental object and the target distance, and the target distance is the distance between the environmental object and the front camera.

[0083] In a specific implementation, the first model can be a machine learning model constructed and trained based on a convolutional neural network. Among them, the process of obtaining the model weights of the first model, that is, the model training process, is as Figure 3 shown below:

[0084] Step 301: Collect a large number of environmental images of the road environment in front in different situations to prepare for subsequent production of the data set. Based on the complex road environment scenes in the actual driving scenario, the collected environmental images of the road environment in front at least include images in different weathers (sunny, rainy, snowy, foggy, etc.), different times (day, evening, and night), different roads (rural roads, urban roads, highways, etc.), different congestion levels, and other scenarios.

[0085] Step 302: Divide and label the important target categories in the collected environmental images of the road environment in front in a large number of different scenarios to obtain an image data set. The important target categories here include pedestrians, vehicles, traffic lights, etc.

[0086] Step 303: Preprocess the obtained image data set, standardize and normalize the image data therein, and perform data augmentation on the image data to increase the diversity of the image data and reduce overfitting.

[0087] Step 304: Build a convolutional neural network model and set appropriate configuration parameters, and then perform convolutional neural network training. The configuration parameters can include loss function, number of training epochs, learning rate change function, etc.

[0088] Step 305: After optimization and adjustment, obtain the weight file of the model. Optimization includes adjusting the model hierarchical structure, various parameters during training, etc.

[0089] As Figure 1 shown, the front view coordinate system is represented by O c -X c Y c Z c That is, with the centroid of the front camera as the origin O c , the X c direction is the optical axis direction of the camera, and the Zc The direction is vertically upward from the camera, Y c The direction is perpendicular to X c O c Z c plane. The front camera is responsible for sensing the environmental image of the road environment ahead. Taking the vehicle's centroid as the origin O v The vehicle coordinate system is with O v -X v Y v Z v represented, where X v The direction is the vehicle's forward direction, Z v The direction is vertically upward from the vehicle plane, Y v The direction is perpendicular to X v O v Z v plane.

[0090] The driver monitoring coordinate system is with O d -X d Y d Z d represented, where taking the centroid of the driver monitoring camera as the origin O d , X d The direction is the optical axis direction of the camera, Z d The direction is vertically upward from the camera, Y d The direction is perpendicular to X d O d Z d plane.

[0091] Among them, the relative positions of the driver monitoring camera and the front camera with respect to the vehicle centroid are fixed.

[0092] Based on this, in this embodiment, using the first model, that is, a machine learning model based on a convolutional neural network, the two-dimensional coordinates of environmental objects such as pedestrians, vehicles, traffic lights, etc. in the road environment ahead can be obtained in the front view coordinate system O c -X c Y c Z c below, each environmental object is represented by a target detection box, and the target detection box is represented by represented, where is the point at the upper left corner of the target detection box, is the point at the lower right corner of the target detection box. In addition, the target distance can be added during the construction of the training set before training the convolutional neural network. Thus, through the machine learning model of the convolutional neural network, while obtaining the two-dimensional coordinates of the target detection box, the target distance of the environmental object corresponding to the target detection box relative to the front camera can be obtained. Therefore, the three-dimensional information of the target detection box of the environmental object is

[0093] In one implementation, in this embodiment, a driver monitoring camera can be used to collect the facial image of the driver. The facial image is an image based on the driver monitoring coordinate system. Then, a second model is used to process the facial image to obtain the pupil center position and the eyeball center position of the driver. Finally, based on the pupil center position and the eyeball center position, the line-of-sight unit vector of the driver is obtained. The line-of-sight unit vector is a vector based on the driver monitoring coordinate system.

[0094] Among them, the driver monitoring camera is a high-resolution camera. Thus, after capturing the facial image of the driver through the high-resolution camera, the line-of-sight analysis of the driver can be performed based on the facial image.

[0095] Specifically, the second model can be a machine learning model based on a convolutional neural network. In this embodiment, the facial image collected by the driver monitoring camera can be input into the second model for line-of-sight analysis to obtain the pupil center position and the eyeball center position of the driver.

[0096] Among them, the training process of the convolutional neural network of the second model is as Figure 4 shown below:

[0097] Step 401: Collect a large number of facial images of drivers in different scenarios. The scenarios here include scenarios where the driver is looking in different directions, whether wearing sunglasses, etc.

[0098] Step 402: Label the face, pupil center, and eyeball center (i.e., iris center) of the driver to obtain an image data set.

[0099] Step 403: Preprocess the obtained image data set, standardize and normalize the image data, and perform data augmentation on the image data to increase the diversity of the image data and reduce overfitting.

[0100] Step 404: Build a convolutional neural network and set the corresponding configuration parameters, and then perform network training. The configuration parameters can include: the cross-entropy loss function selected for the classification task, the mean square error loss function selected for the regression on the pupil center coordinates and iris center coordinates, the stochastic gradient descent (SGD) optimizer selected for the optimizer, etc.

[0101] Step 405: Obtain a weight file through optimization and adjustment. The optimization here includes adjusting the model hierarchical structure, training various parameters, etc.

[0102] Thus, using the above second model to obtain the pupil center position and the eyeball center position (i.e., the iris center position) in the driver monitoring coordinate system Od -X d Y d Z d Under the following circumstances, the coordinates of the iris center position of the driver are The coordinates of the pupil center position of the driver are

[0103] Furthermore, after obtaining the pupil center position and the eyeball center position, the driver's line-of-sight unit vector can be obtained in the following manner:

[0104] First, the vector from the eyeball center position to the pupil center position Can be approximately regarded as the line-of-sight vector and is calculated through formula (1):

[0105]

[0106] Then, calculate the modulus length according to formula (2)

[0107]

[0108] Finally, calculate the line-of-sight unit vector according to formula (3)

[0109]

[0110] For the sake of simplified representation, in this embodiment, the line-of-sight unit vector can be represented as in formula (4):

[0111]

[0112] In one implementation manner, in this embodiment, the display area projected by the head-up display on the front windshield can be icon-calibrated to obtain the icon positions of each display icon output in the display area.

[0113] For example, in this embodiment, it can be known through calibration the position information of the area frame of the display area of the head-up display on the front windshield in the vehicle coordinate system Among them, one area frame corresponds to one display icon, and the display icon can be represented by the four vertex coordinates of the corresponding area frame, where Is the coordinate of the upper left corner point of the area frame of the display icon, Is the coordinate of the lower left corner point of the area frame of the display icon, Is the coordinate of the lower right corner point of the area frame of the display icon, Is the coordinate of the upper left corner point of the area frame of the display icon.

[0114] Among them, taking a 2D icon frame in the display area of the head-up display as an example, the icon position of this 2D icon frame can be represented by the coordinates indicates that, among them are the upper left coordinates of a 2D icon within the display area of the head-up display, and are the lower right coordinates.

[0115] Step 202: According to the icon position of the display icon, the object position of the environmental object, and the line-of-sight unit vector, monitor whether there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector.

[0116] Among them, as Figure 5 shown, the black frame in the figure is the entire field of view of the driver, which means the field of view that can be seen when the driver only turns the eyes without moving the head. The image dotted frames are the upper eye box area, the middle eye box area, and the lower eye box area respectively, which means that when the driver sits in the cab with different heights, the display area of the head-up display seen is also different. Or rather, the position of the iris center of the driver will affect the display area of the head-up display seen by the driver. For example, for a taller driver will see Figure 5 the upper eye box area in a medium-height driver will see Figure 5 the middle eye box area in a shorter driver will see Figure 5 the lower eye box area in. The effective display areas of the head-up display corresponding to different eye box areas have different coordinates in the vehicle coordinate system O v -X v Y v Z v Below, the coordinate information of the specific eye box area in the vehicle coordinate system needs to be calibrated according to the specific situation and then corresponding can be carried out. It can be obtained from Figure 5 that the interference degrees of the display icons projected by the head-up display in different eye box areas on environmental objects such as pedestrians in the front road environment are different.

[0117] Based on this, monitoring whether there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector in step 202 can be understood as: in the eye box area corresponding to the line-of-sight unit vector, monitoring whether there are both a display icon and an environmental object at the same time. If there are both a display icon and an environmental object at the same time, then there may be interference between the display icon and the environmental object.

[0118] In one implementation, that the display icon and the environmental object simultaneously match the line-of-sight unit vector can be: the icon area of the display icon and the object area of the environmental object match in the area position, and there is an intersection point between the line-of-sight vector corresponding to the line-of-sight unit vector and the icon area of the matching object area.

[0119] Among them, both the icon area and the object area are two-dimensional areas in the vehicle coordinate system; the line-of-sight unit vector is a vector in the vehicle coordinate system. The icon area is determined based on the two-dimensional coordinates representing the icon position of the display icon, the object area is determined based on the two-dimensional coordinates representing the object position of the environmental object, and the line-of-sight vector corresponding to the line-of-sight unit vector can be: a vector determined with the center position of the driver's pupil as the starting point based on the line-of-sight direction corresponding to the line-of-sight unit vector.

[0120] Specifically, in this embodiment, first, the three-dimensional information of the environmental object in the front road environment in the front view coordinate system O c -X c Y c Z c is converted into the coordinate representation in the vehicle coordinate system Then, based on the three-dimensional coordinates of the environmental object and the two-dimensional coordinates of the display icon, it is determined whether the object area of the environmental object and the icon area of the display icon match in the regional position, that is, whether there is an overlap; in the case where the object area and the icon area overlap, it is determined whether the line-of-sight vector corresponding to the line-of-sight unit vector has an intersection point on the icon area that overlaps with the object area. If there is an intersection point, then it can be determined that there is a display icon and an environmental object that simultaneously match the line-of-sight unit vector.

[0121] In one implementation, the object area and the icon area match in the regional position, that is, there is an overlap, which can be: the overlap parameter between the icon area of the display icon and the object area of the environmental object is greater than or equal to the first threshold.

[0122] In a specific implementation, the overlap parameter can be the intersection over union (IoU) between the icon area and the object area. Based on this, in the case where the intersection over union is greater than or equal to the first threshold, it can be determined that the object area and the icon area match in the regional position, that is, there is an overlap.

[0123] Specifically, as Figure 6 shown in, where S A can be the area of the object area of the environmental object, S B can be the area of the icon area of the display icon, S C can be the area of the intersection of the object area and the icon area, and the intersection over union between the object area and the icon area can be represented by IoU, and IoU can be calculated by formula (5):

[0124]

[0125] In another specific implementation, the overlap parameter can be determined based on the intersection over union between the icon area and the object area and the influence parameter.

[0126] Among them, the influencing parameters include at least one of the following:

[0127] The distance between the icon area and the object area at the center point of the area, and the diagonal length of the minimum bounding rectangle of the icon area and the object area.

[0128] For example, due to the existence of environmental objects with small sizes in the front road environment, such as children with short heights, although they can be relatively close to the transportation vehicle, due to size reasons, environmental objects such as the target detection frame and the display icon such as the 2D icon frame may have a small overlapping area, which is prone to misjudgment. Based on this, in this embodiment, in order to more accurately measure whether the environmental object and the display icon match, that is, whether they overlap, the distance between the icon area and the object area at the center point of the area can be added to the calculation of the intersection over union according to the corresponding weight, and the obtained overlapping parameter can be called the enhanced intersection over union EIoU (Enhance IoU).

[0129] As Figure 7 shown in a , the center point of the box 1 where the display icon is located is a(x a ), and the center point of the box 2 where the environmental object is located is b(x b ), y b ), and the Euclidean distance between the two boxes can be calculated according to formula (6), and this Euclidean distance is the distance between the icon area and the object area at the center point of the area:

[0130]

[0131] Furthermore, in this embodiment, in order to more accurately measure whether the icon area and the object area match, that is, whether they overlap, the diagonal length of the minimum bounding rectangle of the icon area and the object area and the distance between the icon area and the object area at the center point of the area can be added to the calculation of the intersection over union according to the corresponding weight to obtain a new overlapping parameter. For example, the upper right vertex coordinates of box 1 are (x 1rmax ), y 1rmin ), and the lower left vertex coordinates of box 2 are (x 2lmin ), y 2lmax ), and the diagonal length c 1r2l of one of the bounding rectangles can be calculated according to formula (7):

[0132]

[0133] And the upper left vertex coordinates of box 1 are (x 1lmin ), y 1lmin ), and the lower right vertex coordinates of box 2 are (x 2rmax ), y 2rmax), the diagonal length c of another bounding rectangle can be calculated according to formula (8). 1l2r :[[]]END]]

[0134]

[0135] The minimum bounding rectangle is the smallest matrix that contains the icon area and the object area. Based on this, the diagonal length c of the minimum bounding rectangle is determined according to formula (9). 12 :[[]]END]]

[0136] c 12 = max{c 1r2l , c 1l2r}}(9)

[0137] Based on this, the coincidence parameter, i.e., the enhanced intersection over union EIoU, can be calculated according to formula (10):

[0138]

[0139] where λ 1 is the adjustment weight used to control the influence of the distance between the centers of the two bounding boxes and the diagonal length of the minimum bounding rectangle on the EIoU. The larger λ 1 is, the greater the influence of the distance between the icon area and the object area at the center point of the area on the EIoU, and the greater the influence of the diagonal length of the minimum bounding rectangle of the icon area and the object area on the EIoU. When d 12 is relatively small, will be closer to λ 1 , resulting in an increase in EIoU; when d 12 is relatively large, will be closer to 0, making the EIoU closer to the ordinary IoU.

[0140] Based on this, after obtaining the coincidence parameter such as EIoU between the object area of the environmental object, i.e., the target detection bounding box and the icon area of the displayed icon, i.e., the 2D icon bounding box , it can be determined whether the object area of the environmental object, i.e., the target detection bounding box 1 matches, i.e., coincides with, the icon area of the displayed icon, i.e., the 2D icon bounding box according to the first threshold w . For example, it is determined according to formula (11) whether the 2D icon bounding box of the displayed icon noticed by the driver's line of sight will interfere with the target detection bounding box

[0141]

[0142] In one implementation, the sight vector is: a vector determined based on the sight direction corresponding to the sight unit vector and with the center position of the driver's pupil as the starting point. Based on this, the sight vector corresponding to the sight unit vector and the icon area of ​​the matching object area have an intersection, which can be: the intersection coordinates of the sight vector on the display area are within the icon area of ​​the matching object area.

[0143] For example, in this embodiment, when determining the 2D icon frame of the display icon Will it detect the target frame of environmental objects such as pedestrians? After the interference is caused, the driver's line of sight unit vector can be determined Is it in 2D icon frame? Specifically, we can determine the driver's sight unit vector Display area with head-up display The plane where Is the intersection point in the 2D icon box of the displayed icon? The specific process includes the following:

[0144] 1. The driver's sight unit vector can be From the driver monitoring coordinate system O d -X d Y d Z d Transform to vehicle coordinate system O v -X v Y v Z v Get Specifically, the driver monitoring coordinate system O d -X d Y d Z d The point under the vehicle coordinate system O v -X v Y v Z v The conversion operations between the following points are as follows:

[0145] In the vehicle coordinate system O v -X v Y v Z v Point O d The coordinates (x v，d ,y v，d , z v，d ), X v Around X d The rotation angle is α d , T v Around Y d The rotation angle is β d , Z v Around Zd The rotation angle is γ d . The rotation matrix R d As shown in formula (12):

[0146]

[0147] And the translation matrix T d As shown in formula (13):

[0148]

[0149] Convert the points in the driver monitoring coordinate system O d -X d Y d Z d to the vehicle coordinate system O v -X v Y v Z v . The rotation matrix R d and the translation matrix T d are used to make the conversion as shown in formula (14):

[0150]

[0151] Among them, the rotation angles α d , β d and γ d as well as the translation parameters x v，d , y v，d and z v，d can be obtained from the digital model file of ADAS. Similarly, in this embodiment, the points in the front view coordinate system O c -X c Y c Z c can be converted with the points in the vehicle coordinate system O v -X v Y v Z v .

[0152] II. Calculate the display area of the head-up display of the plane where it is located Specifically, in this embodiment, three points in the display area of the head-up display can be selected: and Based on these three points, calculate the approximate plane of the display area Specifically as follows:

[0153] First, construct two vectors based on the three selected points and As shown in formula (15) and formula (16):

[0154]

[0155] Then, according to formula (17), the normal vector can be calculated based on the cross product of the vectors and as

[0156]

[0157] where the normal vector is perpendicular to the plane Further calculation of (17) gives the following as shown in formula (18):

[0158]

[0159] For the sake of simplified expression, in this embodiment, the normal vector can be denoted as Furthermore, the equation of the plane can be calculated, as shown in formula (19):

[0160]

[0161] where and can be solved by substituting three points and as shown in formula (20), formula (21) and formula (22):

[0162]

[0163] Furthermore, the plane is simplified as shown in formula (23):

[0164]

[0165] where

[0166] III. After obtaining the plane v -X v Y v Z v of the driver's line of sight and the display area of the head-up display in the vehicle coordinate system O the intersection point of the line-of-sight vector and the plane can be calculated, and then it is determined whether the intersection point is within the display area of the head-up display, that is, whether the line-of-sight vector is within the 2D icon frame Intersect. The specific process is as follows:

[0167] First, convert the center position of the eyeball (i.e., the center position of the iris) to the vehicle coordinate system to obtain

[0168] Then, parameterize the line-of-sight vector with the center position E of the eyeball v as the starting point, and the result is as shown in formula (24):

[0169]

[0170] After that, substitute the parameterized line-of-sight vector into formula (23), and the result is as shown in formula (25):

[0171]

[0172] Further solution gives the result as shown in formula (26):

[0173]

[0174] where,

[0175] Then, substitute the obtained into formula (24) to get the line-of-sight vector and the intersection point with the plane where the display area of the head-up display is located For simplicity of representation, can be used for substitution.

[0176] Finally, after determining that the 2D icon box interferes with the target detection box of the environmental object, further determine whether the intersection point of the driver's line-of-sight vector with the plane where the display area of the head-up display is located is within the 2D icon box as shown in formula (27):

[0177]

[0178] Step 203: In response to the existence of a display icon and an environmental object that simultaneously match the line-of-sight unit vector, adjust the display parameters of the display icon.

[0179] Among them, the display parameters of the display icon may include at least one of the following:

[0180] The icon position of the displayed icon, the icon size of the displayed icon, the icon color lightness of the displayed icon, the icon brightness of the displayed icon, and the color lightness of the background area of the displayed icon.

[0181] In one implementation, when adjusting the display parameters of the displayed icon in step 203, it can be achieved in the following manner:

[0182] Adjust the displayed icon from the first icon position to the second icon position, and the displayed icon at the second icon position does not match the object area of any environmental object in terms of area position.

[0183] Among them, the second icon position can be a position in the display area where there is no other displayed icon, and at the second icon position, the displayed icon does not match other environmental objects. Specifically, in this embodiment, a display position that does not match other environmental objects, that is, the second icon position, can be searched for in the display area. Based on this, the displayed icon is adjusted to the second icon position.

[0184] For example, as Figure 8 shown, it is the partition of the display area of the head-up display, which can be specifically divided into 64 partitions in 4 rows and 16 columns. In this embodiment, the icon position of the 2D icon of the vehicle speed information "30 km / h" in the display area projected by the head-up display can be adjusted. As Figure 9 shown, the icon position is changed from the partitions of 3o, 3p, 4o, and 4p in the lower right corner to the partitions of 3a, 3b, 4a, and 4b in the lower left corner. The partitions of 3a, 3b, 4a, and 4b in the lower left corner do not match the line-of-sight unit vector of any environmental object at the same time.

[0185] It should be noted that the displayed icon at the second icon position can match the line-of-sight unit vector, but the displayed icon at the second icon position does not match the line-of-sight unit vector of any environmental object at the same time. For example, the displayed icon at the second icon position is within the eye box area corresponding to the driver, but the coincidence parameter with any environmental object in terms of area position is less than the first threshold.

[0186] Or, the displayed icon at the second icon position can not match the line-of-sight unit vector. For example, the displayed icon at the second icon position is not within the eye box area corresponding to the driver. At this time, the displayed icon at the second icon position can coincide with a certain environmental object or not coincide with any environmental object, but does not match the line-of-sight unit vector.

[0187] In one implementation, when adjusting the display parameters of the displayed icon in step 203, it can be achieved in the following manner:

[0188] Adjust the display icon from the first icon size to the second icon size, and the display icon of the second icon size does not match the object area of any environmental object in terms of regional position.

[0189] Among them, since the size of the display icon of the second icon size changes relative to that before adjustment, the intersection over union or enhanced intersection over union between the icon area of the display icon and the object area of the environmental object is small, less than the first threshold. At this time, the display icon and the environmental object do not overlap, that is, they do not interfere, so the display icon and the environmental object no longer match the line-of-sight unit vector at the same time.

[0190] For example, as Figure 10 shown in, reduce the 2D icon of the vehicle speed information "30 km / h" by half, so that the intersection over union between the 2D icon box and the target detection box of the pedestrian is less than the first threshold w 1 , and at this time, the 2D icon of the vehicle speed information does not affect the driver's observation of the pedestrian.

[0191] In one implementation, when adjusting the display parameters of the display icon in step 203, it can be achieved in the following way:

[0192] Adjust the display icon from the first icon size to the second icon size, and adjust the display icon from the first icon position to the second icon position.

[0193] For example, as Figure 11 shown in, is the display icon of the real-time vehicle speed, Figure 12 is the display icon of the speed limit prompt, Figure 13 is the display icon of the navigation information. The 2D icons projected by the specific head-up display include but are not limited to this. Based on this, in this embodiment, when transforming the position and size of the 2D icon, the area position without important targets such as pedestrians can be searched first. If there is no such area, the 2D icon can be scaled down proportionally, and then the area position without important targets can be searched again. It should be noted that there is a minimum limit for the icon size of the display icon to avoid shrinking the 2D icon too much so that it is difficult to observe.

[0194] For example, as Figure 14 shown in, in this embodiment, the 2D icon of the vehicle speed information "30 km / h" can be reduced by half and replaced with the partition of 1p in the upper right corner, so that the intersection over union between the 2D icon box and the target detection box of the pedestrian is less than the first threshold w 1 , and moreover, the upper right corner 1p does not match the line-of-sight unit vector of any environmental object at the same time.

[0195] In one implementation, when adjusting the display parameters of the display icon in step 203, it includes:

[0196] Adjust the display icon from the first icon brightness to the second icon brightness according to the ambient brightness of the background environment area corresponding to the display icon in the front road environment;

[0197] Wherein, the second icon brightness is determined based on the brightness adjustment coefficient, the maximum brightness of the display icon, the minimum brightness of the display icon, the maximum ambient brightness and the minimum ambient brightness of the background environment area corresponding to the display icon in the front road environment.

[0198] Specifically, the brightness adjustment coefficient is determined based on the ambient brightness of the background environment area corresponding to the display icon in the front road environment.

[0199] In one case, if the ambient brightness of the background environment area corresponding to the display icon in the front road environment is greater than or equal to the second threshold, it indicates that the front road environment is too bright. At this time, the brightness adjustment coefficient is a coefficient indicating an increase in the icon brightness. Correspondingly, based on the brightness adjustment coefficient, the maximum brightness of the display icon, the minimum brightness of the display icon, the maximum ambient brightness and the minimum ambient brightness of the background environment area corresponding to the display icon in the front road environment, the second icon brightness is determined according to the first method.

[0200] In another case, if the ambient brightness of the background environment area corresponding to the display icon in the front road environment is less than or equal to the third threshold, it indicates that the front road environment is too dark. At this time, the brightness adjustment coefficient is a coefficient indicating a decrease in the icon brightness. Correspondingly, based on the brightness adjustment coefficient, the maximum brightness of the display icon, the minimum brightness of the display icon, the maximum ambient brightness and the minimum ambient brightness of the background environment area corresponding to the display icon in the front road environment, the second icon brightness is determined according to the second method.

[0201] Wherein, the second method is different from the second method.

[0202] In specific implementation, in this embodiment, after obtaining the display area of the head-up display in the vehicle coordinate system it is possible to obtain the color and brightness information of the actual front road environment background corresponding to the display area of the head-up display in the driver's eyes Among them, the environmental image of the front road environment obtained by the front view camera is not completely consistent with the environmental image of the front road environment seen by the driver. In this embodiment, coordinate transformation can be performed on the environmental image.

[0203] It should be noted that after obtaining the eye box area of the driver, the areas corresponding to different eye box areas are in the front view camera and the front view coordinate system O c -X c Y c Z cThe corresponding area below can be obtained through multiple calibrations. Although each pixel does not exactly correspond one-to-one to the area of the forward road environment covered by the display area of the head-up display as seen by the driver in the driver's seat, the brightness and color used to statistically analyze this area can meet the requirements. Among them, the forward road environment covered by the display area of the head-up display in the forward view coordinate system is Among them is the coordinate of its upper left corner, is the coordinate of its lower right corner. As shown in the partition of the display area of the head-up display in Figure 8 , the same is done for for partitioning. Statistically analyze the brightness information in each partition to obtain

[0204] Based on this, a second threshold such as the brightness threshold w 2 is preset. When the brightness of a certain area is higher than the threshold w 2 , it can be considered that the road environment at this time will prevent the driver from obtaining the display icon projected by the head-up display, and this display icon coincides with an environmental object in the area where the brightness is higher than the threshold w 2 . And, a third threshold such as the brightness threshold w 3 is preset. When the brightness of a certain area is lower than the threshold w 3 , it can be considered that the display icon projected by the head-up display at this time is too bright and may prevent the driver from observing the environmental object that coincides with the overly bright display icon in the forward road environment. As shown in formula (28):

[0205]

[0206] Among them, i and j are the row and column as shown in Figure 8 , i ∈ [1, 4], j ∈ [a, b,..., p].

[0207] Among them, when , it is determined that the road environment is too bright. At this time, the brightness of the partition in the display area of the head-up display can be increased simultaneously to cope with the overly bright road environment brightness. Specifically, the increased icon brightness can be calculated according to formula (29)

[0208]

[0209] Among them, λ 2 is the brightness adjustment coefficient for increasing the brightness, is the minimum brightness of the icon projected by the head-up display in the current partition, is the maximum brightness of the icon projected by the head-up display for the current partition, is the minimum ambient brightness of the current partition, is the maximum ambient brightness of the current partition. Specifically, the brightness when the high beam of a vehicle in the oncoming lane is turned on at night can be selected, the road ambient brightness under extremely dark conditions at night can be selected.

[0210] Based on this, the situation where the content projected by the head-up display is interfered when the brightness is too high in the front road environment can be alleviated. By adaptively increasing the brightness of the image projected by the head-up display, and selecting the logarithmic conditional method can be more suitable for the human eye's perception of brightness.

[0211] It should be noted that λ 2 as the brightness adjustment coefficient for increasing brightness can be effectively selected and calibrated according to the actual situation, making the brightness prompt closer to the actual situation.

[0212] When , it is determined that the road environment is too dark. At this time, the brightness of the AR-HUD projection partition should be reduced simultaneously to cope with the too dark road environment brightness. Specifically, the increased icon brightness can be calculated according to formula (30)

[0213] where λ 3 is the brightness adjustment coefficient for reducing brightness. λ 3 as the brightness adjustment coefficient for reducing brightness can be effectively selected and calibrated according to the actual situation, making the brightness prompt closer to the actual situation.

[0214] As Figure 15 shown in, it is a schematic diagram of the 2D icon projected by the head-up display when the road environment brightness is relatively bright. After the brightness of the 2D icon is adjusted in this embodiment, as Figure 16 shown in, it is a schematic diagram of the 2D icon projected by the head-up display after adaptive brightness adjustment. Figure 16 The brightness of the 2D icon projected by the head-up display in can be adaptively increased when the road environment is relatively bright, which can better produce a contrast difference with the road environment and is convenient for the driver to read the required information from it.

[0215] As Figure 17 shown in, it is a schematic diagram of the 2D icon projected by the head-up display when the road environment brightness is relatively low. After the brightness of the 2D icon is adjusted in this embodiment, as Figure 18 shown in, it is a schematic diagram of the 2D icon projected by the head-up display after adaptive brightness adjustment.Figure 18 The brightness of the 2D icons projected by the head-up display adaptively decreases in a darker road environment, which can, on the other hand, effectively reduce the interference of the backlight.

[0216] When the environment is darker, the higher the brightness of the 2D icons projected by the head-up display, the higher the backlight brightness of its partitions will be, thus generating a light window effect, which in turn interferes with the normal driving of the driver, as Figure 19 shown. Based on this, in this embodiment, the backlight brightness of the icons displayed in the display area can be appropriately reduced to avoid the light window effect.

[0217] Furthermore, sometimes the color of the road environment is complex or single, and it is difficult to improve the effect by changing the position and brightness of the 2D or 3D icons projected by the head-up display, and at the same time, some icons may not be suitable for position change. At this time, the color of the icons projected by the head-up display can be adaptively adjusted to adapt to the covered front road environment, so as to distinguish from the complex or single road environment.

[0218] As Figure 20 shown, when the color of the road environment is light and single, and the 2D icons projected by the head-up display are also light-colored, the driver's viewing experience is poor and it is difficult to distinguish the 2D icons. At this time, as Figure 21 shown, the color can be adaptively darkened to improve the driver's viewing experience.

[0219] As Figure 22 shown, when the color of the road environment is complex, the light-colored 2D icons projected by the head-up display are difficult to be recognized quickly. As Figure 23 shown in Figure 24 shown, in order to improve the driver's ability to identify the 2D icons projected by the head-up display with less attention in a complex road environment, a darker background color can be added around the background of the light-colored 2D icons. In this way, the 2D icons can be well highlighted in a complex road environment, reducing the driver's distraction of more attention.

[0220] Furthermore, in this embodiment, the light band icons in the display area can be adjusted according to the target distance. The flow speed of the light band in the light band icon corresponds to the target distance. Specifically, in this embodiment, the three-dimensional information of the front environmental object can be obtained After being converted to the vehicle coordinate system, it can be obtained Furthermore, the flow speed of the processed light band can be determined, as shown in formula (31):

[0221]

[0222] For example, if there are environmental objects such as vehicles or obstacles within 30 meters ahead and the vehicle is moving straight ahead, as Figure 25 shown, the flow velocity of the light band shown in Figure 25 can be reduced to prompt the driver that there are environmental objects such as vehicles ahead and to pay attention to maintaining a safe distance. If there are no vehicles or obstacles within 30 meters ahead, the flow velocity of the light band can be increased to indicate to the driver that normal driving can be maintained.

[0223] With the above technical solution, in a display control method of a head-up display provided by the present application, by obtaining the display icons output in the display area of the head-up display on the front windshield, at least one environmental object in the front road environment, and the line-of-sight unit vector of the driver, and accordingly monitoring whether there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector. If there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector, it indicates that there is interference between the display icon and the environmental object from the driver's perspective, which will affect the driver's viewing of the display icon or the environmental object. At this time, in the present application, when it is monitored that there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector, the display parameters of the display icon can be adjusted so that the display icon and the environmental object do not simultaneously match the line-of-sight unit vector, thus not affecting the driver's viewing of the display icon or the environmental object that matches the line-of-sight unit vector, thereby improving the driving experience.

[0224] Referring to Figure 26 , which is a schematic structural diagram of a display control device of a head-up display provided by an embodiment of the present application. This device can be configured in an electronic device capable of data processing. The electronic device can be a head-up display deployed on a transportation vehicle such as a vehicle, or it can be an in-vehicle terminal connected to the head-up display. The technical solution in this embodiment is mainly used to improve the driving experience of the transportation vehicle.

[0225] Specifically, the device in this embodiment may include the following units:

[0226] An obtaining unit 2601, configured to obtain the display icons output in the display area of the vehicle head-up display on the front windshield, at least one environmental object in the vehicle front road environment, and the line-of-sight unit vector of the driver;

[0227] A monitoring unit 2602, configured to monitor whether there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector according to the icon position of the display icon, the object position of the environmental object, and the line-of-sight unit vector;

[0228] An adjustment unit 2603, configured to adjust the display parameters of the display icon in response to the situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector.

[0229] With the above technical solution, in a display control device of a head-up display provided by the present application, by obtaining a display icon output in a display area of the head-up display on the front windshield, at least one environmental object in the front road environment, and a line-of-sight unit vector of the driver, and accordingly monitoring whether there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector. If there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector, it indicates that there is interference between the display icon and the environmental object from the driver's perspective, which will further affect the driver's viewing of the display icon or the environmental object. At this time, in the present application, when it is monitored that there is a situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector, the display parameters of the display icon can be adjusted so that the display icon and the environmental object do not simultaneously match the line-of-sight unit vector, thus not affecting the driver's viewing of the display icon or the environmental object that the line-of-sight unit vector matches, thereby improving the driving experience.

[0230] In one implementation, the situation where the display icon and the environmental object simultaneously match the line-of-sight unit vector includes:

[0231] The icon area of the display icon and the object area of the environmental object are matched in regional position; both the icon area and the object area are two-dimensional areas in the vehicle coordinate system; and, the line-of-sight vector corresponding to the line-of-sight unit vector has an intersection point with the icon area that matches the object area, and the line-of-sight unit vector is a vector in the vehicle coordinate system.

[0232] Among them, the situation where the icon area of the display icon and the object area of the environmental object are matched in regional position includes: the coincidence parameter between the icon area of the display icon and the object area of the environmental object is greater than or equal to a first threshold; where the coincidence parameter is the area intersection-over-union ratio between the icon area and the object area; or, the coincidence parameter is determined based on the area intersection-over-union ratio between the icon area and the object area and an influence parameter; the influence parameter includes at least one of the following: the distance between the regional center points of the icon area and the object area, the diagonal length of the minimum bounding rectangle of the icon area and the object area.

[0233] And the line-of-sight vector is: a vector determined with the pupil center position of the driver as the starting point based on the line-of-sight direction corresponding to the line-of-sight unit vector; among them, the situation where the line-of-sight vector corresponding to the line-of-sight unit vector has an intersection point with the icon area that matches the object area includes: the intersection point coordinates of the line-of-sight vector on the display area are within the icon area that matches the object area.

[0234] In one implementation, the display parameters of the display icon include at least one of the following: the icon position of the display icon, the icon size of the display icon, the icon color lightness of the display icon, the icon brightness of the display icon, and the color lightness of the background area of the display icon.

[0235] In one implementation, the adjustment unit 2603 is specifically configured to: adjust the display icon from a first icon position to a second icon position, where the display icon at the second icon position does not match the object area of any of the environmental objects in terms of regional position.

[0236] In one implementation, the adjustment unit 2603 is specifically configured to: adjust the display icon from a first icon size to a second icon size, where the display icon with the second icon size does not match the object area of any of the environmental objects in terms of regional position.

[0237] In one implementation, the adjustment unit 2603 is specifically configured to: adjust the display icon from a first icon brightness to a second icon brightness according to the ambient brightness of the background environment area corresponding to the display icon in the front road environment; wherein, the second icon brightness is determined based on a brightness adjustment coefficient, the maximum brightness of the display icon, the minimum brightness of the display icon, the maximum ambient brightness and the minimum ambient brightness of the background environment area corresponding to the display icon in the front road environment; and the brightness adjustment coefficient is determined based on the ambient brightness of the background environment area corresponding to the display icon in the front road environment.

[0238] It should be noted that the specific implementation manners of the units in this embodiment can refer to the corresponding content in the previous text and will not be elaborated here.

[0239] Reference Figure 27 , is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The electronic device can be a head-up display deployed on a transportation vehicle such as a vehicle, or can be on a vehicle terminal connected to the head-up display. The technical solution in this embodiment is mainly used to improve the driving experience of the transportation vehicle.

[0240] Specifically, the electronic device in this embodiment may include at least one processor 2701 and a memory 2702 connected to the processor 2701, where:

[0241] The memory 2702 is used to store computer programs;

[0242] The processor 2701 is used to execute the computer programs so that the electronic device can implement the display control method of the head-up display as described in any one of the above embodiments.

[0243] An embodiment of the present application further provides a computer program product, including computer-readable instructions, which, when running on an electronic device, enable the electronic device to implement any one of the display control methods of the head-up display provided by the embodiments of the present application.

[0244] An embodiment of the present application further provides a computer-readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can be enabled to implement any one of the display control methods of the head-up display provided by the embodiments of the present application.

[0245] Taking a vehicle as an example of a means of transportation, referring to Figure 1 , when a pedestrian crosses the road in front of the vehicle, at this time the vehicle is relatively close to the pedestrian, and the speed "30 km / h" displayed by the display icon in the HUD display content area S HUD (i.e., the display area) coincides with the pedestrian in front at the lower right corner of the display area. If the speed information "30 km / h" in the display area is not adjusted in terms of position, color or brightness, when the driver's line of sight looks at the speed information, it may distract the driver from identifying the speed information and cause the driver to ignore the more important pedestrian information that coincides with his line of sight.

[0246] If the display area of the head-up display is partitioned as Figure 8 shown, specifically divided into 64 partitions in 4 rows and 16 columns, then the speed information "30 km / h" is in the lower right corner areas 3o, 3p, 4o and 4p. The ADAS can be used to obtain the road environment information in front, and the convolutional neural network detection can be used to identify the position information of the pedestrian. After operations such as coordinate conversion, it can be obtained that the position of the pedestrian in front is in columns m, n, o and p of the display area partition of the head-up display. The binocular pupil information of the driver is detected by the driver monitoring camera and analyzed to obtain the current line of sight direction of the driver. When the driver's line of sight pays attention to the speed information "30 km / h", if the speed information "30 km / h" coincides with the pedestrian at this time, through analysis and processing, the position, color or brightness of the speed information "30 km / h" can be adaptively adjusted so as not to coincide with the more important pedestrian display area.

[0247] Specifically, the adjustment method of the speed information "30 km / h" can be as Figure 9 and Figure 14 shown. The position adjustment of the 2D icon includes moving to different areas and shrinking in proportion, where Figure 9 is Figure 1 the initial speed information "30 km / h" shown in Figure 8 moved to the lower left corner areas 3a, 3b, 4a and 4b of the display partition. Figure 14 is to shrink the speed information "30 km / h" in proportion and move it toFigure 8 The partition at the upper right corner of the display partition, 1p. When moving the position of the 2D icon, it should be noted that it should not coincide with other important targets in the road environment ahead except for the current pedestrian, and frequent position movements should be avoided to interfere with the normal driving of the driver. When scaling down the 2D icon proportionally, it should also be noted that it should not be too small, so that the driver needs more attention during normal observation, which may interfere with normal driving.

[0248] In the face of a complex road environment, sometimes it is necessary to adaptively adjust the display content of the head-up display so as not to overly distract the driver's attention from more important road environment targets, such as adaptively adjusting the icon position and size; sometimes it is necessary to adaptively adjust so that the driver can notice the display content of the head-up display from the complex and unimportant road environment with less attention, such as adaptively adjusting the icon brightness and color. Sometimes it is necessary to change the display content of the head-up display according to the actual road environment conditions to give the driver better driving prompts, such as Figure 25 As shown, when there is a vehicle within 30m ahead, the guiding light band can slow down the change speed, and when there is no vehicle within 30m ahead, the change speed of the guiding light band can be increased.

[0249] In summary, the technical solution of the adaptive adjustment of the display icon position, color, and brightness of the head-up display proposed in this application can, compared with the traditional icon adjustment, first analyze and process the complex road environment information ahead obtained by ADAS, and then adaptively adjust information such as the icon position, color, and brightness through an algorithm, so that the driver can obtain a better driving experience in different road environments.

[0250] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0251] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions accomplished by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, in more cases, software program implementation is a better embodiment. Based on such understanding, the technical solution of the present application, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disc of a computer, and includes several instructions for causing a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0252] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0253] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store, or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, hard disk, magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

Claims

1. A display control method for a head-up display, characterized in that: The method comprises: Obtaining a display icon output by a head-up display in a display area on a front windshield, at least one environmental object in a front road environment, and a sight line unit vector of a driver; According to the icon position of the display icon, the object position of the environmental object and the sight line unit vector, monitoring whether the display icon and the environmental object simultaneously match the sight line unit vector; In response to the display icon and the environmental object simultaneously matching the sight line unit vector, a display parameter of the display icon is adjusted.

2. The method according to claim 1, characterized in that The display icon and the environmental object simultaneously match the sight line unit vector, including: The icon area of ​​the display icon matches the object area of ​​the environmental object in terms of area position; both the icon area and the object area are two-dimensional areas in a vehicle coordinate system; Furthermore, a sight line vector corresponding to the sight line unit vector and an icon area matching the object area have an intersection, and the sight line unit vector is a vector in the vehicle coordinate system.

3. The method according to claim 2, characterized in that The icon area of ​​the display icon matches the object area of ​​the environmental object in terms of area position, including: An overlap parameter between the icon region of the display icon and the object region of the environmental object in terms of region position is greater than or equal to a first threshold; The overlap parameter is an area intersection-over-combination ratio between the icon area and the object area; or, the overlap parameter is determined based on the area intersection-over-combination ratio between the icon area and the object area and an influencing parameter; The influencing parameters include at least one of the following: The distance between the icon area and the object area at the center point of the area, and the diagonal length of the minimum enclosing rectangle of the icon area and the object area.

4. The method according to claim 2, characterized in that: The sight line vector is: a vector determined based on the sight line direction corresponding to the sight line unit vector and taking the pupil center position of the driver as the starting point; The sight line vector corresponding to the sight line unit vector and the icon area matching the object area have an intersection point, including: The coordinates of the intersection point of the sight line vector on the display area are within the icon area matching the object area.

5. The method according to claim 1 or 2, characterized in that: The display parameters of the display icon include at least one of the following: The icon position of the displayed icon, the icon size of the displayed icon, the icon color brightness of the displayed icon, the icon brightness of the displayed icon, and the color brightness of the background area of ​​the displayed icon.

6. The method according to claim 2, characterized in that Adjusting the display parameters of the display icon includes: The displayed icon is adjusted from a first icon position to a second icon position, wherein the displayed icon at the second icon position does not match an object region of any of the environmental objects in terms of region position.

7. The method according to claim 2, characterized in that Adjusting the display parameters of the display icon includes: The displayed icon is adjusted from a first icon size to a second icon size, wherein the displayed icon of the second icon size does not match an object area of ​​any of the environmental objects in terms of area position.

8. The method according to claim 2, characterized in that: Adjusting the display parameters of the display icon includes: According to the ambient brightness of the background environment area corresponding to the displayed icon in the road environment ahead, adjusting the displayed icon from a first icon brightness to a second icon brightness; Among them, the brightness of the second icon is determined based on the brightness adjustment coefficient, the maximum brightness of the displayed icon, the minimum brightness of the displayed icon, the maximum ambient brightness and the minimum ambient brightness of the background environment area corresponding to the displayed icon in the road environment ahead; the brightness adjustment coefficient is determined based on the ambient brightness corresponding to the displayed icon in the road environment ahead.

9. A display control device for a head-up display, characterized in that: The device comprises: an obtaining unit, configured to obtain a display icon outputted by the head-up display in a display area on the front windshield, at least one environmental object in a front road environment, and a sight line unit vector of the driver; A monitoring unit, configured to monitor whether the displayed icon and the environmental object simultaneously match the visual unit vector according to the icon position of the displayed icon, the object position of the environmental object and the visual unit vector; An adjustment unit is used to adjust the display parameters of the display icon in response to the presence of the display icon and the environmental object matching the sight line unit vector at the same time.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the display control method of the head-up display according to any one of claims 1 to 8.

Images