Display effect fusion method, system, storage medium and main control unit

Abstract

The present application provides a display effect fusion method, system, storage medium and main control unit. The method is applied to a head-up display system, including: obtaining a driving image of a driving vehicle, wherein the driving image contains an object of interest; obtaining the eye position of a driver in the driving vehicle; displaying a virtual image of the object in the virtual image display surface according to the eye position, the driving image and a preset virtual image display surface, so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene. In this way, factors such as the height, driving habits and visual habits of different people can be taken into account, so that the virtual image can be well integrated with the real environment, and the virtual image offset can be avoided as much as possible, and the display of the virtual image can be adjusted in real time according to the change of the driver's eye position, so as to adapt to drivers of different heights, driving habits and visual habits, ensure the virtual image display effect, and enhance the driving experience of the driver.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle driving, and more specifically, to a display effect fusion method, system, storage medium and main control unit. Background Art

[0002] Head Up Display (HUD), also known as parallel display system, refers to a driver-centric blind-operation, multi-functional dashboard. AR HUD is a HUD technology that integrates AR (Augmented Reality) technology.

[0003] HUD technology is currently used in vehicles. The HUD system is an integrated automotive electronic display device composed of electronic components, optical display components, and control units. The HUD used in vehicles can display information from the instrument cluster and the interior of the vehicle and an object fusion frame on the front windshield, allowing the driver to easily obtain driving information while driving. For example, the vehicle speed, navigation information, active safety warning signals, etc. are projected in the form of images and characters through optical components directly in front of the driver to form a virtual image, so that the driver can simultaneously see the images, characters and other information superimposed on the exterior scene while observing the road conditions outside the vehicle.

[0004] When HUDs are currently used in specific applications, due to the different heights, driving habits and visual habits of different people, the virtual image observed by the driver cannot be well integrated with the reality due to these differentiated influencing factors, resulting in poor virtual image display effects. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a display effect fusion method, system, storage medium and main control unit to improve the virtual image display effect of the automotive HUD.

[0006] In order to achieve the above purpose, the embodiments of the present application are implemented in the following manner:

[0007] In a first aspect, an embodiment of the present application provides a display effect fusion method, which is applied to a head-up display system, the method comprising: obtaining a driving image of a driving vehicle, wherein the driving image contains an object of interest, and the object of interest includes one or more of lane lines, road signs, other vehicles, and pedestrians; obtaining the eye position of a driver in the driving vehicle; and displaying a virtual image of the object in the virtual image display surface according to the eye position, the driving image, and a preset virtual image display surface, so that the virtual image of the object observed from the eye position coincides with the object of interest in an actual scene, wherein the virtual image of the object represents a virtual image of the object of interest displayed in the virtual image display surface, and the actual scene represents a real scene including the object of interest observed by the driver's eyes.

[0008] In the embodiment of the present application, by acquiring the driving image of the driving vehicle (including one or more objects of interest such as lane lines, road signs, other vehicles, pedestrians, etc.), and acquiring the eye position of the driver, based on the eye position and the driving image, the virtual image of the object (the virtual image of the object of interest displayed in the virtual image display surface) is displayed on the preset virtual image display surface, so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene (the actual scene including the object of interest observed by the driver's eyes). In this way, factors such as the height, driving habits and visual habits (eye position, line of sight, etc.) of different people can be taken into account, so that the virtual image can be well integrated with the real environment, and the virtual image offset (i.e., the offset between the virtual image seen by the driver and the actual scene) can be avoided as much as possible, and the display of the virtual image can be adjusted in real time following the change of the driver's eye position, so as to adapt to drivers of different heights, driving habits, and visual habits, ensure the virtual image display effect, and enhance the driving experience of the driver.

[0009] In combination with the first aspect, in a first possible implementation method of the first aspect, obtaining the eye position of the driver in the driving vehicle includes: obtaining a person image containing the driver in the driving vehicle; and determining the eye position of the driver's eyes in a world coordinate system based on the person image, wherein the world coordinate system is established based on the driving vehicle.

[0010] In this implementation, an image of a person including a driver in a driving vehicle is obtained, and the eye position of the driver's eyes in a world coordinate system (established based on the driving vehicle) is determined from the image. In this way, the eye position of the driver can be obtained in real time, which is conducive to the real-time adjustment of the virtual image display, so that the displayed virtual image of the object can be adjusted quickly when the driver's eye position changes, ensuring the virtual image display effect, avoiding the long-term misalignment of the virtual image of the object and the object of attention, which may bring negative effects to the driver, thereby helping to improve the user experience of the driver.

[0011] In combination with the first aspect, in a second possible implementation of the first aspect, the eye position is the position of the driver's eyes in a world coordinate system, and displaying a virtual image of an object on the virtual image display surface based on the eye position, the driving image and a preset virtual image display surface includes: determining the object position of the object of interest in the world coordinate system based on the driving image; determining a transformation matrix based on the eye position and the virtual image display surface, wherein the transformation matrix is ​​used to convert position coordinates in the world coordinate system into virtual image coordinates of the virtual image display surface; determining the object virtual image position of the object of interest on the virtual image display surface based on the object position and the transformation matrix; and displaying the object virtual image on the virtual image display surface based on the object virtual image position.

[0012] In this implementation, the object position of the object of interest in the world coordinate system is determined based on the driving image, and the conversion matrix determined based on the eye position and the virtual image display surface (used to convert the position coordinates in the world coordinate system into the virtual image coordinates of the virtual image display surface) is combined to determine the object virtual image position of the object of interest on the virtual image display surface, and the virtual image is displayed accordingly. This method can accurately determine the position and size of the object virtual image that should be displayed on the virtual image display surface, so that the virtual image object under the driver's perspective accurately overlaps with the object of interest, and can well achieve the fusion effect of virtual image and reality.

[0013] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, determining the transformation matrix based on the eye position and the virtual image display surface includes: determining a sight line vector of the eye position toward the virtual image display surface based on the eye position and the virtual image display surface; determining a view matrix based on the eye position and the sight line vector, wherein the view matrix is ​​used to correct the object position; obtaining a virtual image size and near-far interface parameters when displaying the virtual image, and determining a projection matrix based on the virtual image size and the near-far interface parameters; and determining the transformation matrix based on the view matrix and the projection matrix.

[0014] In this implementation, the sight line vector (i.e., the vector determined from the eye position toward the virtual image display surface) can be determined by the eye position and the virtual image display surface, and the view matrix can be further determined based on the eye position and the sight line vector. In this way, on the one hand, the accuracy of the view matrix can be guaranteed, which is conducive to the display fusion of the virtual image of the object and the object of interest. On the other hand, this method is simple, accurate and efficient, and can quickly determine the view matrix (used to correct the object position) so as to correct the object position of the object of interest. After the projection matrix is ​​determined according to the virtual image size and the near-far interface parameters, the conversion matrix is ​​determined according to the view matrix and the projection matrix, so that the conversion matrix can be quickly and accurately determined, thereby realizing the rapid adjustment of the virtual image display, which is conducive to ensuring the real-time display fusion.

[0015] In combination with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, determining the view matrix based on the eye position and the line of sight vector includes: determining an eye position matrix based on the eye position; determining an eye viewing angle matrix based on the line of sight vector; and determining the view matrix by left-multiplying the eye viewing angle matrix by the eye position matrix.

[0016] In this implementation, the view matrix can be determined quickly and accurately by left-multiplying the eye perspective matrix determined based on the sight line vector by the eye position matrix determined based on the eye position.

[0017] In combination with the third possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, determining the view matrix based on the eye position and the line of sight vector includes: obtaining a reference vector of the virtual image display surface, wherein the reference vector represents a vector perpendicular to the center of the virtual image display surface and toward the outside of the driving vehicle; and calculating the view matrix through OpenGL based on the eye position, the line of sight vector and the reference vector.

[0018] In this implementation, the view matrix can be quickly and efficiently calculated by OpenGL based on the eye position, the sight line vector and the reference vector (a vector perpendicular to the center of the virtual image display surface and facing the outside of the driving vehicle). In addition, by using OpenGL to calculate the transformation matrix, it is also beneficial to achieve GPU acceleration when displaying virtual images, improve the virtual image display speed, and thus improve the visual experience of the driver.

[0019] In combination with the second possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, after determining the object position of the object of interest in the world coordinate system based on the driving image, the method further includes: constructing an object three-dimensional model of the object of interest including the object position based on the object of interest and the object position; and after determining the transformation matrix based on the eye position and the virtual image display surface, it further includes: determining, based on the object three-dimensional model and the transformation matrix, a two-dimensional object graphic of the object three-dimensional model for display on the virtual image display surface; correspondingly, displaying the object virtual image on the virtual image display surface based on the object virtual image position includes: displaying the object virtual image on the virtual image display surface based on the object virtual image position and the two-dimensional object graphic.

[0020] In this implementation, by constructing a three-dimensional object model of the object of interest (including the object position), a two-dimensional object graphic of the three-dimensional object model can be determined in combination with the transformation matrix (for display on the virtual image display surface), and further, according to the virtual image position of the object and the two-dimensional object graphic, the virtual image of the object is displayed on the virtual image display surface. This method can, on the one hand, ensure the accuracy of the display of the virtual image of the object; on the other hand, it can ensure the consistency of the display effect of the virtual image of the object with the visual effect of the actual object of interest, thereby improving the fusion effect of the virtual image and reality.

[0021] In a second aspect, an embodiment of the present application provides a storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement a display effect fusion method as described in any one of the first aspect or possible implementation methods of the first aspect.

[0022] In a third aspect, an embodiment of the present application provides a main control unit, comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, a display effect fusion method as described in any one of the first aspect or possible implementation methods of the first aspect is implemented.

[0023] In a fourth aspect, an embodiment of the present application provides a head-up display system, comprising: a first camera mechanism, for capturing a driving image of a driving vehicle, wherein the driving image contains an object of interest, and the object of interest includes one or more of lane lines, road signs, other vehicles, and pedestrians; a second camera mechanism, for capturing a person image of a driver in the driving vehicle; a main control unit, for acquiring the driving image and the person image, and determining, based on the person image, an eye position of the driver's eyes in a world coordinate system, and, based on the eye position, the driving image, and a preset virtual image display surface, displaying a virtual image of the object in the virtual image display surface, so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene, wherein the world coordinate system is established based on the driving vehicle, the object virtual image represents a virtual image of the object of interest displayed on the virtual image display surface, and the actual scene represents a real scene including the object of interest observed by the driver's eyes.

[0024] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a head-up display system provided in an embodiment of the present application.

[0027] Figure 2 A flowchart of a display effect fusion method provided in an embodiment of the present application.

[0028] Figure 3 A process diagram of an exemplary display effect fusion method provided in an embodiment of the present application.

[0029] Figure 4 A structural block diagram of a main control unit provided in an embodiment of the present application.

[0030] Icons: 100 - head-up display system; 110 - first camera mechanism; 120 - second camera mechanism; 130 - main control unit; 131 - memory; 132 - communication module; 133 - bus; 134 - processor; 140 - virtual image display unit. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0032] See also Figure 1 , Figure 1 A schematic diagram of a head-up display system 100 provided in an embodiment of the present application.

[0033] In this embodiment, the head-up display system 100 may include a first camera mechanism 110, a second camera mechanism 120, a main control unit 130 and a virtual image display unit 140, and the first camera mechanism 110, the second camera mechanism 120, and the virtual image display unit 140 are respectively connected to the main control unit 130.

[0034] Exemplarily, the first camera mechanism 110 is used to capture driving images of the driving vehicle, which may contain objects of interest (including one or more of lane lines, road signs, other vehicles, and pedestrians) as the basis for the virtual image objects that the virtual image display unit 140 needs to display.

[0035] In order to obtain a more comprehensive driving image, the first camera mechanism 110 can be exemplarily arranged above the windshield of the driving vehicle, as close to the center of the windshield as possible (for example, just in the center), so as to obtain a more comprehensive driving image. Of course, it can also be arranged above the windshield and biased towards the position of the driver to minimize the difference between the driving image and the scene actually observed by the driver, which is conducive to ensuring the quality of the driving image.

[0036] In this embodiment, the first camera mechanism 110 may use a driving assistance module including a camera function to combine with the auxiliary unit of the driving vehicle as much as possible, thereby improving the compatibility of the head-up display system 100 and the driving vehicle. Exemplarily, the first camera mechanism 110 may use an advanced driving assistance system (ADAS), such as the ADAS of the Youjia X1 or other models. Of course, other driving assistance modules including a camera function may also be used, which is not limited here.

[0037] Exemplarily, the second camera mechanism 120 is used to capture the image of the driver in the vehicle. By acquiring the image of the driver through the second camera mechanism 120, the state of the driver can be acquired in real time, and the state of the driver can be taken into account during the virtual image display process, which is conducive to the head-up display system 100 to adjust the virtual image display in real time based on the state of the driver, thereby facilitating the display fusion effect (fusing the displayed virtual image with the actual scene).

[0038] Exemplarily, in order to better obtain the image of the driver in the driving vehicle, the second camera mechanism 120 can be set inside the vehicle, for example, suspended in the upper front of the interior of the driving vehicle, or set on the windshield, located in the upper front of the driver, so as to obtain the image of the driver as accurately as possible, which is beneficial to the real-time adjustment of the virtual image display by the head-up display system 100.

[0039] In order to better obtain the driver's status information (such as position, distance, etc.), the second camera mechanism 120 can use a depth camera device, such as a 3D structured light module, a binocular vision camera device, etc., which is not limited here. The personnel image obtained in this way also contains depth information, which is conducive to accurately perceiving the real-time status of the driver.

[0040] In this embodiment, the main control unit 130 is used to display the virtual image after corresponding processing based on the driving image and the personnel image, so that the displayed virtual image has a good fusion effect with the real scene, thereby improving the virtual image display quality of the head-up display system 100 and improving the user experience.

[0041] Exemplarily, the main control unit 130 can be an independent control unit (for example, a specially developed intelligent terminal, such as a controller designed with NXP development board; or some commonly used intelligent terminals such as smart phones, tablet computers, etc.; or, the main control unit 130 can also be a server. When the main control unit 130 is a server, the first camera mechanism 110 and the second camera mechanism 120 can be wirelessly connected to it), which is convenient for adapting to different vehicles and has very low requirements for vehicle configuration. Of course, in order to improve the compatibility of the head-up display system 100 with the vehicle and save costs, the main control unit 130 can also be an on-board computer on the vehicle to improve the compatibility of the head-up display system 100 with the vehicle as much as possible.

[0042] Exemplarily, the connection between the first camera mechanism 110 and the main control unit 130 can be achieved through CAN (Controller Area Network), which can ensure the communication quality and reliability. The requirements for the second camera mechanism 120 can be slightly lower, for example, USB (Universal Serial Bus) can be used to achieve communication with the main control unit 130, which is convenient for the use of the second camera mechanism 120 and can also ensure the communication quality; of course, CAN can also be used to achieve communication with the main control unit, or other communication methods, which are not limited here.

[0043] In this embodiment, in order to improve the virtual image display effect and display quality of the head-up display system 100, the head-up display system 100 can also be connected to other auxiliary function modules of the driving vehicle: for example, it can be connected to the vehicle controller (on-board computer, chassis control module, etc.) to obtain vehicle status parameters (such as vehicle speed, steering angle and other parameters) and vehicle driving environment parameters (such as vehicle distance), so that the head-up display system 100 can make real-time adjustments to the virtual image display, so that the displayed virtual image is better integrated with the real scene, thereby further improving the virtual image display effect.

[0044] Furthermore, in this embodiment, a HUD unit (i.e., virtual image display unit 140) may be used as a component for realizing virtual image display, and the HUD unit may be connected to the main control unit 130. The main control unit 130 may receive the driving image sent by the first camera mechanism 110 and the person image sent by the second camera mechanism 120, and determine the virtual image to be displayed based on the image, so that the HUD unit displays the virtual image.

[0045] For example, in order to facilitate the driver's observation, the HUD unit can be set under the windshield of the vehicle body, so that the virtual image display surface determined by the HUD unit is within the driver's line of sight, on a surface at a certain distance (for example, 1 to 3 meters) outside the windshield of the driving vehicle, which is convenient for the driver to observe.

[0046] The above is an introduction to the head-up display system 100 provided in an embodiment of the present application. In order to ensure the display effect of the virtual image, the embodiment of the present application also provides a display effect fusion method, which can be executed by the main control unit 130 (the method can also be executed by the entire head-up display system 100, which is not limited here) to merge the displayed virtual image with the real scene, thereby improving the display effect of the head-up display system 100.

[0047] See also Figure 2 , Figure 2 This is a flow chart of a display effect fusion method provided in an embodiment of the present application. The display effect fusion method may include step S10, step S20, and step S30.

[0048] Before the main control unit executes step S10, the first camera unit may capture a driving image of the driving vehicle, and the driving image may contain objects of interest (including one or more of lane lines, road signs, other vehicles, and pedestrians). Of course, other objects may be selected as objects of interest as needed, such as road boundaries, road slopes, etc., which are not limited here. In addition, the second camera unit may capture an image of a driver in the driving vehicle.

[0049] After the first camera mechanism (ie, the first camera mechanism 110 ) captures the vehicle image and the second camera mechanism (ie, the second camera mechanism 120 ) captures the person image, the main control unit may execute step S10 and step S20 .

[0050] Step S10: Acquire a driving image of the driving vehicle, wherein the driving image contains objects of interest, and the objects of interest include one or more of lane lines, road signs, other vehicles, and pedestrians.

[0051] Step S20: Acquire the eye position of the driver in the driving vehicle.

[0052] It should be noted that the execution order of step S10 and step S20 is not limited, and step S10 can be executed first and then step S20, or step S20 can be executed first and then step S10, or step S10 and step S20 can be executed at the same time, which is not limited here. Step S10 can be executed after the first camera unit captures the driving image, and step S20 can be executed after the second camera unit captures the person image.

[0053] Of course, in order to ensure the fusion effect, the shooting time of the driving image and the personnel image can be consistent, and the scene changes caused by time inconsistency can be avoided as much as possible, so as to ensure the fusion effect of the displayed virtual image and the actual scene as much as possible.

[0054] Here, step S10 is introduced first. The main control unit can obtain the driving image, for example, by receiving the driving image sent after being shot by the first camera mechanism, or of course, the driving image can also be received from other devices. It only needs to ensure that the driving image is the driving image shot by the first camera mechanism at a time.

[0055] For the execution of step S20, the main control unit may obtain a person image and determine the eye position of the driver according to the person image.

[0056] Exemplarily, the main control unit may receive a person image captured and sent by the second camera mechanism, wherein the person image includes the driver. Then, the main control unit may determine the eye position of the driver in a world coordinate system (a three-dimensional space coordinate system established based on the driving vehicle, so that the driver, the first camera mechanism, the second camera mechanism, the virtual image display surface, etc., can all reveal their positional relationship with a certain point in the driving vehicle through the world coordinate system) based on the person image.

[0057] For example, the position of the driver's eyes in the image can be determined by extracting facial features from the person image, and then combined with the position of the second camera mechanism in the world coordinate system, the eye position of the driver's eyes in the world coordinate system can be determined.

[0058] In addition, the acquisition of the driver's eye position is not limited to the method provided above, and the eye position can also be sent by other devices. For example, there is a dedicated eye position determination device for obtaining the driver's eye position in real time and sending the real-time eye position to the main control unit. Therefore, this should not be regarded as a limitation of the present application.

[0059] By acquiring a person image including the driver in the driving vehicle, and determining the eye position of the driver's eyes in the world coordinate system (established based on the driving vehicle), the driver's eye position can be acquired in real time, which is conducive to the real-time adjustment of the virtual image display, so that the displayed object virtual image can be quickly adjusted when the driver's eye position changes, ensuring the virtual image display effect, avoiding the long-term misalignment of the object virtual image and the object of attention, which brings negative effects to the driver, and thus helps to improve the driver's user experience.

[0060] After acquiring the driving image and the eye position, the main control unit may execute step S30.

[0061] Step S30: Based on the eye position, the driving image and the preset virtual image display surface, a virtual image of the object is displayed on the virtual image display surface, so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene, wherein the virtual image of the object represents a virtual image of the object of interest displayed on the virtual image display surface, and the actual scene represents a real scene including the object of interest observed by the driver's eyes.

[0062] In this embodiment, the main control unit can display the virtual image of the object (i.e., the virtual image of the object of interest displayed in the virtual image display surface) in the virtual image display surface according to the eye position, the driving image and the preset virtual image display surface (i.e., a surface used to display the virtual image), so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene (i.e., the real scene including the object of interest observed by the driver's eyes).

[0063] In this way, factors such as different people's height, driving habits and visual habits (eye position, line of sight, etc.) can be taken into account, so that the virtual image can be well integrated with the real environment, and virtual image offset (that is, the offset between the virtual image seen by the driver and the actual scene) can be avoided as much as possible. The display of the virtual image can be adjusted in real time as the driver's eye position changes, so that it can adapt to drivers of different heights, driving habits, and visual habits, ensure the virtual image display effect, and enhance the driver's driving experience.

[0064] Exemplarily, the main control unit may determine the object position of the object of interest in the world coordinate system based on the driving image.

[0065] For example, by determining the position of a certain (or multiple) calibration point (e.g., one or more fixed points on the front of the vehicle) in the image, the position of the fixed point is fixed relative to the position of the first camera mechanism, and therefore, the position of the calibration point in the image is also fixed, so that the position information of other pixels in the image can be determined, thereby determining the position of the object of interest in the image. Then, the main control unit can convert the position of the object of interest in the image into the position in the world coordinate system in combination with the position of the first camera mechanism in the world coordinate system, thereby determining the object position of the object of interest in the world coordinate system.

[0066] Of course, the position of the object of interest in the world coordinate system can also be determined in other ways. For example, when the driving image is a depth image, the distance information of the object of interest can be determined, thereby determining the object position of the object of interest in the world coordinate system in combination with the position of the first camera mechanism in the world coordinate system. For another example, the main control unit can also obtain the distance information obtained by the distance sensor set on the vehicle body to determine the distance of the object of interest, thereby determining the object position of the object of interest in the world coordinate system in combination with the position of the first camera mechanism in the world coordinate system.

[0067] Therefore, the manner of determining the object position of the object of interest in the world coordinate system should not be regarded as limiting the present application. However, through these manners, the object position of the object of interest in the world coordinate system can be accurately determined.

[0068] After determining the object position of the object of interest in the world coordinate system, the main control unit can construct an object three-dimensional model of the object of interest including the object position according to the object of interest and the object position. Of course, the method of constructing the object three-dimensional model of the object of interest here can be to extract the contour features of the object of interest to construct the corresponding object three-dimensional model. It can also be to identify the object of interest, determine the corresponding model from the preset three-dimensional model, and then match the corresponding model with the size, angle, etc. of the object of interest in the image (of course, it can also not match, but convert through a subsequent transformation matrix, which is not limited here), so as to obtain the three-dimensional model of the object of interest. Therefore, the method of constructing the object three-dimensional model here should not be regarded as a limitation of the present application.

[0069] By constructing a three-dimensional model of the object of interest in the above manner, the appearance of the object in the driving image can be well restored, ensuring the effect of virtual image display.

[0070] And, the main control unit can determine a conversion matrix according to the eye position and the virtual image display surface. Here, the conversion matrix is ​​used to convert the position coordinates in the world coordinate system into the virtual image coordinates of the virtual image display surface. Of course, in some other possible ways, the parameters used to convert the position coordinates in the world coordinate system into the virtual image coordinates of the virtual image display surface may not be a matrix, for example, a conversion function, therefore, the conversion matrix is ​​taken as an example in this embodiment, but it should not be regarded as a limitation of the present application.

[0071] Exemplarily, the main control unit can determine the sight line vector of the eye position toward the virtual image display surface according to the eye position and the virtual image display surface. For example, the sight line vector can be a vector between the eye position and the center position of the virtual image display surface in the world coordinate system, and the direction is between the eye position and the center position of the virtual image display surface.

[0072] Then, the main control unit may determine a view matrix (for correcting the position of the object) according to the eye position and the sight line vector.

[0073] As for the specific method of determining the view matrix, two exemplary methods will be provided in this embodiment, but they should not be regarded as limitations on the present application.

[0074] The first method of determining the view matrix: the main control unit can determine the eye position matrix according to the eye position, and determine the eye viewing angle matrix according to the line of sight vector; and then multiply the eye viewing angle matrix by the eye position matrix to determine the view matrix.

[0075] For example, if the eye position is (x, y, z), then the eye position matrix T can be:

[0076]

[0077] The parameters of the sight vector are (x1, y1, z1), then the eye perspective matrix R can be:

[0078]

[0079] Then, the view matrix can be determined: C=RT. Thus, the view matrix C can be determined.

[0080] By left-multiplying the eye perspective matrix determined based on the sight line vector by the eye position matrix determined based on the eye position, the view matrix can be determined quickly and accurately. It should be noted that the order of determining the eye position matrix and the eye perspective matrix is ​​not limited.

[0081] The second way to determine the view matrix: the main control unit can obtain the reference vector of the virtual image display surface (that is, the vector perpendicular to the center of the virtual image display surface and pointing to the outside of the driving vehicle, which can also be understood as the normal vector of the virtual image display surface), and calculate the view matrix through OpenGL based on the eye position, line of sight vector and reference vector.

[0082] For example, eye position: cameraPos = QVector3D(x, y, z).

[0083] View vector: cameraFront = QVector3D(x1, y1, z1).

[0084] Reference vector: cameraUp = QVector3D(0, 1, 0).

[0085] Then, the main control unit can calculate the view matrix through OpenGL:

[0086] lookAt(cameraPos, cameraPos+cameraFront, cameraUp).

[0087] Thus, the main control unit can calculate the view matrix lookAt through OpenGL.

[0088] Through OpenGL, the view matrix can be calculated quickly and efficiently based on the eye position, the sight vector and the reference vector (a vector perpendicular to the center of the virtual image display surface and facing the outside of the driving vehicle). In addition, by using OpenGL to calculate the transformation matrix, it is also beneficial to achieve GPU acceleration when displaying virtual images, improve the virtual image display speed, and thus improve the visual experience of the driver.

[0089] After determining the view matrix, the main control unit can obtain the virtual image size (i.e., the height, width and other parameters of the virtual image to be displayed, the distance of the virtual image display surface, etc.) and the near-far interface parameters when displaying the virtual image (i.e., the near-far distance of the virtual image of the object to be displayed in the virtual image display interface), and determine the projection matrix based on the virtual image size and the near-far interface parameters.

[0090] The determination of the corresponding view matrix is ​​also introduced here using two methods as examples.

[0091] Exemplarily, corresponding to the first method of determining the view matrix in the previous text, the first method of determining the projection matrix here is:

[0092] The main control unit can obtain the virtual image size: virtual image width l and virtual image height t; and obtain the near-far interface parameters: virtual image plane distance n, farthest distance f. Based on the virtual image width l and virtual image height t, as well as the virtual image plane distance n and farthest distance f, the projection matrix P is calculated:

[0093]

[0094] In this way, the projection matrix can be accurately calculated.

[0095] Exemplarily, corresponding to the second method of determining the view matrix in the previous text, the second method of determining the projection matrix here is:

[0096] The main control unit can obtain the field of view angle parameters (i.e. the field of view angle of the first camera mechanism, which is also called the field of view in optical engineering, and the size of the field of view angle determines the field of view of the optical instrument), the virtual image size (i.e. the height, width and other parameters of the virtual image to be displayed) and the near-far interface parameters when displaying the virtual image (i.e. the near and far distances of the virtual image of the object to be displayed in the virtual image display interface), and determine the projection matrix based on the field of view angle parameters, virtual image size and near-far interface parameters.

[0097] For example, the field of view parameter: fov (the parameter can be expressed as aspect).

[0098] Virtual image size: width (virtual image width) and height (virtual image height).

[0099] Near and far interface parameters: including n (near), f (far).

[0100] Then, the main control unit can calculate the projection matrix through OpenGL:

[0101] projectMat.perspective(aspect, width() / height(), n, f).

[0102] Therefore, the main control unit can calculate the projection matrix project through OpenGL.

[0103] By using OpenGL, the projection matrix can be determined quickly and efficiently based on the field of view parameters, virtual image size, and near-far interface parameters. In addition, by using OpenGL to calculate the conversion matrix, it is also beneficial to achieve GPU acceleration when displaying virtual images, improve the virtual image display speed, and thus enhance the visual experience of the driver.

[0104] It should be noted that, in this embodiment, the view matrix is ​​calculated first and the projection matrix is ​​calculated later, but this should not be regarded as a limitation of the present application. In practical applications, the order of determining the view matrix and the projection matrix is ​​not limited. Moreover, the method for calculating the view matrix and the projection matrix is ​​not specifically limited, and can be selected according to actual needs. For example, the view matrix is ​​calculated in the first method and the projection matrix is ​​calculated in the second method.

[0105] After determining the view matrix and the projection matrix, the main control unit can determine the conversion matrix according to the view matrix and the projection matrix. Exemplarily, the main control unit can obtain the conversion matrix by multiplying the projection matrix by the view matrix: X=CP. In this way, the conversion matrix can be calculated efficiently and accurately.

[0106] After the conversion matrix is ​​determined, the main control unit can determine the virtual image position of the object of interest on the virtual image display surface according to the object position and the conversion matrix.

[0107] Exemplarily, the main control unit can determine the object 3D model for displaying a 2D object graphic on the virtual image display surface according to the object 3D model and the conversion matrix. That is, the object 3D model can be converted into a 2D object graphic through the conversion matrix.

[0108] Then, the main control unit can display the object virtual image in the virtual image display surface according to the position of the object virtual image. Exemplarily, the main control unit can display the object virtual image in the virtual image display surface according to the position of the object virtual image and the two-dimensional object graphic.

[0109] By determining the object position of the object of interest in the world coordinate system based on the driving image, combined with the conversion matrix determined based on the eye position and the virtual image display surface, the object virtual image position of the object of interest on the virtual image display surface is determined, and the virtual image is displayed accordingly. This method can accurately determine the position and size of the object virtual image that should be displayed on the virtual image display surface, so that the virtual image object under the driver's perspective accurately overlaps with the object of interest, and can well achieve the fusion effect of virtual image and reality. By constructing an object three-dimensional model of the object of interest, the two-dimensional object graphic of the object three-dimensional model can be determined in combination with the conversion matrix, and the object virtual image is further displayed on the virtual image display surface according to the object virtual image position and the two-dimensional object graphic. This method can, on the one hand, ensure the accuracy of the object virtual image display; on the other hand, it can ensure the consistency of the display effect of the object virtual image with the visual effect of the actual object of interest, thereby improving the fusion effect of virtual image and reality.

[0110] The above is an introduction to the display effect fusion method provided in the embodiment of the present application. Here, the application process of the method will be briefly explained in combination with a specific example.

[0111] See also Figure 3 , Figure 3 A process diagram of an exemplary display effect fusion method provided in an embodiment of the present application.

[0112] Exemplarily, on the one hand, ADAS (i.e., the first camera mechanism) can take driving images and send them to the main control unit via CAN (i.e., convert them into CAN data). The main control unit can parse the CAN data and build a three-dimensional model of the object of interest (e.g., a three-dimensional model of lane lines, a three-dimensional model of vehicles, a three-dimensional model of pedestrians, etc.). On the other hand, the depth camera (i.e., the second camera mechanism) can take images of people (including drivers), and then the main control unit can process the images of people (e.g., human eye calibration) to determine the corresponding conversion parameters (i.e., conversion matrix). Through the conversion parameters, the main control unit can perform virtual image conversion on the object three-dimensional model, thereby determining the corresponding object virtual image (e.g., lane line virtual image, vehicle virtual image, pedestrian virtual image, etc.), and then realize the display of the virtual image. The virtual image display achieved in this way can overlap with the actual scene observed by the driver in reality, thereby improving the virtual image display effect and improving the driver's experience.

[0113] See also Figure 4 , the embodiment of the present application also provides a main control unit, Figure 4 A structural block diagram of a main control unit 130 provided in an embodiment of the present application.

[0114] Exemplarily, the main control unit 130 may include: a communication module 132 connected to the outside world through a network, one or more processors 134 for executing program instructions, a bus 133, and different forms of memory 131, such as a disk, ROM (Read-Only Memory), RAM (Random Access Memory), or any combination thereof. Among them, the memory 131, the communication module 132 and the processor 134 are connected through the bus 133.

[0115] Exemplarily, programs are stored in the memory 131. The processor 134 can call and run these programs from the memory 131, so that the display effect fusion method can be executed by running the programs to improve the display effect of the virtual image.

[0116] An embodiment of the present application also provides a storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the display effect fusion method described in this embodiment.

[0117] In summary, the embodiments of the present application provide a display effect fusion method, system, storage medium and main control unit, which obtains the driving image of the driving vehicle (including one or more objects of interest such as lane lines, road signs, other vehicles, pedestrians, etc.), and obtains the eye position of the driver, so as to display the virtual image of the object (the virtual image of the object of interest displayed in the virtual image display surface) on the preset virtual image display surface based on the eye position and the driving image, so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene (the actual scene including the object of interest observed by the driver's eyes). In this way, factors such as the height, driving habits and visual habits (eye position, line of sight, etc.) of different people can be taken into account, so that the virtual image can be well integrated with the real environment, and the virtual image offset (i.e., the offset between the virtual image seen by the driver and the actual scene) can be avoided as much as possible, and the display of the virtual image can be adjusted in real time following the change of the eye position of the driver, so as to adapt to drivers of different heights, driving habits, and visual habits, ensure the virtual image display effect, and enhance the driving experience of the driver.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed methods can be implemented in other ways, and the device embodiments described above are merely illustrative.

[0119] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0120] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A display effect fusion method, characterized in that: Applied to a head-up display system, the method comprises: Acquire a driving image of a driving vehicle, wherein the driving image contains an object of interest, and the object of interest includes one or more of a lane line, a road sign, another vehicle, and a pedestrian; Obtaining the eye position of the driver in the driving vehicle; According to the eye position, the driving image and the preset virtual image display surface, a virtual image of an object is displayed on the virtual image display surface, so that the virtual image of the object observed from the eye position coincides with the object of interest in the actual scene, wherein the virtual image of the object represents a virtual image of the object of interest displayed on the virtual image display surface, and the actual scene represents a real scene including the object of interest observed by the eyes of the driver; The eye position is the position of the driver's eyes in the world coordinate system, and the displaying of the object virtual image on the virtual image display surface according to the eye position, the driving image and a preset virtual image display surface comprises: determining the object position of the object of interest in the world coordinate system according to the driving image; determining a conversion matrix according to the eye position and the virtual image display surface, wherein the conversion matrix is ​​used to convert the position coordinates in the world coordinate system into the virtual image coordinates of the virtual image display surface; determining the object virtual image position of the object of interest on the virtual image display surface according to the object position and the conversion matrix; and displaying the object virtual image on the virtual image display surface according to the object virtual image position; The method of determining a conversion matrix according to the eye position and the virtual image display surface includes: determining a sight line vector of the eye position toward the virtual image display surface according to the eye position and the virtual image display surface; determining a view matrix according to the eye position and the sight line vector, wherein the view matrix is ​​used to correct the object position; obtaining a virtual image size and near-far interface parameters when displaying the virtual image, and determining a projection matrix according to the virtual image size and the near-far interface parameters; and determining the conversion matrix according to the view matrix and the projection matrix.

2. The display effect fusion method according to claim 1, characterized in that: The obtaining of the eye position of the driver in the driving vehicle comprises: Acquiring a person image including the driver in the driving vehicle; The eye positions of the driver's eyes in a world coordinate system are determined according to the person image, wherein the world coordinate system is established based on the driving vehicle.

3. The display effect fusion method according to claim 1, characterized in that: The determining the view matrix according to the eye position and the sight line vector comprises: According to the eye position, determining an eye position matrix; Determining an eye viewing angle matrix according to the sight line vector; The eye viewing angle matrix is ​​multiplied by the eye position matrix to determine the view matrix.

4. The display effect fusion method according to claim 1, characterized in that: The determining the view matrix according to the eye position and the sight line vector comprises: Acquire a reference vector of the virtual image display surface, wherein the reference vector represents a vector perpendicular to the center of the virtual image display surface and directed toward the outside of the driving vehicle; The view matrix is ​​calculated by OpenGL according to the eye position, the sight line vector and the reference vector.

5. The display effect fusion method according to claim 1, characterized in that: After determining the object position of the object of interest in the world coordinate system according to the driving image, the method further includes: Constructing a three-dimensional object model of the object of interest including the object position according to the object of interest and the object position; And, after determining the conversion matrix according to the eye position and the virtual image display surface, the method further includes: Determining, according to the object three-dimensional model and the conversion matrix, a two-dimensional object graphic of the object three-dimensional model for display on the virtual image display surface; Correspondingly, displaying the virtual image of the object on the virtual image display surface according to the position of the virtual image of the object includes: The object virtual image is displayed on the virtual image display surface according to the object virtual image position and the two-dimensional object pattern.

6. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the display effect fusion method as described in any one of claims 1 to 5.

7. A main control unit, characterized in that: It comprises a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, the display effect fusion method as described in any one of claims 1 to 5 is implemented.

8. A head-up display system, characterized in that: include: A first camera mechanism is used to capture a driving image of a driving vehicle, wherein the driving image contains an object of interest, and the object of interest includes one or more of a lane line, a road sign, other vehicles, and a pedestrian; The second camera mechanism is used to capture an image of a driver in the vehicle; a main control unit, configured to acquire the driving image and the person image, and determine the eye position of the driver's eyes in a world coordinate system according to the person image, and display an object virtual image in the virtual image display surface according to the eye position, the driving image and a preset virtual image display surface, so that the object virtual image observed from the eye position coincides with the object of interest in the actual scene, wherein the world coordinate system is established based on the driving vehicle, the object virtual image represents a virtual image of the object of interest displayed in the virtual image display surface, and the actual scene represents a real scene including the object of interest observed by the driver's eyes; Wherein, the eye position is the position of the driver's eyes in the world coordinate system, and the displaying of the object virtual image on the virtual image display surface according to the eye position, the driving image and the preset virtual image display surface comprises: determining the object position of the object of interest in the world coordinate system according to the driving image; determining a conversion matrix according to the eye position and the virtual image display surface, wherein the conversion matrix is ​​used to convert the position coordinates in the world coordinate system into the virtual image coordinates of the virtual image display surface; determining the object virtual image position of the object of interest on the virtual image display surface according to the object position and the conversion matrix; and displaying the object virtual image on the virtual image display surface according to the object virtual image position; The method of determining a conversion matrix according to the eye position and the virtual image display surface includes: determining a sight line vector of the eye position toward the virtual image display surface according to the eye position and the virtual image display surface; determining a view matrix according to the eye position and the sight line vector, wherein the view matrix is ​​used to correct the object position; obtaining a virtual image size and near-far interface parameters when displaying the virtual image, and determining a projection matrix according to the virtual image size and the near-far interface parameters; and determining the conversion matrix according to the view matrix and the projection matrix.

Images