Display method, calibration method, display device, storage medium and vehicle

Abstract

The invention relates to the technical field of projection display, in particular to a display method, a calibration method, display equipment, a storage medium and a vehicle. According to the application, virtual and real fitting is realized by utilizing the first display element and the first object outside the vehicle windshield, the position of the first object to be fitted outside the vehicle windshield is converted into the pre-projection position coordinate corresponding to the image plane, and the pre-projection position is located on the same straight line with the human eyes and the first object; and according to a mapping relation between a pre-calibrated image surface and an image source display surface, the pre-projection position coordinate is inversely mapped to the image source display surface, and a first display element is displayed at a pixel point position determined by inverse mapping. The virtual and real fitting effect of the HUD display equipment can be improved, and the watching experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of projection display technology, and in particular to a display method, a calibration method, a display device, a storage medium and a vehicle. Background Art

[0002] HUD (Head Up Display) is a new way to realize in-vehicle display by reflection on the vehicle windshield. Specifically, the display light is emitted by the optical machine of the HUD display device, and is projected on the windshield through the corresponding optical lens to produce a corresponding virtual image, which forms an enhanced display effect with the real world outside the windshield. With the rise of AR-HUD, the requirements for enhanced display in projection are also constantly increasing, but there is often a deviation between the projected virtual image on the windshield and the real scene outside the windshield. This deviation is mainly due to the production and installation errors of the HUD display device, which causes the position or depth of the elements corresponding to the projected virtual image on the image plane to not meet the design value. If the projection is not relocated, the user's viewing experience will be greatly reduced. Summary of the invention

[0003] The purpose of the present application is to provide a display method, a calibration method, a display device, a storage medium and a vehicle, which solve the technical problem in the prior art that the virtual-real fit of the HUD display device is poor, affecting the user's viewing experience.

[0004] In order to solve the above technical problems, this application adopts the following technical solutions.

[0005] In a first aspect, the present application provides a display method, comprising: Obtaining a position of a first object outside a vehicle windshield; The HUD display device configures a first display element according to an image plane formed by projecting the vehicle windshield, wherein the first display element is projected to be aligned with the first object when viewed by human eyes; Determine, according to the position of the human eye, a first intersection position of the first object corresponding to the first object having a line of sight connection relationship on the image plane; According to the mapping relationship between the image plane and the image source display plane, it is determined that the first intersection position corresponds to a first pixel point position on the image source display plane, and the first display element is configured to be displayed at the first pixel point position.

[0006] In an optional implementation of the first aspect, the image plane is planar.

[0007] In an optional implementation of the first aspect, the image plane is a curved surface.

[0008] According to the above description, an optional implementation method utilizes the mapping relationship between the image plane and the image source display plane to directly convert the spatial scale of the actual virtual image display determined based on the line of sight connection relationship into the plane scale on the image source display plane, thereby improving the accuracy of the display element fitting configuration.

[0009] In an optional implementation of the first aspect, acquiring the position of the first object outside the vehicle windshield includes: The position of the first object is determined by at least one of a millimeter wave radar, a laser radar, and a binocular camera.

[0010] In an optional implementation of the first aspect, the position of the human eye is determined by a camera or a driver monitoring system in the vehicle.

[0011] According to the above description, an optional implementation method determines the position of the human eye and the object to be bonded through sensing equipment inside and outside the vehicle, so that the direction of the human eye's line of sight in space when viewing the object outside the vehicle can be accurately determined.

[0012] In an optional implementation of the first aspect, determining the first intersection position of the first object corresponding to the first intersection position having a line of sight connection relationship on the image plane according to the position of the human eye includes: The first intersection position is determined according to an image plane fitting set corresponding to the image plane.

[0013] In an optional implementation of the first aspect, the image plane fitting set is determined by the spatial distribution of pre-calibrated detection control points.

[0014] In an optional implementation of the first aspect, the image plane fitting set is an image plane equation.

[0015] In an optional implementation of the first aspect, determining the first intersection position of the first object corresponding to the first intersection position having a line of sight connection relationship on the image plane according to the position of the human eye includes: The first intersection position is determined according to a straight line equation and an image plane equation between the human eye and the first object.

[0016] According to the above description, an optional implementation method uses the actual spatial relationship determined by optics to accurately calculate the first intersection position. The first intersection position serves as a pre-projection position to ensure fit, which can provide a reliable anchor point for the display of display elements on the image source display surface, while also reducing the difficulty of calibration and improving the accuracy of calibration.

[0017] In an optional implementation of the first aspect, determining, according to a mapping relationship between the image plane and the image source display surface, that the first intersection position corresponds to a first pixel point position on the image source display surface includes: The position of the first intersection point is represented by the three-dimensional coordinates of the space where the image plane is located; The position of the first pixel point is represented by the two-dimensional coordinates of the plane where the image source display surface is located.

[0018] According to the above description, an optional implementation manner establishes a direct mapping relationship between spatial coordinates and plane coordinates to improve the convenience of conversion between them.

[0019] In an optional implementation of the first aspect, determining, according to a mapping relationship between the image plane and the image source display surface, that the first intersection position corresponds to a first pixel point position on the image source display surface includes: The image plane is a plane, and the position of the first intersection point is represented by two-dimensional coordinates of the plane where the image plane is located; The position of the first pixel point is represented by the two-dimensional coordinates of the plane where the image source display surface is located.

[0020] According to the above description, an optional implementation manner may perform dimensionality reduction processing on the spatial coordinates of the virtual image projection points on the image plane to ensure direct correspondence with the image source display surface.

[0021] In an optional implementation manner of the first aspect, the first display element is configured to display at the first pixel point position including: Determining a scaling factor according to a mapping relationship between the image plane and the image source display plane; The size of the first display element on the image source display surface is adjusted according to the scaling factor.

[0022] According to the above description, the optional implementation method allows the size of the first display element projected on the image plane to remain the same despite the actual deviation, thereby improving the coordination of the overall viewing.

[0023] In an optional implementation of the first aspect, determining, according to a mapping relationship between the image plane and the image source display plane, that the first intersection position corresponds to a first pixel point position on the image source display plane includes: The image plane fitting set and the mapping relationship between the image plane and the image source display plane are determined according to the viewpoint position of the human eye in the eye box of the HUD display device.

[0024] According to the above description, the optional implementation manner can adapt to the viewing effect of the human eye at different viewpoints, thereby effectively dealing with different optical deviation problems at different viewpoints.

[0025] In a second aspect, the present application provides a calibration method, comprising: Controlling the HUD display device to output reference control points according to pixel distribution on the image source display surface, so that there are detection control points corresponding to the reference control points in the space where the image surface formed by the projection vehicle windshield is located; Scanning and acquiring the positions of the detection control points by an optical sensor device to determine an image plane fitting set representing the spatial distribution of the detection control points; A mapping relationship between the image plane and the image source display plane is determined according to the corresponding relationship between the reference control points and the detection control points and the image plane fitting set.

[0026] In an optional implementation of the second aspect, the reference control points are regularly arranged dot matrix images.

[0027] In an optional implementation of the second aspect, the reference control points are a regularly arranged checkerboard image.

[0028] According to the above description, an optional implementation method can simulate the projection distribution of the HUD display device in the space where the image plane is located by referring to the control points, thereby determining the spatial characteristics of the image plane and finding the mapping relationship before and after the actual projection, which is used as the processing parameter called during the actual projection display, thereby reducing the deviation between the design value and the actual value.

[0029] In an optional implementation of the second aspect, scanning and acquiring the position of the detection control point by an optical sensing device includes: The three-dimensional coordinates of the detection control point in the space where the image plane is located are obtained through a binocular camera.

[0030] According to the above description, an optional implementation manner may collect the actual positions of the detection control points in the space where the image plane is located, thereby providing a data basis for constructing an image plane fitting set.

[0031] In an optional implementation of the second aspect, after acquiring the three-dimensional coordinates of the detection control points in space through a binocular camera, determining an image plane fitting set representing the spatial distribution of the detection control points includes: The image plane equation is fitted according to the three-dimensional coordinates of the detected control points.

[0032] According to the above description, an optional implementation method determines a generalized image plane equation, so that any virtual image projection point position can be expressed, thereby being able to adapt to the first intersection position determined by different human eye and object positions.

[0033] In an optional implementation of the second aspect, fitting the image plane equation according to the three-dimensional coordinates of the detected control point includes: The abnormal three-dimensional coordinates of the detection control points are eliminated.

[0034] According to the above description, an optional implementation manner removes abnormal detection control points that are obviously beyond the fluctuation range, thereby improving the accuracy of image plane equation fitting.

[0035] In an optional implementation of the second aspect, the correspondence between the reference control points and the detection control points includes: A relationship is established between the three-dimensional coordinates of the detection control point and the two-dimensional coordinates of the reference control point.

[0036] According to the above description, an optional implementation establishes a connection between the spatial scale and the plane scale to determine the mapping relationship between the image plane and the image source display plane, thereby providing parameters for inverse mapping the corresponding virtual image projection points in space to the image source display plane.

[0037] In an optional implementation of the second aspect, the correspondence between the reference control points and the detection control points includes: A relationship is established between the two-dimensional coordinates of the detected control point and the two-dimensional coordinates of the reference control point.

[0038] In an optional implementation of the second aspect, the correspondence between the reference control points and the detection control points includes: When the image plane is a plane, the three-dimensional coordinates corresponding to the detection control points are converted into two-dimensional coordinates of the plane where the image plane is located.

[0039] According to the above description, an optional implementation manner replaces the spatial scale of the detection control point with a two-dimensional representation on the image plane, thereby establishing a direct mapping relationship between the image plane and the image source display plane.

[0040] In an optional implementation of the second aspect, when the image plane is a plane, converting the three-dimensional coordinates corresponding to the detection control points into two-dimensional coordinates of the plane where the image plane is located includes: The image plane is a standard plane defined near the detection control point, and the two-dimensional coordinates of the detection control point are determined by mapping the detection control point onto the standard plane according to the position of the optical sensor device.

[0041] According to the above description, an optional implementation manner normalizes the actual surface spatial distribution of the detection control points onto a virtual standard plane, thereby facilitating the dimensionality reduction processing of the detection control points.

[0042] In an optional implementation manner of the second aspect, scanning and acquiring the positions of the detection control points by an optical sensing device to determine an image plane fitting set representing the spatial distribution of the detection control points includes: Using a camera in a first posture to photograph the detection control point to acquire a first image; Using a camera in a second position to photograph the detected control points to acquire a second image; An image plane equation including the detection control points is determined according to the first image and the second image.

[0043] In an optional implementation manner of the second aspect, determining the image plane equation including the detection control point according to the first image and the second image includes: The coordinates of any virtual image projection point on the image plane are determined according to the image plane equation.

[0044] According to the above description, an optional implementation method directly determines the image plane equation based on the spatial transformation relationship of the camera shooting content under different postures, avoiding the complexity of directly measuring the three-dimensional coordinates of the detection control point using an optical sensor device such as a camera.

[0045] In an optional implementation of the second aspect, the calibration method further includes: In response to different viewpoint positions of the optical sensing device in the eye box of the HUD display device, the image plane fitting set and the mapping relationship between the image plane and the image source display plane are determined respectively.

[0046] According to the above description, in an optional implementation manner, the optical sensing device simulates different viewpoint positions of the human eye in the eye box of the HUD display device to determine calibration data such as the image plane fitting set under different optical deviations, thereby improving the accuracy of determining the fitting relationship during projection display.

[0047] In a third aspect, the present application provides a display device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the display method described in the first aspect or the calibration method described in the second aspect when executing the computer program.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display method described in the first aspect or the calibration method described in the second aspect are implemented.

[0049] In a fifth aspect, the present application provides a vehicle comprising the display device described in the third aspect or the computer-readable storage medium described in the fourth aspect.

[0050] Compared with the prior art, the present application uses the first display element to achieve virtual-real bonding with the first object outside the vehicle windshield, which converts the position of the first object to be bonded outside the vehicle windshield into the pre-projection position coordinates corresponding to the image plane, and the pre-projection position is in the same straight line as the human eye and the first object, and then the pre-projection position coordinates are inversely mapped to the image source display surface according to the mapping relationship between the image plane and the image source display surface that has been calibrated in advance, and the first display element is displayed at the pixel point position determined by the inverse mapping. The present application can improve the virtual-real bonding effect of the HUD display device and enhance the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the description of the technical solution. Obviously, the drawings described below are only some examples recorded in the present application, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0052] Figure 1 Schematic diagram of HUD projection display in some examples of this application.

[0053] Figure 2 Schematic diagram of the virtual-real fit deviation of HUD projection in some examples of this application.

[0054] Figure 3 This is a schematic diagram of the distribution of detection control points formed by projection in some examples of this application.

[0055] Figure 4 Schematic diagram of reference control points for calibrating image sources in some examples of this application.

[0056] Figure 5 Schematic diagram of a checkerboard image displayed for calibrating image sources in some examples of this application.

[0057] Figure 6 Schematic diagram of camera scanning and detecting control points in some examples of this application.

[0058] Figure 7 This is a schematic diagram of the association between reference control points and detection control points in some examples of this application.

[0059] Figure 8 This is an illustration of the reference control points and detection control point mapping in some examples of this application.

[0060] Fig. 9 This is a schematic diagram of the association between image source display and projection display in some examples of this application.

[0061] Fig.10 This is a schematic diagram of determining the projection position of virtual-real matching in some examples of this application.

[0062] Fig.11 Schematic diagram of determining the position of the image source by inverse mapping in some examples of the present application.

[0063] Fig.12 This is a schematic diagram of determining the projection position of virtual-real matching in some examples of this application.

[0064] Fig.13 In some examples of this application, the detected control points are normalized to a standard plane diagram.

[0065] Fig.14 This is an illustration of the reference control points and detection control point mapping in some examples of this application.

[0066] Fig.15 This is a schematic diagram of determining the projection position of virtual-real matching in some examples of this application.

[0067] Fig.16 This is a schematic diagram of image plane equation collection in some examples of this application.

[0068] Fig.17 This is a schematic diagram of multi-viewpoint calibration in some examples of this application.

[0069] Fig.18 Schematic diagram of HUD display device module in some examples of this application.

[0070] Fig.19 This is a schematic diagram of the composition of the HUD display device in some examples of this application.

[0071] Fig. 20 This is a schematic diagram of projection display in a vehicle in some examples of the present application. DETAILED DESCRIPTION

[0072] The present application will be described in detail below in conjunction with the accompanying drawings, but the content of the description is merely some examples recorded in the present application and does not limit the present application. Any changes in structure, method or function made by ordinary technicians in this field based on these examples are included in the protection scope of the present application.

[0073] It should be noted that the same reference numerals or marks may be used in different examples, but these do not represent an absolute connection in structure or function. In addition, the "first", "second", etc. mentioned in each example are only for the convenience of description, and do not represent an absolute distinction in structure or function, nor can they be understood as indicating or implying the relative importance or the number of corresponding objects. Unless otherwise specified, the "at least one" that may be involved in the description refers to one or more, and "more than one" refers to two or more.

[0074] In addition, when expressing a feature, the character " / " can indicate that there is an or relationship between the front and back associated objects. For example, head-up display / head-up display can be expressed as head-up display or head-up display. When expressing an operation, the character " / " can indicate that there is a division relationship between the front and back associated objects. For example, the magnification M=L / P can be expressed as L (virtual image size) divided by P (image source size). Moreover, the "and / or" in different examples is only to describe the association relationship between the front and back associated objects. This association relationship can include three situations. For example, a concave mirror and / or a convex mirror can be expressed as the existence of a concave mirror alone, the existence of a convex mirror alone, and the existence of both a concave mirror and a convex mirror.

[0075] HUD projection display mainly uses the optical reflection principle to reflect the imaging light to be displayed through a transparent surface into the viewer's eyes. The human eye can see the virtual image information in the opposite direction of the light. Correspondingly, the transparent surface can be the windshield of the vehicle, and the windshield can be used as a display screen to display the navigation instructions of the vehicle, the vehicle's driving speed, etc. Figure 1As shown, the HUD display device may include at least an optical machine 1, a first reflector 2, a second reflector 3, etc., wherein the optical machine 1 includes a backlight source and an image source (not shown), the backlight source is used to provide illumination light and adjust the brightness of the illumination light according to control, for example, the backlight source may be an LED (Light Emitting Diode), a laser, etc. Under the illumination light provided by the backlight source, the image source adjusts the corresponding display content according to control and projects the display light from the surface of the image source, for example, the image source may be an LCD (Liquid Crystal Display), a DMD (Digital Micromirror Devices), a MEMS (Micro-Electro-Mechanical System) micromirror, a LCOS (Liquid Crystal on Silicon), etc. The first reflector 2 and the second reflector 3 can project the display light projected by the optical machine 1 onto the windshield 4, so as to customize the optical path in a smaller space and meet different projection display requirements. The first reflector 2 and the second reflector 3 can be set as a concave mirror, a convex mirror, a concave lens, a convex lens, etc. according to the requirements of optical planning, and the surface shape of the lens can adopt a free-form surface. Optionally, at least one of the first reflector 2 and the second reflector 3 can also be adjusted to a certain degree of angle, so as to change the projection position of the display light on the windshield 4 to meet the needs of viewers of different heights. The display light of the optical machine 1 is finally reflected on the windshield 4 of the vehicle to form a virtual image 5. When the human eye 6 observes the virtual image 5 facing the windshield 4, it can feel a certain sense of depth, just like watching a real object at a specific distance outside the windshield. The virtual image 5 can be the navigation instruction content, vehicle speed, etc. as described above. It should be added that, according to the characteristics of different optical machines, the HUD display device can also be provided with a diffuser. In some examples, the HUD display device can also include Fresnel lenses, waveguide optical devices, diffractive optical devices, holographic optical devices, tapered optical fibers, etc.

[0076] As mentioned above, the optical lenses, windshields, etc. in the optical system will affect the actual imaging effect of the virtual image 5. Taking the optical lenses as an example, in the actual production process, the surface shape of the optical lenses after actual processing may deviate from the designed surface shape. At the same time, in the actual installation process of the optical lenses, there will also be deviations between the installation position and the designed position. These deviations will cause the final projected virtual image to differ from the designed imaging effect. Specifically, in addition to the distortion of the virtual image itself that needs to be corrected, there will also be inconsistencies in the projection position and size relative to the windshield that are inconsistent with the design values ​​and need to be corrected. Accordingly, it is impossible to achieve the design goal of accurate fitting with objects outside the windshield. Figure 2As shown, due to the need for enhanced display, the display elements that need to be configured in the HUD display device must be fitted with the object 71 outside the windshield 4, that is, the human eye 6, the virtual image corresponding to the display element and the object 71 must be on the same straight line, and for the HUD display device, only the position of the display element corresponding to the virtual image can be controlled and adjusted. Correspondingly, the virtual image formed by the HUD display device projected on the windshield 4 will be equivalently displayed near positions 51 and 52 according to the projection control parameters, that is, when the human eye 6 looks at the windshield 4, it will feel that the projected virtual image is near positions 51 and 52. In order to make the projected display element fit with the object 71, the display element should be projected at position 52 according to the design value. However, due to the above-mentioned deviations in production and installation, the display element can actually only be seen by the human eye 6 at position 51. Since it is not in the line of sight path determined by the object 71 and the human eye 6, the human eye 6 feels that the display element is not perfectly fitted with the object 71. If a better fitting effect is required, it is necessary to correct the deviation through software or other means so that it is normally projected at position 52. More specifically, the reason why these deviations affect the effect of virtual-real fitting may be directly related to the error between the lower viewing angle and field of view designed by the HUD display device and the actual projection result. Among them, due to the error in the lower viewing angle, the display element will not be projected on the designed coordinate point, such as Figure 2The display element is not projected at position 52 as designed, resulting in deviation from the corresponding position of the actual object. At the same time, due to the error in the field of view angle, the display element does not project the size that should be projected in the design, resulting in a mismatch with the size of the actual object, increasing the visual experience of misalignment. In addition, the different viewpoints of the human eye in the eye box will not only produce the above-mentioned different errors in the lower viewing angle and field of view angle, which will affect the fitting effect, but also have the objective dynamic parallax that affects the fitting relationship, that is, the change in the position of the human eye will cause the range of vision to change. At the same time, the distortion in the depth direction will also aggravate the fluctuation of the fitting relationship caused by the dynamic parallax, that is, the human eye will move, and the virtual image corresponding to the display element will also move in the depth direction, and the straight line viewing angle determined by the human eye and the virtual image is also moving, which will not match the relatively fixed object position. It should be noted that the eyebox refers to a conical area between the optical system of the HUD display device and the human eye. It is also the area where the display content meets the optimal optical design indicators. Within the eyebox range, the human eye can see the complete FOV (i.e. the entire image) projected by the HUD. When the human eye exceeds this range, it may present serious image distortion, color errors, or even no content display. The viewpoint position refers to the specific position of the viewer's eye when observing the virtual image within the eyebox range. Since the pupil (generally with a radius of 2 to 4 mm) moves within the eyebox range (generally designed to be 130*50 mm), that is, at different viewpoint positions, different optical transmission paths will be generated, so there will be certain differences in the images seen, which is also a specific manifestation of dynamic distortion. At the same time, different deviations such as the downward viewing angle and field of view will also be generated. The following will detail how to overcome these deviations and eliminate possible virtual and real mismatches.

[0077] like Figure 3As shown, the optical machine 1 (specifically, the image source display surface of the optical machine 1) in the HUD display device emits the display light, passes through the first reflector 2 and the second reflector 3 to the windshield 4, and finally reflects to the human eye through the windshield 4. The human eye will see the virtual image formed by the display light projected on the windshield 4 in the reverse direction near the position P0 based on the principle of optics. Correspondingly, it can be an image plane composed of virtual image projection points, which reflects the projection of the HUD display device in the projection space near the position P0. Based on the principle of optical transmission, the first pixel 501, the second pixel 502, the third pixel 503 and the fourth pixel 504 in the virtual image can be specific virtual image projection points, which can be generated by the projection of specific pixels on the image source display surface, and have a one-to-one corresponding relationship. In this example, the first pixel 501, the second pixel 502, the third pixel 503 and the fourth pixel 504 are projected at different positions near the position P0, and have certain spatial characteristics, which are determined by the entire optical system, and also respectively reflect the information presented by the corresponding pixels on the image source display surface. In some examples, the static distortion formed by the optical system is not only reflected in the plane facing the human eye, but also in the horizontal and vertical deformations. There will also be misalignment in the depth direction (i.e., the direction from the human eye to the virtual image). For example, the distances of the first pixel 501, the second pixel 502, the third pixel 503, and the fourth pixel 504 from the human eye are inconsistent, and the pixels on the image source display surface are not on the same plane, that is, the image plane composed of all virtual image projection points such as the first pixel 501, the second pixel 502, the third pixel 503, and the fourth pixel 504 is not a plane, and the actual presentation may be a curved surface. Referring to the above description of the deviation, the image plane will also be affected by the errors of the lower viewing angle and the field of view, resulting in overall deviation and size expansion. Although the above non-ideal imaging effects are complicated, once the optical system including the optical machine 1, the first reflector 2, the second reflector 3, and the windshield 4 is installed, it will be relatively fixed, because the actual production and installation deviations will not change, so the actual deviation information contained in the image plane after projection can be quantified by calibration. Specifically, the parameters of the virtual image can be collected in advance using the camera 61 within the eye box range of the human eye, so as to evaluate a model that can actually reflect the projection image surface. Since this model is derived from the actual shooting of the camera, it can represent the actual deviation. The calibrated data can be used to adjust the actual projection, which will be described in detail below.

[0078] In some examples, in order to improve the convenience of camera acquisition, a standard reference control point image can be displayed on the image source display surface of the optical machine 1, such as Figure 4 In the 4*4 dot matrix image, each row and each column can be arranged according to the pixel distribution rule on the image source display surface, which is convenient for subsequent calculations. Optionally, the reference control point can also be used as follows Figure 5In the chessboard image shown, the cross corners of the chessboard are used as reference control points for projection, which can better reflect the regularity of the regular arrangement. Taking the dot matrix image as an example, each reference control point can represent a pixel on the image source display surface, and each reference control point will have a two-dimensional coordinate on the image source display surface. The pixels corresponding to several reference control points can be represented by (u1, v1), (u2, v2), etc. Figure 3 , the reference control point 501r on the image source display surface can form the first pixel 501 of the projected virtual image near the position P0, and accordingly, the reference control point 502r, the reference control point 503r, and the reference control point 504r correspond to the second pixel 502, the third pixel 503, and the fourth pixel 504 respectively. By analogy, other reference control points will also form corresponding virtual image projection points near the position P0, and these virtual image projection points can represent some spatial features of the composed image surface. Figure 6 As shown, these virtual image projection points can be Figure 3 The detection control points are captured by the camera 61 in the image processing unit, which can be collectively referred to as detection control points. Although the detection control points correspond to the reference control points one by one, they are not arranged as regularly as the reference control points due to the influence of optical transmission. In addition, their spacing, center position, etc. are likely to be different from those of the reference control points. For details, please refer to the above description of the causes of deviation. In this example, the distribution of the detection control points is the scan result after distortion correction, that is, there is basically no distortion on the plane facing the human eye, and Figure 6 The misalignment of the reference control points in can be understood as the inconsistency in the depth direction, which causes the appearance of a misalignment effect on the two-dimensional plane. In addition, the partial misalignment effect caused by imperfect distortion correction can also be considered. Correspondingly, the detection control points captured by the camera can be the three-dimensional coordinates of the space where the image plane is located, such as (x1, y1, z1), (x2, y2, z2), etc. Specifically, a binocular camera can be used. The binocular camera can not only obtain the horizontal coordinates and vertical coordinates of any detection control point on the plane facing the human eye, but also has its coordinates in the depth direction. In this way, the specific spatial position of the virtual image projection point in space can be accurately determined, so as to further analyze its relative position relationship with the human eye and objects in space. Optionally, the three-dimensional coordinates of the detection control point in space can also be detected by other optical sensing devices, such as phase focusing.

[0079] As mentioned above, the reference control points and the detection control points have a one-to-one matching relationship based on the projection relationship, that is, a 4*4 reference control point will project a 4*4 detection control point. Figure 7As shown, the direction of the arrow indicates the match between a specific reference control point and a specific detection control point. Specifically, these control points can be sorted in two dimensions, sorted according to the vertical arrangement order of each control point, and the control points belonging to the same row are temporarily stored in the corresponding list. Then, they are sorted according to the horizontal arrangement order of each row of control points, and the different columns where the control points are located are determined in the corresponding list. The reference control point or the detection control point can be uniquely determined through the sorting information of the horizontal and vertical dimensions. Optionally, the ID can be assigned in order from the upper left to the lower right, and the reference control points with the same assignments are matched with the detection control points to complete the determination of the corresponding relationship between the reference control points and the detection control points. In this way, each reference control point can uniquely point to a detection control point, and then the different coordinate information of the two pairs of control points is stored, such as the reference control point uses two-dimensional coordinates, the detection control point uses three-dimensional coordinates, or the reference control point and the detection control point both use two-dimensional coordinates. Furthermore, the reference control points and the detection control points represent the pixel distribution of the image source display surface and the distribution of the virtual image projection points of the image surface, respectively. Therefore, the reference control points and the detection control points can be used to fit the entire image source display surface and the image surface, thereby determining the mapping relationship between the entire image surface and the image source display surface. In this way, the spatial characteristics of the image surface can be determined, thereby determining the positional relationship with other objects in the space, and finding the mapping relationship before and after the actual projection, so that the image source display can be controlled to achieve an ideal projection display effect.

[0080] In some examples, such as Figure 8 As shown, the corresponding relationship between the reference control point and the detection control point is specifically determined by associating the three-dimensional coordinates of the detection control point with the two-dimensional coordinates of the reference control point. The specific matching relationship can be determined by referring to Figure 7 , and the two-dimensional coordinates of the reference control points can be determined by the pixel distribution in the image source display surface itself, and the three-dimensional coordinates of the detection control points can be determined by the above-mentioned optical sensing device. Furthermore, it is necessary to expand the distribution of the detection control points to the entire image surface, so that any virtual image projection point in the projection space can be inversely mapped to the image source display surface for display control. Optionally, the positions of the virtual image projection points other than the detection control points are determined by interpolation or fitting, thereby obtaining a complete image surface fitting set, so that a connection can be established between the spatial scale of the image surface and the plane scale of the image source display surface. Fig. 9As shown, for the image source display surface, it not only has the pixel information where the reference control point is located, but also has the pixel information outside the reference control point. For the projection space where the image plane is located, it not only has the virtual image projection point where the detection control point is located, but also has the virtual image projection point outside the detection control point. In this way, a complete mapping relationship between the image plane and the image source display surface is established, and the image plane fitting set can represent the spatial position of all virtual image projection points in the projection space, thereby serving as the source data for inverse mapping to the image source display surface. Optionally, in order to more accurately represent the image plane, a continuous image plane equation can be directly fitted according to the three-dimensional coordinates of the detection control point. The image plane equation, as a continuous image plane fitting set, can be used to determine any position in the projection space where the image plane is located, so that it can adapt to the line of sight determined by any human eye and object. Accordingly, when calculating the image plane equation, the abnormal three-dimensional coordinates in the detection control point will also be eliminated to reduce the interference of abnormal coordinate detection on it and improve the accuracy of image plane equation fitting.

[0081] As mentioned above, through calibration, we can have an image plane fitting set when the optical system is stable (i.e. after installation), and a mapping relationship between the image plane and the image source display surface. The image plane fitting set can reflect the distribution of the virtual image projection points of the actual projection in space, and the mapping relationship between the image plane and the image source display surface can inversely map any virtual image projection point to the image source display surface, thereby restoring the parameters required for the actual projection display. Therefore, if a fitting display is required on a specific object, it is necessary to find a virtual image projection point in the image plane fitting set that is in the same straight line as the specific object and the human eye, and then determine the actual display position on the image source display surface based on the mapping relationship between the image plane and the image source display surface. Fig.10 As shown, according to the actual projection scene, from front to back are object 71, an image plane fitting set including all virtual image projection points, and human eye 6. In this example, the position of object 71 in space can be determined by at least one of millimeter wave radar, laser radar, and binocular camera, and the position of human eye 6 in space can be determined by a camera in the car or a driver monitoring system. Optionally, human eye 6 can also adopt a default position, such as directly using the three-dimensional coordinates of the human eye when a specific driver sits normally in the driver's seat based on face recognition or default settings. Correspondingly, the projection position that can form a fitting relationship with object 71 is the virtual image projection point on the straight line between human eye 6 and object 71, that is, the position where the line of sight of human eye 6 viewing object 71 is connected to the image plane (image plane fitting set) to form an intersection. Because the image plane fitting set corresponding to the image plane reflects the distribution of all virtual image projection points in space, it can be easily determined. As shown in Fig.11As shown, when the coordinate position in the image plane fitting set that fits the object 71 is determined, such as (x0, y0, z0), it can be inversely mapped to the image source display surface according to the mapping relationship between the image plane and the image source display surface, such as (u0, v0), which means that if the display element is displayed on (u0, v0) of the image source display surface, it will actually be projected on (x0, y0, z0) in space, so as to fit the object 71. In this example, the image plane fitting set is a set of virtual image projection points formed by the projection of all pixels on the image source display surface in space, so it is relatively discrete. In particular, the size expansion and depth distortion of the projection may cause gaps between adjacent virtual image projection points, such as Fig.12 As shown, when the straight line between the human eye 6 and the object 71 is used to determine the intersection in the image plane fitting set, the straight line may pass through the gap without intersecting with any actual virtual image projection point. At this time, there is no way to directly find the corresponding pixel position on the image source display surface. Accordingly, approximate processing can be performed based on the recognition accuracy of the human eye. For example, the straight line passes between the virtual image projection point 505 and the virtual image projection point 506, but the virtual image projection point 505 is closer to the straight line between the human eye 6 and the object 71. Therefore, the virtual image projection point 505 can be determined as the intersection position, and then the corresponding pixel position of the virtual image projection point 505 on the image source display surface is determined according to the mapping relationship between the image plane and the image source display surface. Similarly, the display element to be attached to the object 71 is displayed at the corresponding pixel position.

[0082] In some examples, when the image plane fitting set is a specific image plane equation, the discrete virtual image projection points can be converted into a continuous image plane representation, thus eliminating Fig.12 The gap in the image plane is used to determine the position where the image plane fits with the object 71, which is to directly calculate the intersection point between the human eye 6 and the object 71 that has a visual connection with the image plane, that is, the result of the calculation of the linear equation between the human eye 6 and the object 71 and the image plane equation. This example directly converts the geometric representation into a pure algebraic calculation, which improves the feasibility and accuracy. Similarly, the three-dimensional coordinates or two-dimensional coordinates determined in the image plane equation can be inversely mapped to the two-dimensional coordinates on the image source display surface according to the mapping relationship between the image plane and the image source display surface, so that the image source can be controlled to display specific display elements.

[0083] In some examples, such as Fig.13As shown, considering that the projection display has distortion in the depth direction, the virtual image projection points of the actual projection will not be on the same depth plane, which shows that the actual image plane composed of the virtual image projection points is a curved surface. Therefore, the above-determined image plane fitting set contains three-dimensional coordinate information, specifically, a mapping relationship is established between the three-dimensional coordinates and the two-dimensional coordinates on the image source display surface. In order to simplify the mapping relationship and eliminate possible errors, a virtual standard plane P1 can be defined near the detection control point, that is, the image plane fitting set to be determined is not for the actual image plane composed of the virtual image projection points, because this image plane is a curved surface, but a virtual image plane. This image plane is a plane, which is more convenient for subsequent projection display processing. Specifically, according to the linear relationship between the acquired detection control point position and the camera 61, the position of the detection control point on the standard plane P1 is determined, that is, the intersection of the straight line between the camera 61 and the virtual image projection point and the standard plane P1, as shown in Fig.13 As shown, the detection control point 501 corresponds to the position 501a on the standard plane P1, the camera 61, the detection control point 501 and the position 501a are on the same straight line, similarly, the detection control point 502 corresponds to the position 502a on the standard plane P1, the detection control point 503 corresponds to the position 503a on the standard plane P1, and the detection control point 504 corresponds to the position 504a on the standard plane P1. Therefore, when determining the image plane fitting set, the interpolation calculation or fitting is performed based on the position 501a, the position 502a, the position 503a, and the position 504a, or the image plane equation calculation is performed directly. For details, please refer to the above example. Fig.14 As shown in the figure, the correspondence between the reference control points and the detection control points is not only through Figure 7 In addition to determining the match in the example, the two-dimensional coordinates of the detected control points are also associated with the two-dimensional coordinates of the reference control points, where the position of the detected control points is represented by the horizontal and vertical coordinates corresponding to the standard plane P1. In some examples, when the actual image plane composed of the virtual image projection points does not have fluctuations in the depth direction, that is, the actual image plane is also a plane, the three-dimensional coordinates of the detected control points can also be directly reduced to two-dimensional coordinates for representation, so that a good association relationship can be established with the pixel coordinates on the image source display surface.

[0084] Accordingly, if Fig.15As shown, the corresponding image plane fitting set and the mapping relationship between the image plane and the image source display surface can also be calibrated. The image plane fitting set reflects the distribution characteristics of the actual virtual image projection points in the projection space. Unlike the above example, the mapping relationship between the image plane and the image source display surface is to convert the virtual image projection points in the space into a two-dimensional representation and establish an association with the two-dimensional representation in the image source display surface. When the display element needs to be projected on the object 71, it is also necessary to determine the position where the straight line between the human eye 6 and the object 71 intersects the image plane. Unlike the above example, the position of the intersection is determined by using the standard plane P1. Optionally, the standard plane P1 can also have a corresponding image plane equation, that is, the focus coordinates can be determined according to the straight line equation between the human eye 6 and the object 71 and the image plane equation, such as (x0, y0). Further, since the mapping relationship between the image plane and the image source display surface can be inversely mapped to the two-dimensional coordinates on the image source display surface based on the two-dimensional coordinates, the corresponding position on the image source display surface can be determined, such as (u0, v0). The display element is displayed on (u0, v0) of the image source display surface and is projected through the optical system of the HUD display device, so that it will fit normally on the object 71, that is, the display element that needs to be fitted is projected on the line of sight between the human eye 6 and the object 71.

[0085] In some examples, the image plane equations determined by the virtual image projection points are not determined by fitting the three-dimensional coordinates of the detection control points, but are determined by the distribution of the virtual image projection points captured by cameras at different positions. The cameras at different positions can be monocular cameras that shoot virtual images at different positions, or binocular cameras that output two captured images at the same time. In this example, regular reference control points are also output during calibration, and the corresponding detection control points are captured in the projection space. This example is more suitable for the HUD display device itself, where the dynamic distortion is not so obvious, and the static distortion is corrected so that the spacing between adjacent detection control points is consistent and basically in the same plane. Specifically, if Fig.16 As shown, after the reference control points displayed on the image source display surface are projected by the optical system, there will be detection control points corresponding to all reference control points on the image plane P2 in the projection space, including detection control points 501, detection control points 502, detection control points 503 and detection control points 504. These detection control points are photographed by a camera 61 within the eye box during calibration. The camera is used to photograph the detection control points in the first posture 611 to obtain the first image, and to photograph the detection control points in the second posture 612 to obtain the second image. All detection control points are present in the first image and the second image, but the distribution conditions presented are inconsistent due to the different postures of the camera shooting. Accordingly, the coordinate information of the detection control points in the first image and the second image will be extracted. Specifically, the image plane equation including the detection control points can be calculated according to the following formula, and finally the spatial characteristics of all virtual image projection points can be represented:

[0086] in, represents the coordinates of all detected control points in the first image, represents the coordinates of all detected control points in the second image, K represents the intrinsic parameter matrix between the two pose cameras, l represents the baseline length between the two pose cameras, represents the actual image plane pose, Represents the coordinates of all detected control points on the actual image plane, represents the first pose of the camera, d represents the distance between adjacent detection control points in the actual image plane, i represents the row where the detection control point is located in the actual image plane, j represents the column where the detection control point is located in the actual image plane, represents the normalization process. , , d can be solved, so the corresponding image surface equation can be constructed based on this.

[0087] Furthermore, the position coordinates of all virtual image projection points can be obtained according to the image plane equation, specifically, the two-dimensional coordinates relative to the image plane, thereby establishing an association with the two-dimensional coordinates on the image source display surface. Figure 7-15 In this example, the optical sensor device is also used to obtain the position of the detection control point. Specifically, the position of the detection control point relative to the first image and the second image is obtained by using cameras with different postures, so as to finally determine the corresponding image plane equation and the mapping relationship between the image plane and the image source display surface, so as to express the actual deviation during projection display, so as to perform inverse operation on the projection display based on the above data. For details, please refer to Fig.11 , Fig.15 , and finally determine the position that actually needs to be displayed on the image source display surface, ensuring that the display elements on the image source display surface are projected at a spatial position that fits the specific object.

[0088] As described above, the HUD display device calculates the position of the display element to be bonded on the image source display surface through the calibration data stored in advance, and the calibration data stored in advance includes the image plane fitting set and the mapping relationship between the image plane and the image source display surface. However, the size of the display element projected on the image plane will also affect the bonding effect. For example, if the display element projection is too large, it will be difficult to form a good bonding effect with an object that is not actually large. The projection size of the display element is not only related to the standard display size on the image source display surface, but also to the parameters of the optical system. When the design value of the optical system is inconsistent with the actual value, it is necessary to adjust the scaling ratio on the image source display surface to offset the size affected by the deviation of the optical system. Specifically, the scaling factor can be determined according to the mapping relationship between the image plane and the image source display surface. For example, the mapping relationship between the image plane and the image source display surface includes the distance between adjacent virtual image projection points in the image plane and the distance between adjacent pixels on the image source display surface. The scaling factor can be determined according to the ratio of the distances between the two. Optionally, the calculated distance ratio between the two will have a lookup table relationship with a specific scaling factor. When the scaling factor is determined, the display element of the standard size is enlarged or reduced in size, and the processed display element is displayed at the pixel position calculated in the above example, thereby improving the coordination of the entire projection fit.

[0089] In the above examples, to achieve a perfect fit for projection display based on calibration data, the camera position during calibration generally needs to be consistent with the actual eye position during projection display, or the projection effect does not change much when the eye moves within the eye box. In some examples, in order to eliminate the different deviations that may exist in the movement of the eye, the actual parameters at different viewpoints can be calibrated. Fig.17 As shown in the figure, the viewpoints of the human eye at the lower left corner and the upper right corner may show different downward viewing angles, field angles, depth distortion, etc. due to the inconsistency of the optical transmission path and the deviation of the optical system itself in production and installation. In this way, problems will arise when using a set of calibrated data for projection display. Figure 8 Based on the example, during calibration, optical sensing devices such as cameras are used to scan at different viewpoints within the eye box range, that is, the same reference control points can be output on the image source display surface, and the detection control points at different viewpoints can be collected by moving the camera. For binocular cameras, the viewpoint position of the main camera can be used as the standard. For other optical sensing devices, single-point detection can also be achieved by passing through different viewpoints. If the viewing effects at different viewpoints vary greatly, then there will be certain differences in the detection control points at different viewpoints obtained by camera scanning. At this time, the above example method can be used to calculate the image plane fitting set at different viewpoints and the mapping relationship between the image plane and the image source display surface. Fig.17As shown in the figure, when there are 15 viewpoints within the eye box, the camera will be calibrated at each of the 15 viewpoints, and 15 sets of calibration data will be generated, thereby simulating the viewing situation of the human eye at different viewpoints. In projection display, when the position of the human eye is obtained through the camera in the car or the driver monitoring system, the position of the human eye is not only used to determine the direction of the line of sight to view the object, but also to determine the viewpoint position of the human eye in the eye box. Different image plane fitting sets and the mapping relationship between the image plane and the image source display plane are called according to the specific viewpoint position. Fig.17 These data are obtained by calibrating the camera to simulate the human eye at different viewpoints and are stored in the memory of the HUD display device in advance for use during projection display. This can adapt to changes in the human eye and improve the accuracy of fitting.

[0090] When the HUD display device implementing the display method and calibration method in the above example is applied to a vehicle, a better virtual-real fitting effect can be achieved when the HUD display device is projected. Specifically, an image plane fitting set that can be represented by calibration in the projection space is stored internally. When projecting, the best fitting position is selected from the image plane fitting set, and then the mapping relationship between the image plane and the image source display surface is used to finally determine the best display position on the image source display surface. Fig.18 As shown, the HUD display device integrated in the vehicle can be powered and data by the vehicle computer 92, or the HUD display device itself can provide power and generate data. The HUD display device can specifically include a processor 91, an Ethernet interface 901, a CAN (Controller Area Network) interface 902, a power management module 903, a running memory 904, a storage memory 905, a temperature detection 906, a motor 907, a backlight source 908, an image source 909, a positioning module 910, a radar 911, a camera 912, etc. It should be noted that Fig.18 The modules listed in the description are merely exemplary and do not constitute any limitation. In some examples, the HUD display device may further include other modules. In addition, the above modules may be implemented in one or more hardware in different examples, or a single module may be implemented by a combination of multiple hardware.

[0091] The processor 91 serves as the control center of the HUD display device, and includes one or more processing units of any type, including but not limited to a microcontroller, a microcontroller, a DSP (Digital Signal Processor) or any combination thereof. The processor 91 is used to generate operation control signals according to a computer program, realize the control of other modules, and cooperate with corresponding modules to process the acquired or inherent data, instructions, etc.

[0092] The Ethernet interface 901 is a network data connection port for local area network communication, which defines a series of software and hardware standards. Multiple electronic devices can be connected together through the Ethernet interface 901. In this example, the processor 91 can exchange information with the vehicle computer 92 through the Ethernet interface 901, such as sending data to the vehicle computer 92 or receiving data sent by the vehicle computer 92.

[0093] The CAN interface 902 is a network data connection port of the controller area network, which provides a standard bus for the control system and embedded industrial control inside the car, and realizes the communication interaction between the control nodes. In this example, the processor 91 can also exchange information with the vehicle computer 92 through the CAN interface 902. Optionally, the processor 91 can also connect to other external devices through the CAN interface 902. In some examples, the processor 91 can also be provided with a GPIO (General-purpose input / output) interface to improve the compatibility of peripheral connections.

[0094] The power management module 903 is connected to the vehicle computer 92 and can receive the power provided by the vehicle computer 92 to provide a regulated power supply for each module of the HUD display device, thereby ensuring that the processor 91 and each module operate under normal voltage supply and avoid damage due to overvoltage.

[0095] The running memory 904 is used to store computer programs executed by the processor 91, temporarily stored calculation data, data exchanged with the storage memory, etc. The running memory 904 can be a memory such as SDRAM (Synchronous Dynamic Random-access Memory).

[0096] The storage memory 905 is used to store resources such as display content related to the HUD display device, and long-term storage of running programs and data, etc. The storage memory 905 can be a memory such as Flash (flash memory). In some examples, the processor 91 can also provide an interface to access an external memory.

[0097] Temperature detection 906 is used to detect the temperature inside the HUD display device, and may specifically include several temperature sensors. Since the resistance value of the temperature sensor changes with the change of temperature, the processor 91 can determine the resistance value of the temperature sensor at the corresponding temperature according to the voltage change between each temperature sensor and the voltage divider resistor under a fixed power supply voltage, thereby reversely deducing the temperature at the location of the temperature sensor. In some examples, the processor 91 can control several temperature sensors through the GPIO interface, and several temperature sensors can be set at different locations inside the HUD display device. The processor 91 can use time-sharing detection to obtain the temperature values ​​fed back by several temperature sensors respectively.

[0098] Motor 907 is used to drive the optical lens in the HUD display device to rotate under the control of processor 91, so as to achieve the change of the corresponding optical path. For example, when the backflow of sunlight causes the temperature rise on the surface of the image source, the motor can be used to drive the optical lens to prevent the external sunlight from reaching the surface of the image source. In some examples, the processor 91 can also drive the fan provided on the HUD display device through motor 907 to increase the speed of air exchange between the inside and outside of the HUD display device to achieve heat dissipation. Specifically, the motor 907 is connected to the processor 91 through a motor driver chip, and the motor driver chip provides high-performance power output for the motor 907, and can also communicate and control with the processor 91 through interfaces such as SPI (Serial Peripheral Interface).

[0099] The backlight source 908 is used to provide illumination light and adjust the brightness of the illumination light according to the control of the processor 91, so as to adjust the projection display brightness of the entire HUD display device. The backlight source 908 cooperates with the image source 909 to realize the main function of optical machine projection display. The backlight source 908 can be an LED (Light Emitting Diode), a laser, etc. Specifically, the backlight source 908 is connected to the processor 91 through a backlight driver chip, and the backlight driver chip provides a driving voltage for the backlight source 908 and controls the brightness of the backlight source 908 under the pulse width signal output by the processor 91.

[0100] The image source 909 is used to display an image of corresponding content and project the display light corresponding to the image according to the control of the processor 91. The image source 909 can be an LCD (Liquid Crystal Display), a DMD (Digital Micromirror Devices), a MEMS (Micro-Electro-Mechanical System) micromirror, a LCOS (Liquid Crystal on silicon), etc.

[0101] The positioning module 910 is used to monitor the position of the HUD display device and the corresponding vehicle. The positioning module 910 can be a global navigation satellite system such as GPS (Global Positioning System) and Beidou Satellite Navigation System. By measuring the distance between the satellite and the receiver on the positioning module 910 at different positions, the corresponding position and orientation data are determined. In some examples, the positioning module 910 may also include an inertial navigation system, which is based on Newton's laws of mechanics. By measuring the acceleration of the positioning module 910 in the inertial reference system, integrating it over time, and transforming it into the navigation coordinate system, the speed, yaw angle, and position data in the navigation coordinate system are obtained. Optionally, the inertial navigation system can assist the global navigation satellite system to achieve more accurate positioning and provide the processor 91 with corresponding position information.

[0102] The radar 911 is used to determine the position of a target object through electromagnetic waves, and can usually determine the distance of the target object from the vehicle where the radar 911 is located.

[0103] Camera 912 includes a vehicle body camera and an in-vehicle camera, wherein the vehicle body camera is used to determine the position of the target object through visual recognition, and the vehicle body camera can be a monocular camera or a binocular camera. The biggest difference between a monocular camera and a binocular camera is that a binocular camera can capture images from two different perspectives, thereby obtaining distance information in three-dimensional space. The in-vehicle camera is used to identify the behavior status of the driver and passengers in the vehicle, including fatigue detection, distraction detection, expression recognition, gesture recognition, sight tracking, etc. In this example, the in-vehicle camera can also implement eye tracking.

[0104] In some examples, the positioning module 910, radar 911 and camera 912 can also be directly connected to the vehicle computer 92, and are not directly connected to the processor 91 of the HUD display device. For example, the vehicle computer 92 itself is integrated with a positioning module for position tracking and a radar and camera for automatic driving. The HUD display device can obtain the collected data of the positioning module, radar and camera in real time through communication with the vehicle computer 92.

[0105] like Fig.19As shown, the HUD display device that implements the above-mentioned example display method or calibration method may specifically include a processor 931, a memory 932, an input device 933 and an output device 934, wherein the input device 933 may include an operation button integrated on the display device, etc., and the display device may receive input control instructions and data through the input device 933. The output device 934 may include an image source integrated on the display device, etc., and the display device may output corresponding instructions or data to the output device 934. Further, the memory 932 stores a computer program running on the processor 931, and the processor 931 implements the above-mentioned example display method or calibration method when executing the computer program. In some examples, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above-mentioned example display method or calibration method when executed by the processor.

[0106] like Fig. 20 As shown, the vehicle can be provided with the above-mentioned HUD display device. Specifically, the HUD display device is integrated inside the center console 10, such as in front of the steering wheel. The corresponding display light is projected onto the opposite vehicle windshield 4 through the projection window 102 of the HUD display device. The effect of the viewer observing the windshield 4 from the cockpit is that the corresponding virtual image can be directly seen. Specifically, the corresponding display area 50 of the windshield 4 includes basic display information (vehicle speed information, gear information, etc.) and extended display information (warning information, navigation information, etc.) and other display elements. The driver can view the corresponding content without lowering his head while driving, which improves safety. More importantly, the enhanced display information such as the navigation icon viewed by the driver can find the accurate display position on the image source display surface through inverse calculation, ensuring that the enhanced display information can be well fitted with the object outside the windshield 4. In some examples, the vehicle can also distribute the program for obtaining the display method or calibration method in the above-mentioned example through the above-mentioned computer-readable storage medium, so that the display device can be updated and upgraded accordingly. It should be noted that the means of transportation are not limited to cars as a means of transportation, but may also include buses, trucks, excavators, motorcycles, trains, high-speed trains, ships, yachts, airplanes, spacecraft, etc. The projected windshield is not limited to the front windshield of a car, but may also be a transparent surface at other locations.

[0107] In combination with the above examples, the technical solutions involved in the present application can be directly embodied as hardware, a software module executed by a control unit, or a combination of the two, that is, one or more steps and / or one or more step combinations, which can correspond to various software modules of a computer program flow, or to various hardware modules, such as ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any appropriate combination thereof. For the convenience of description, the above description is divided into various modules and described separately according to their functions. Of course, when implementing the present application, the functions of each module can be implemented in the same or more software and / or hardware.

[0108] Through the description of the above examples, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution involved in the present application can be essentially or partly contributed to the prior art in the form of a software product. The software is executed by a microcontroller unit and, depending on the required configuration, may include one or more microcontroller units of any type, including but not limited to a microcontroller unit, a microcontroller, a DSP (Digital Signal Processor) or any combination thereof. The software is stored in a memory, such as a volatile memory (such as a random access memory, etc.), a non-volatile memory (such as a read-only memory, a flash memory, etc.) or any combination thereof.

[0109] In summary, the present application uses the first display element to achieve virtual-real bonding with the first object outside the vehicle windshield, which converts the position of the first object to be bonded outside the vehicle windshield into the pre-projection position coordinates corresponding to the image plane, and the pre-projection position is in the same straight line as the human eye and the first object, and then the pre-projection position coordinates are inversely mapped to the image source display surface according to the mapping relationship between the image plane and the image source display surface that has been calibrated in advance, and the first display element is displayed at the pixel point position determined by the inverse mapping. The present application can improve the virtual-real bonding effect of the HUD display device and enhance the user's viewing experience.

[0110] It should be understood that although this specification includes some examples, none of these examples contains only one independent technical solution, and this description of the specification is only for the purpose of clarity. A person skilled in the art should take the specification as a whole, and the technical solutions in each example can also be appropriately combined to form other implementation methods that can be understood by a person skilled in the art.

[0111] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present application. They are not intended to limit the scope of protection of the present application. All equivalent implementation methods or modifications that do not deviate from the teachings of the present application should be included in the scope of protection of the present application.

Claims

1. A display method, characterized in that: include: Obtaining a position of a first object outside a vehicle windshield; The HUD display device configures a first display element according to an image plane formed by projecting the vehicle windshield, wherein the first display element is projected to be aligned with the first object when viewed by human eyes; Determine, according to the position of the human eye, a first intersection position of the first object corresponding to the first object having a line of sight connection relationship on the image plane; According to the mapping relationship between the image plane and the image source display plane, it is determined that the first intersection position corresponds to a first pixel point position on the image source display plane, and the first display element is configured to be displayed at the first pixel point position.

2. The display method according to claim 1, characterized in that: The step of determining the first intersection position of the first object corresponding to the first intersection position having a line of sight connection relationship on the image plane according to the position of the human eye comprises: The first intersection position is determined according to a straight line equation and an image plane equation between the human eye and the first object.

3. The display method according to claim 1, characterized in that: Determining, according to the mapping relationship between the image plane and the image source display plane, that the first intersection position corresponds to the first pixel point position on the image source display plane comprises: The image plane is a plane, and the position of the first intersection point is represented by two-dimensional coordinates of the plane where the image plane is located; The position of the first pixel point is represented by the two-dimensional coordinates of the plane where the image source display surface is located.

4. The display method according to claim 1, characterized in that: The first display element is configured to display at the first pixel position: Determining a scaling factor according to a mapping relationship between the image plane and the image source display plane; The size of the first display element on the image source display surface is adjusted according to the scaling factor.

5. A calibration method, characterized in that: include: Controlling the HUD display device to output reference control points according to pixel distribution on the image source display surface, so that there are detection control points corresponding to the reference control points in the space where the image surface formed by the projection vehicle windshield is located; Scanning and acquiring the positions of the detection control points by an optical sensor device to determine an image plane fitting set representing the spatial distribution of the detection control points; A mapping relationship between the image plane and the image source display plane is determined according to the corresponding relationship between the reference control points and the detection control points and the image plane fitting set.

6. The calibration method according to claim 5, characterized in that: The scanning and obtaining of the position of the detection control point by an optical sensor device comprises: The three-dimensional coordinates of the detection control point in space are obtained through a binocular camera.

7. The calibration method according to claim 5, characterized in that: The scanning and acquiring the positions of the detection control points by an optical sensor device to determine an image plane fitting set representing the spatial distribution of the detection control points comprises: Using a camera in a first posture to photograph the detection control point to acquire a first image; Using a camera in a second position to photograph the detected control points to acquire a second image; An image plane equation including the detection control points is determined according to the first image and the second image.

8. A display device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the display method according to any one of claims 1 to 4 or the calibration method according to any one of claims 5 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the display method according to any one of claims 1 to 4 or the calibration method according to any one of claims 5 to 7 are implemented.

10. A means of transport, characterized in that: Includes the display device of claim 8 or the computer-readable storage medium of claim 9.

Images