AR-HUD calibration and display method and system

Abstract

The invention discloses an AR-HUD calibration and display method and system, and the method achieves the multi-dimensional technical breakthrough through obtaining the distribution region, position and distortion characteristics of an actual virtual image in front of a vehicle, dynamically adjusting an off-screen image parameter and a mapping relation. The problem that the display content is missing and incomplete due to the fact that the actual range of the virtual image is not consistent with the design parameters is solved, and it is ensured that the non-AR information is completely presented in the virtual image area; on the other hand, stretching distortion caused by a traditional compression strategy is restrained, the aspect ratio of non-AR information and the size of AR information are kept unchanged, the fitting precision of the AR target and the real environment is remarkably improved, and the problem of virtual-real dislocation is solved.

Description

Technical Field

[0001] The present invention belongs to the field of head-up display technology, and specifically relates to an AR-HUD calibration and display method and system. Background Art

[0002] Existing AR-HUD (augmented reality head-up display) calibration and display solutions typically perform distortion calibration and display based on preset design parameters (such as fixed virtual image position and size). However, due to hardware manufacturing errors, installation deviations, and environmental factors, the actual distribution area, position, shape, and size of the virtual image in front of the vehicle often differ significantly from the design parameters. This discrepancy leads to the following problems:

[0003] Missing or incomplete display content: When the off-screen image is generated directly based on the design parameters, the actual virtual image area may be smaller than the design range, causing some information to exceed the visible area (such as Figure 5 、 Figure 6 shown);

[0004] Non-uniform stretching distortion: To fit the maximum inscribed rectangle of the actual virtual image, the off-screen image needs to be compressed or cropped, resulting in geometric distortion of AR information and non-AR information (such as Figure 7 shown);

[0005] AR information fitting error: Pre-distortion processing without combining actual virtual image parameters will destroy the perspective relationship between the AR target and the real environment, resulting in a decrease in the accuracy of virtual-reality fitting.

[0006] At present, how to eliminate the display content missing, non-uniform distortion and AR information fitting error caused by the difference between the actual virtual image parameters and the design parameters and achieve complete distortion-free display is an urgent problem to be solved. Summary of the Invention

[0007] The purpose of the present invention is to provide an AR-HUD calibration and display method and system. By obtaining the distribution area, position and distortion characteristics of the actual virtual image, the off-screen image parameters and mapping relationship are dynamically adjusted to make the displayed content fully adapt to the actual virtual image space, eliminate stretching distortion, and ensure that the AR information is accurately aligned with the real object in front of the vehicle, thereby improving visual consistency and driving safety, and solving the problems raised in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: an AR-HUD calibration and display method, comprising the following steps:

[0009] Obtaining the distribution area and position information of the actual virtual image in front of the vehicle, the information of the actual virtual image is determined by projecting a test image and collecting sample images and then detecting the sample dot matrix;

[0010] Determining off-screen image parameters based on information about the actual virtual image, wherein the off-screen image parameters include off-screen image area, resolution, and field of view;

[0011] Calculating a mapping relationship between the off-screen image and the projected image based on off-screen image parameters and information about the actual virtual image;

[0012] Rendering display information under the constraints of off-screen image parameters and information of an actual virtual image, wherein the display information includes non-AR information and AR information;

[0013] The off-screen image is converted into a pre-distorted projection image through the mapping relationship, and the AR-HUD optical path is generated through the PGU to display it in front of the vehicle.

[0014] Preferably, the obtaining of the distribution area and position information of the actual virtual image in front of the vehicle includes:

[0015] Projecting a test image in front of the vehicle, wherein the test image includes a preset dot matrix;

[0016] Acquire a sample image of the test image by a viewpoint camera;

[0017] Detecting a sample dot matrix in the sample image, where the sample dot matrix has a mapping relationship with a preset dot matrix of the test image;

[0018] The distribution area and position information of the actual virtual image are determined based on the sample lattice.

[0019] Preferably, determining off-screen image parameters according to information of the actual virtual image includes:

[0020] determining a drawing area of the off-screen image based on a distribution area of the actual virtual image;

[0021] Determining the resolution or field of view of the off-screen image according to the resolution of the actual virtual image or the distance between the virtual image and the vehicle;

[0022] Off-screen image parameters are used to constrain the drawing range and scale of display information.

[0023] Preferably, the drawing of display information under the constraints of off-screen image parameters and information of the actual virtual image includes:

[0024] Drawing the non-AR information within the off-screen image area of the off-screen image;

[0025] Filter the AR information to be displayed based on the distribution area of the actual virtual image and off-screen image parameters;

[0026] The target position of the AR information is determined in the perspective space and mapped into the virtual image area of the off-screen image.

[0027] In another aspect, the present invention provides an AR-HUD calibration and display system, comprising:

[0028] An acquisition module is used to obtain the distribution area and position information of the actual virtual image in front of the vehicle. The information of the actual virtual image is determined by projecting a test image and collecting sample images and then detecting the sample dot matrix;

[0029] a processing module, configured to determine off-screen image parameters according to information of the actual virtual image, wherein the off-screen image parameters include an off-screen image area, a resolution, and a field of view;

[0030] A mapping module, configured to calculate a mapping relationship between the off-screen image and the projected image based on off-screen image parameters and information about the actual virtual image;

[0031] a rendering module, configured to render display information under the constraints of off-screen image parameters and information of an actual virtual image, wherein the display information includes non-AR information and AR information;

[0032] The display module is used to convert the off-screen image into a pre-distorted projection image through the mapping relationship, and display it in front of the vehicle through the PGU generation and AR-HUD optical path.

[0033] Preferably, the acquisition module includes:

[0034] A projection unit, used for projecting a test image containing a preset dot matrix toward the front of the vehicle;

[0035] an acquisition unit, configured to acquire a sample image of the test image through a viewpoint camera;

[0036] The detection unit is used to extract a sample dot matrix from the sample image and determine the distribution area and position information of the actual virtual image based on the sample dot matrix.

[0037] Preferably, the processing module includes:

[0038] an area determination unit, configured to determine a drawing area of an off-screen image based on a distribution area of an actual virtual image;

[0039] a parameter calculation unit, configured to determine a resolution or a field of view of an off-screen image based on a resolution of an actual virtual image or a distance between the virtual image and the vehicle;

[0040] The constraint unit is used to associate off-screen image parameters with information of actual virtual images to form constraint conditions for display content.

[0041] Preferably, the drawing module includes:

[0042] a non-AR drawing unit, configured to draw non-AR information in the off-screen image area of the off-screen image to ensure its integrity and distortion-free;

[0043] An AR screening unit, used to screen the AR information to be displayed according to the distribution area of the actual virtual image;

[0044] The AR mapping unit is used to determine the target position of the AR information in the perspective space and map it into the virtual image area of the off-screen image through a mapping relationship.

[0045] Preferably, the display module includes:

[0046] A pre-distortion unit, configured to convert an off-screen image into a pre-distorted projection image through a mapping relationship;

[0047] A generating unit, configured to generate a pre-distorted projection image through a PGU;

[0048] The optical path display unit is used to project the pre-distorted projection image onto the front windshield of the vehicle through the AR-HUD optical path.

[0049] Technical effects and advantages of the present invention: Compared with the existing technology, the AR-HUD calibration and display method and system proposed in the present invention have the following advantages:

[0050] The present invention achieves multi-dimensional technological breakthroughs by obtaining the distribution area, position and distortion characteristics of the actual virtual image in front of the vehicle, and dynamically adjusting the off-screen image parameters and mapping relationship: on the one hand, it eliminates the problem of missing and incomplete display content caused by the discrepancy between the actual range of the virtual image and the design parameters, ensuring that non-AR information is completely presented in the virtual image area; on the other hand, it suppresses the stretching distortion caused by traditional compression strategies, maintains the size of AR information unchanged and the aspect ratio of non-AR information unchanged; at the same time, through the drawing mechanism of joint constraints of virtual and real parameters, it significantly improves the fitting accuracy of AR targets and real environments, solves the problem of virtual and real misalignment; enhances the system's adaptability to hardware errors and environmental changes; finally, through the coordinated optimization of the off-screen image and the actual virtual image area, it reduces redundant calculations, improves the PGU operation efficiency, and achieves a high-precision, low-distortion, adaptive AR-HUD display effect, taking into account both safety and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flowchart of an AR-HUD calibration and display method in Example 1 of the present invention;

[0052] Figure 2 This is a block diagram of an AR-HUD calibration and display system in Example 2 of the present invention;

[0053] Figure 3 This is a schematic diagram of the AR-HUD system in Example 3 of the present invention;

[0054] Figure 4 This is an example diagram of the original projection image in Example 3 of the present invention;

[0055] Figure 5 This is an example diagram of the dot matrix under the viewpoint camera in Example 3 of the present invention;

[0056] Figure 6 This is an example diagram of the AR-HUD virtual image position in Example 3 of the present invention;

[0057] Figure 7 Schematic diagram of the AR-HUD display area and its maximum inscribed rectangle in Example 3 of the present invention;

[0058] Figure 8 This is an example diagram of not compressing the projected image in embodiment 3 of the present invention;

[0059] Figure 9 This is an example diagram of compressing a projected image into a maximum inscribed rectangle in embodiment 3 of the present invention;

[0060] Figure 10 This is an example diagram of the method for obtaining the AR information drawing position in Example 3 of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0062] Example 1

[0063] The present invention provides Figure 1 The AR-HUD calibration and display method shown here obtains the distribution area, position, and distortion characteristics of the actual virtual image, dynamically adjusts the off-screen image parameters and mapping relationship, and makes the displayed content fully adapt to the actual virtual image space, eliminates stretching distortion, and ensures that the AR information is accurately aligned with the physical object in front of the vehicle, thereby improving visual consistency and driving safety. The details are as follows:

[0064] In this embodiment, the AR-HUD calibration and display method includes the following steps:

[0065] Step 1: Obtain the distribution area and position information of the actual virtual image in front of the vehicle. The information of the actual virtual image is determined by projecting a test image, collecting sample images, and then detecting the sample dot matrix. Specifically, it includes:

[0066] A test image is projected in front of the vehicle, the test image including a preset dot matrix; a sample image of the test image is captured using a viewpoint camera; a sample dot matrix in the sample image is detected, and the correspondence between the sample dot matrix and the preset dot matrix of the test image can determine a mapping relationship between the off-screen image and the projected image; and the distribution area and position information of the actual virtual image are determined based on the sample dot matrix.

[0067] Step 2: Determine off-screen image parameters based on the actual virtual image information, wherein the off-screen image parameters include off-screen image area, resolution, and field of view angle; specifically, the following steps are performed:

[0068] The drawing area of the off-screen image is determined based on the distribution area of the actual virtual image; the resolution or field of view of the off-screen image is determined according to the resolution of the actual virtual image or the distance between the virtual image and the vehicle; the off-screen image parameters are used to constrain the drawing range and proportion of the display information.

[0069] Step 3: Calculating the mapping relationship between the off-screen image and the projected image based on the off-screen image parameters and the actual virtual image information; specifically including:

[0070] The mapping relationship between the off-screen image and the projected image is determined based on the correspondence between the sample dot matrix and the preset dot matrix; the off-screen image is a modeling of the real virtual image in front of the vehicle in the simulated space; the mapping relationship is used to realize the conversion of the off-screen image to the projected image, which eliminates the distortion caused by the AR-HUD hardware.

[0071] Step 4: Rendering display information under the constraints of off-screen image parameters and actual virtual image information, wherein the display information includes non-AR information and AR information; specifically, the following steps are performed:

[0072] The non-AR information is drawn in the off-screen image area of the off-screen image; the AR information to be displayed is filtered according to the distribution area of the actual virtual image and the off-screen image parameters; the target position of the AR information is determined in the perspective space, and mapped to the virtual image area of the off-screen image through a mapping relationship.

[0073] Step 5: Convert the off-screen image into a pre-distorted projection image through the mapping relationship, and display it in front of the vehicle through the PGU generation and AR-HUD optical path.

[0074] This invention achieves a multi-dimensional technological breakthrough by obtaining the distribution area, position and distortion characteristics of the actual virtual image in front of the vehicle and dynamically adjusting the off-screen image parameters and mapping relationship:

[0075] On the one hand, it eliminates the problem of missing or incomplete display content caused by the actual range of the virtual image not matching the design parameters, ensuring that non-AR information is fully presented within the virtual image area;

[0076] On the other hand, it suppresses the stretching distortion caused by traditional compression strategies, maintains the size of AR information unchanged and the aspect ratio of non-AR information unchanged; at the same time, through the drawing mechanism of joint constraints of virtual and real parameters, it significantly improves the fitting accuracy of AR targets and real environments, solves the problem of virtual and real misalignment; enhances the system's adaptability to hardware errors and environmental changes; finally, through the coordinated optimization of off-screen images and actual virtual image areas, it reduces redundant calculations, improves PGU operating efficiency, and achieves high-precision, low-distortion, adaptive AR-HUD display effects, taking into account both safety and user experience.

[0077] Example 2

[0078] This embodiment provides an AR-HUD calibration and display system as shown in Figure 2, including:

[0079] The acquisition module is used to obtain the distribution area and position information of the actual virtual image in front of the vehicle. The information of the actual virtual image is determined by projecting a test image and collecting sample images and then detecting the sample dot matrix. The acquisition module specifically includes:

[0080] A projection unit is used to project a test image containing a preset dot matrix toward the front of the vehicle; an acquisition unit is used to acquire a sample image of the test image through a viewpoint camera; and a detection unit is used to extract a sample dot matrix from the sample image and determine the distribution area and position information of the actual virtual image based on the sample dot matrix.

[0081] A processing module is used to determine off-screen image parameters based on information about the actual virtual image, wherein the off-screen image parameters include off-screen image area, resolution, and field of view. The processing module specifically includes:

[0082] An area determination unit is used to determine the drawing area of the off-screen image based on the distribution area of the actual virtual image; a parameter calculation unit is used to determine the resolution or field of view of the off-screen image according to the resolution of the actual virtual image or the distance between the virtual image and the vehicle; and a constraint unit is used to associate the off-screen image parameters with the information of the actual virtual image to form constraints on the displayed content.

[0083] A mapping module, configured to calculate a mapping relationship between the off-screen image and the projected image based on off-screen image parameters and information about the actual virtual image;

[0084] A rendering module is configured to render display information under the constraints of off-screen image parameters and information about the actual virtual image, wherein the display information includes non-AR information and AR information. The rendering module specifically includes:

[0085] The non-AR drawing unit is used to draw non-AR information in the off-screen image area of the off-screen image to ensure its integrity and distortion-free. The AR screening unit is used to screen the AR information to be displayed based on the distribution area of the actual virtual image. The AR mapping unit is used to determine the target position of the AR information in the perspective space and map it to the virtual image area of the off-screen image through a mapping relationship.

[0086] The display module is used to convert the off-screen image into a pre-distorted projection image through the mapping relationship, and display it in front of the vehicle through the PGU generation and AR-HUD optical path; the display module specifically includes:

[0087] The pre-distortion unit is used to convert the off-screen image into a pre-distorted projection image through a mapping relationship; the generation unit is used to generate the pre-distorted projection image through the PGU; and the optical path display unit is used to project the pre-distorted projection image onto the front windshield of the vehicle through the AR-HUD optical path.

[0088] Example 3

[0089] This embodiment provides a new AR-HUD distortion calibration and display solution. Before introducing this solution, the following terminology is explained:

[0090] Explanation of terms:

[0091] AR-HUD (AR-Head-up-Display): The fusion of AR technology and HUD display reflects the image through the windshield, and the image is integrated with the environment outside the car, allowing the driver to see a picture that combines the displayed content with the actual environment.

[0092] Viewpoint camera: such as Figure 3 As shown, a camera is placed at the position of the driver's eyes (viewpoint) to obtain the image seen by the human eye.

[0093] Test image: When calculating pre-distortion parameters, the AR-HUD projects a checkerboard, circular dot array, and other pattern images. The virtual image of this image in front of the vehicle can reflect the distortion distribution of the AR-HUD. Figure 4 This is an example image where the original projection image is a circular speckle array.

[0094] Preset Dot Matrix: The dot matrix in the test image. If the test image is a checkerboard, the preset dot matrix consists of the coordinates of the corner points. If the test image is a circular dot matrix, the preset dot matrix consists of the coordinates of the circle center.

[0095] Sample image: The image of the virtual image captured by the viewpoint camera and presented in front of the windshield. The so-called virtual image is the image of the AR-HUD in front of the vehicle windshield. After passing through the complex optical path of the AR-HUD, the virtual image in the sample image has a certain deformation (i.e. distortion) in its spatial arrangement. Figure 5The image composed of hollow circles is an example of correlated distortion.

[0096] Sample lattice: the lattice extracted from the sample image, Figure 5 The dot matrix formed by the centers of the hollow circular spots in is an example of a sample dot matrix.

[0097] Reference dot matrix: A dot matrix that is generated based on parameters such as number and arrangement and has a clear correspondence with the test dot matrix. Its function is to provide a calibration benchmark for the sample dot matrix. Figure 5 The point matrix formed by the centers of the solid circles in the figure is an example of the reference point matrix.

[0098] Pre-distortion: Based on the distortion characteristics of the AR-HUD, the projected image is reverse-distorted in advance to offset the distortion of the AR-HUD itself, so that the human eye sees a normal image.

[0099] Pre-distortion parameters: Parameters used to pre-distort the projected image, calculated based on the relationship between the dot matrix, including pixel-to-pixel mapping or surface expression.

[0100] AR and non-AR information: AR information is information that needs to be aligned with the physical objects in front of the vehicle, including pedestrians, vehicles, and lanes. Non-AR information is information that does not need to be aligned with the physical objects in front of the vehicle, including vehicle speed, speed limit, battery level, navigation maps, etc. The size and shape of the former should not be changed after dedistortion, as this will cause alignment issues between the virtual image and the physical object, but may be missing or incomplete at the edges of the virtual image. The position and size of the latter can be changed, but it cannot be missing, incomplete, or have uneven stretching distortion.

[0101] Projected image: An image that needs to be generated by the PGU and projected and displayed by the AR-HUD.

[0102] Off-screen image: It is the modeling of the real virtual image in front of the vehicle in the data space, which is used to draw images of AR information and non-AR information. It then needs to be transformed into a projected image with the help of mapping relationships (such as pre-distortion parameters, etc.).

[0103] PGU (Picture Generation Unit): Image generation unit, used to generate images to be displayed by AR-HUD.

[0104] like Figure 6As shown in the figure, the virtual image position is fixed in AR-HUD design. However, in the actual finished product, due to errors in the hardware itself and installation, the actual virtual image position of different samples will vary from the designed position (such as virtual images ① and ② in the figure). Furthermore, hardware and installation errors can also cause distortion in the size, shape, and distribution of the virtual image. The virtual image position and distortion ultimately lead to differences in the final image of different samples.

[0105] like Figure 7 As shown in the figure below, the left and right figures are examples of virtual images displayed by two AR-HUD samples. The outer curved box is the virtual image area determined by the outermost dot matrix of the sample dot matrix. Figure 7 The outer curved lines of the left and right images in the figure show that the virtual images of different samples vary in size and shape. If the designed parameters are used to determine the off-screen image, its size and distribution will be uniform. However, directly applying these parameters to actual samples may result in incomplete or even missing displayed content.

[0106] like Figure 8 As shown, the target in the off-screen image exceeds the virtual image range (curved box). However, in actual applications, only the content within the curved box can be displayed in the virtual image in front of the vehicle, which may lead to problems such as incomplete or missing display content.

[0107] In order to ensure the integrity of the displayed content, the prior art proposes a strategy of compressing the off-screen image into the maximum inscribed rectangle of the actual virtual image. However, this strategy will lead to uneven stretching and distortion of the displayed information and affect the fit of the AR information. Figure 7 As shown in , the virtual image sizes and shapes of different samples are different, so the obtained maximum inscribed rectangle also has different sizes and shapes (such as aspect ratio). If the aspect ratio and size of the off-screen image are different from the aspect ratio and size of the maximum inscribed rectangle, the off-screen image is compressed into the maximum inscribed rectangle (such as Figure 9 In addition to causing stretching and distortion of the displayed information, it will also cause problems in the fit between the AR information and the actual object in front of the car, ultimately affecting the overall visual effect.

[0108] This method proposes a new AR-HUD distortion calibration scheme. This scheme accounts for the influence of the actual virtual image information when calculating the mapping relationship between the off-screen image (area) and the projected image. It further incorporates the actual virtual image information when drawing the off-screen image. This fundamentally eliminates the problems caused by the mismatch between design parameters and actual parameters. This effectively ensures that distortion removal does not introduce stretching distortion, that non-AR information is not lost, and that the alignment of AR information with physical objects is not affected, thereby improving visual quality and user experience.

[0109] The overall plan is as follows:

[0110] Obtain the actual parameters of the virtual image, that is, obtain information about the distribution area and position of the actual virtual image in front of the vehicle. This information will be used to determine the off-screen image parameters and constrain the drawing position of the information to be displayed. The specific acquisition method is:

[0111] First, the test image is projected and the corresponding image is captured using a viewpoint camera to obtain a sample image. Next, the sample image is detected to obtain a sample lattice. Finally, the virtual image region is determined based on the sample lattice, such as the region enclosed by the envelope defined by the outermost points as the virtual image region. Note that the virtual image parameters (e.g., the virtual image region) can be on the sample image, or on the actual virtual image obtained by further mapping relationships such as homography, or on the test image.

[0112] Determining off-screen image parameters based on actual parameters, that is, obtaining corresponding off-screen image parameters based on the actual parameters of the virtual image, these parameters may be expressed in forms that are convertible, including but not limited to, the off-screen image area of the plane where the virtual image resides, the off-screen image resolution and position of the plane where the virtual image resides, or the field of view (FOV) of the off-screen image and its distance. The area determined by the off-screen image parameters may include or partially include the area determined by the virtual image parameters, or vice versa. The off-screen image parameters are used to determine the off-screen image area, filter the information that can be displayed, and constrain the drawing location of the information to be displayed.

[0113] Calculate the mapping relationship between the off-screen image and the projected image, that is, calculate the mapping relationship between the off-screen image (also the off-screen image area) determined according to the off-screen image parameters and the projected image. This mapping relationship is calculated by means of the correspondence between the sample lattice and the preset lattice, which can realize the conversion between the off-screen image (area) and the projected image. The projected image obtained after the conversion is the pre-distorted image. Unlike existing solutions, the off-screen image of this method is obtained based on the actual virtual image rather than the design parameters (the size of the off-screen image in the AR-HUD design).

[0114] The information to be displayed is drawn based on the actual parameters of the virtual image and the parameters of the off-screen image. Specifically, the information to be displayed is drawn without distortion under the joint constraints of the actual parameters of the virtual image and the parameters of the off-screen image. Constraints on the actual parameters of the virtual image, such as the virtual image distribution area, can ensure that targets drawn within this area are effectively displayed in front of the vehicle. Constraints on the off-screen image parameters, such as the off-screen image area, can also filter targets (or potential targets) that can be displayed in front of the vehicle via the AR-HUD. The combined constraints of these two ensure that targets that need to be fully displayed (such as non-AR information) are fully displayed in front of the vehicle, while effectively filtering targets that can be displayed in front of the vehicle via the AR-HUD (such as AR information), reducing the algorithm's time consumption.

[0115] The off-screen image is transformed using the mapping relationship to obtain a pre-distorted projected image, which is then generated using the PGU and displayed in front of the vehicle through the AR-HUD optical path. Because the off-screen image area is obtained based on the actual virtual image distribution of the AR-HUD, the objects drawn into the off-screen image area are also drawn into the virtual image in front of the AR-HUD. These drawn objects are undistorted, and therefore the objects converted through the mapping relationship and displayed in front of the vehicle by the AR-HUD are also undistorted.

[0116] The specific method is as follows:

[0117] Get the virtual image area of the plane where the virtual image is located. Figure 3 As shown in the figure, binocular cameras are often used in existing production lines to test the distance, FOV, etc. of the AR-HUD virtual image. The virtual image area can be easily obtained by combining the regional separation of the virtual image in the camera. The virtual image area of the plane where the virtual image is located can be further obtained by combining the virtual image distance.

[0118] Get the off-screen image area of the plane where the virtual image is located. After getting the virtual image area of the plane where the virtual image is located, the off-screen image area can be easily obtained based on this area. In order to ensure the freshness and maximum size of the picture, the maximum inscribed rectangle method is also used here to get the off-screen image area. The virtual image area and the off-screen image area can be respectively Figure 7 Take the curved box and dotted box as examples.

[0119] Calculate the mapping relationship between the off-screen image and the projected image. There are many methods to implement this step, which will not be described in detail here. The off-screen image in this method is obtained based on the actual virtual image rather than the design parameters.

[0120] The information to be displayed is drawn according to the virtual image area and the off-screen image area of the plane where the virtual image is located. This method divides this information into non-AR information and AR information when drawing. The focus of drawing non-AR information is to ensure that its display in the virtual image in front of the vehicle is complete and without distortion. Since the information in the virtual image area can be effectively displayed on the virtual image in front of the vehicle, it is sufficient to draw the non-AR information without distortion directly on the virtual image area in the off-screen image area. The focus of drawing AR information is to determine the correspondence of the relevant targets on the off-screen image. Assuming that the relevant targets have been provided by the ADAS system and transferred to the viewpoint camera coordinate system, then as Figure 10As shown, first, a viewing cone is defined in perspective space. The origin of this viewing cone is the viewpoint camera (i.e., the driver's viewpoint), and the off-screen image area is a section of the viewing cone. Next, the objects within the viewing cone are identified, and their perspective intersections on the virtual image area of the off-screen image are determined. Finally, the corresponding AR information is drawn based on the perspective intersections on the off-screen image. Since the size and aspect ratio of the relevant objects and their corresponding AR information are not adjusted throughout this process, this information is distortion-free in the virtual image in front of the vehicle and effectively matches the real objects (targets).

[0121] The off-screen image is transformed with the help of the mapping relationship to obtain the pre-distorted projected image, which is then displayed in front of the vehicle with the help of PGU generation and AR-HUD optical path.

[0122] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An AR-HUD calibration and display method, characterized in that: The following steps are involved: Obtaining the distribution area and position information of the actual virtual image in front of the vehicle, the information of the actual virtual image is determined by projecting a test image and collecting sample images and then detecting the sample dot matrix; Determining off-screen image parameters based on information about the actual virtual image, wherein the off-screen image parameters include off-screen image area, resolution, and field of view; Calculating a mapping relationship between the off-screen image and the projected image based on off-screen image parameters and information about the actual virtual image; Rendering display information under the constraints of off-screen image parameters and information of an actual virtual image, wherein the display information includes non-AR information and AR information; The off-screen image is converted into a pre-distorted projection image through the mapping relationship, and is displayed in front of the vehicle through the PGU generation and AR-HUD optical path.

2. The AR-HUD calibration and display method according to claim 1, characterized in that: The obtaining of the distribution area and position information of the actual virtual image in front of the vehicle includes: Projecting a test image in front of the vehicle, wherein the test image includes a preset dot matrix; Acquire a sample image of the test image by a viewpoint camera; Detecting a sample dot matrix in the sample image, where the sample dot matrix has a mapping relationship with a preset dot matrix of the test image; The distribution area and position information of the actual virtual image are determined based on the sample lattice.

3. The AR-HUD calibration and display method according to claim 1, characterized in that: The determining of off-screen image parameters according to information of the actual virtual image includes: determining a drawing area of the off-screen image based on a distribution area of the actual virtual image; Determining the resolution or field of view of the off-screen image according to the resolution of the actual virtual image or the distance between the virtual image and the vehicle; Off-screen image parameters are used to constrain the drawing range and scale of display information.

4. The AR-HUD calibration and display method according to claim 1, characterized in that: The drawing of display information under the constraints of off-screen image parameters and information of the actual virtual image includes: Drawing the non-AR information within the off-screen image area of the off-screen image; Filter the AR information to be displayed based on the distribution area of the actual virtual image and off-screen image parameters; The target position of the AR information is determined in the perspective space and mapped into the virtual image area of the off-screen image.

5. An AR-HUD calibration and display system for implementing the method according to any one of claims 1 to 4, characterized in that: include: An acquisition module is used to obtain the distribution area and position information of the actual virtual image in front of the vehicle. The information of the actual virtual image is determined by projecting a test image and collecting sample images and then detecting the sample dot matrix; a processing module, configured to determine off-screen image parameters according to information of the actual virtual image, wherein the off-screen image parameters include an off-screen image area, a resolution, and a field of view; A mapping module, configured to calculate a mapping relationship between the off-screen image and the projected image based on off-screen image parameters and information about the actual virtual image; a rendering module, configured to render display information under the constraints of off-screen image parameters and information of an actual virtual image, wherein the display information includes non-AR information and AR information; The display module is used to convert the off-screen image into a pre-distorted projection image through the mapping relationship, and generate an AR-HUD optical path through the PGU to display it in front of the vehicle.

6. The AR-HUD calibration and display system according to claim 5, characterized in that: The acquisition module includes: A projection unit, used for projecting a test image containing a preset dot matrix toward the front of the vehicle; an acquisition unit, configured to acquire a sample image of the test image through a viewpoint camera; The detection unit is used to extract a sample dot matrix from the sample image and determine the distribution area and position information of the actual virtual image based on the sample dot matrix.

7. The AR-HUD calibration and display system according to claim 5, characterized in that: The processing module includes: an area determination unit, configured to determine a drawing area of an off-screen image based on a distribution area of an actual virtual image; a parameter calculation unit, configured to determine a resolution or a field of view of an off-screen image based on a resolution of an actual virtual image or a distance between the virtual image and the vehicle; The constraint unit is used to associate off-screen image parameters with information of actual virtual images to form constraint conditions for display content.

8. The AR-HUD calibration and display system according to claim 5, characterized in that: The drawing module includes: a non-AR drawing unit, configured to draw non-AR information in the off-screen image area of the off-screen image to ensure its integrity and distortion-free; An AR screening unit, used to screen the AR information to be displayed according to the distribution area of the actual virtual image; The AR mapping unit is used to determine the target position of the AR information in the perspective space and map it into the virtual image area of the off-screen image through a mapping relationship.

9. The AR-HUD calibration and display system according to claim 5, characterized in that: The display module includes: A pre-distortion unit, configured to convert an off-screen image into a pre-distorted projection image through a mapping relationship; A generating unit, configured to generate a pre-distorted projection image through a PGU; The optical path display unit is used to project the pre-distorted projection image onto the front windshield of the vehicle through the AR-HUD optical path.

Images