Augmented reality heads-up display imaging method, apparatus, device, and storage medium

Abstract

The application discloses an augmented reality head-up display imaging method and device, equipment and a storage medium. The method comprises the following steps: acquiring information of a target object, determining a view cone for imaging according to current optical-mechanical parameters, the current optical-mechanical parameters being determined according to a target image and theoretical optical-mechanical parameters of a head-up display device, the target image being an image of a calibration object, the current optical-mechanical parameters comprising a current eye point coordinate, a current virtual image resolution, a current imaging plane center point coordinate, a size of the current imaging plane and a normal line of the current imaging plane, generating a virtual image of the target object by using the view cone according to the information of the target object, and displaying the virtual image of the target object in a virtual image display area. Thus, the position deviation between an actual object and the virtual image can be reduced, the degree of the actual object and the virtual image being displayed in conjunction can be improved, and the AR experience effect can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an augmented reality head-up display imaging method, apparatus, device, and storage medium. Background Technology

[0002] Augmented Reality Head-Up Display (AR-HUD) is one type of head-up display (HUD). AR-HUD technology brings safe augmented reality head-up display functionality to the automotive industry. Its imaging principle involves creating a virtual image in front of the vehicle using the windshield, and employing 3D imaging technology to achieve an augmented reality user experience. The virtual image corresponding to the actual object can change position and size to follow the constantly changing real-world scenery, such as other vehicles, pedestrians, and intersections. It can also project important information displayed on the vehicle's instruments (such as speed, navigation information, and traffic signs) onto the windshield, allowing the driver to see the information without looking down.

[0003] In existing technologies, during the installation of AR-HUDs on vehicles, issues with installation accuracy often lead to deviations between the installation position and the preset theoretical position. This results in errors between the optical-mechanical parameters of the installed AR-HUD and the theoretical optical-mechanical parameters. Consequently, when the AR-HUD images based on the theoretical optical-mechanical parameters, a positional deviation will appear between the actual object and its corresponding virtual image, making it impossible to display correctly and resulting in poor AR effects. Summary of the Invention

[0004] This application provides an augmented reality head-up display imaging method, apparatus, device, and storage medium, which can reduce the positional deviation between the actual object and the virtual image, improve the degree of fit between the actual object and the virtual image, and enhance the AR experience.

[0005] In a first aspect, this application provides an augmented reality head-up display imaging method, comprising:

[0006] Obtain information about the target object;

[0007] The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of the calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the current imaging plane center point coordinates, the current imaging plane size, and the current imaging plane normal.

[0008] Based on the information of the target object, a virtual image of the target object is generated using the view frustum;

[0009] A virtual image of the target object is displayed in the virtual image display area.

[0010] Optionally, the method further includes:

[0011] Obtain the theoretical optomechanical parameters of the head-up display device;

[0012] The calibration object is displayed in the virtual image display area;

[0013] The target image is obtained by capturing the calibration object using a camera device;

[0014] The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters.

[0015] Optionally, the calibration object includes multiple calibration points, and determining the current optomechanical parameters based on the target image and the theoretical optomechanical parameters includes:

[0016] Obtain the pixel coordinates of multiple calibration points in the target image;

[0017] The current optomechanical parameters are determined based on the pixel coordinates of multiple calibration points in the target image and the theoretical optomechanical parameters.

[0018] Optionally, before displaying the calibration object in the virtual image display area, the method further includes:

[0019] Receive optical-mechanical parameter calibration commands;

[0020] The step of obtaining the pixel coordinates of multiple calibration points in the target image includes:

[0021] Receive the pixel coordinates of the plurality of calibration points sent by the camera device.

[0022] Optionally, the theoretical optomechanical parameters include theoretical virtual image distance, theoretical eye point coordinates, and theoretical virtual image resolution. Determining the current optomechanical parameters based on the pixel coordinates of multiple calibration points in the target image and the theoretical optomechanical parameters includes:

[0023] Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, determine the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the plane where the target image is located;

[0024] The normal to the plane containing the target image is determined as the normal to the current imaging plane, the theoretical eye point coordinates are determined as the current eye point coordinates, and the theoretical virtual image resolution is determined as the current virtual image resolution.

[0025] Optionally, determining the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal to the plane containing the target image based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance includes:

[0026] Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, the coordinates of the center calibration point of the multiple calibration points in the spatial rectangular coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane are determined. The target plane is the plane whose distance from the camera device is the theoretical virtual image distance, and the target calibration point is the other calibration point besides the center calibration point.

[0027] Based on the coordinates of the center calibration point in the spatial rectangular coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane, the normal of the plane where the target image is located and the size of the target image are determined.

[0028] The coordinates of the center calibration point in the spatial rectangular coordinate system are determined as the coordinates of the center point of the current imaging plane, and the size of the target image is determined as the size of the current imaging plane.

[0029] Optionally, determining the coordinates of the center calibration point of the plurality of calibration points in a Cartesian coordinate system and the physical coordinates of the projection point of the target calibration point onto the target plane based on the pixel coordinates of the plurality of calibration points in the target image and the theoretical virtual image distance includes:

[0030] Using the theoretical virtual image distance as the Z-axis coordinate of the center calibration point in a spatial rectangular coordinate system, and based on the coordinate transformation formula and the pixel coordinates of the center calibration point, the X-axis and Y-axis coordinates of the center calibration point in a spatial rectangular coordinate system are obtained.

[0031] For each target calibration point, the theoretical virtual image distance is used as the Z-axis coordinate of the target calibration point under the physical coordinates of the projection point on the target plane. Based on the coordinate transformation formula and the pixel coordinates of the target calibration point, the X-axis coordinates and Y-axis coordinates of the target calibration point under the physical coordinates of the projection point on the target plane are obtained.

[0032] Secondly, this application provides an augmented reality head-up display imaging device, comprising:

[0033] The acquisition module is used to obtain information about the target object.

[0034] The determination module is used to determine the viewing cone for imaging based on the current optomechanical parameters. The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of the calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the current imaging plane center point coordinates, the current imaging plane size, and the current imaging plane normal.

[0035] A generation module is used to generate a virtual image of the target object using the view frustum based on the information of the target object;

[0036] The display module is used to display the virtual image of the target object in the virtual image display area.

[0037] Thirdly, this application provides a head-up display device, comprising:

[0038] Processor; and

[0039] Memory for storing the executable instructions of the processor;

[0040] The processor is configured to perform the augmented reality head-up display imaging method described in the first aspect or any of the possible implementations of the first aspect.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the augmented reality head-up display imaging method described in the first aspect or any of the possible implementations of the first aspect.

[0042] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the augmented reality head-up display imaging method described in the first aspect or any of the possible implementations of the first aspect.

[0043] The augmented reality head-up display imaging method, apparatus, device, and storage medium provided in this application first acquire information about the target object, then determine the viewing cone used for imaging based on the current optomechanical parameters. These current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of a calibration object. Since the current optomechanical parameters used for imaging are determined based on the image of the calibration object and the theoretical optomechanical parameters—that is, re-determined and calibrated optomechanical parameters based on the image of the calibration object and the theoretical optomechanical parameters—the current optomechanical parameters used for imaging are relatively accurate. Therefore, when imaging based on more accurate optomechanical parameters, the positional deviation between the target object (such as a real object or person) and the virtual image can be reduced, improving the degree of fit between the target object and the virtual image and enhancing the AR experience. Attached Figure Description

[0044] Figure 1 This is a schematic diagram illustrating an application scenario of the augmented reality head-up display imaging method provided in the embodiments of this application;

[0045] Figure 2 A flowchart illustrating an augmented reality head-up display imaging method provided in this application embodiment;

[0046] Figure 3 This is a schematic diagram of a viewing cone;

[0047] Figure 4 A schematic diagram illustrating the theoretical optomechanical parameters of a head-up display device provided in an embodiment of this application;

[0048] Figure 5 A flowchart illustrating an augmented reality head-up display imaging method provided in this application embodiment;

[0049] Figure 6 A schematic diagram of a target image corresponding to a 3*3 dot matrix pattern provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of the imaging plane coordinate system;

[0051] Figure 8 This is a schematic diagram of the coordinate system of the camera device;

[0052] Figure 9 A schematic diagram of the target image corresponding to a 3x3 dot matrix image of the calibration object;

[0053] Figure 10 A schematic diagram of a spatial rectangular coordinate system established using P2' and P8' as examples, provided for embodiments of this application;

[0054] Figure 11 This application provides an AR display schematic diagram after calibrating the optomechanical parameters of a head-up display device;

[0055] Figure 12 This is a schematic diagram of an AR display after the optomechanical parameters of a head-up display device have not been calibrated, as provided in an embodiment of this application.

[0056] Figure 13 This is a schematic diagram of the structure of an augmented reality head-up display imaging device provided in an embodiment of this application;

[0057] Figure 14 This is a schematic diagram of the structure of a head-up display device provided in an embodiment of this application. Detailed Implementation

[0058] Embodiments of this application are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting it.

[0059] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0060] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0061] 1. Augmented Reality (AR), also known as Mixed Reality, uses computer technology to apply virtual information to the real world. The real environment and virtual objects are overlaid in real time on the same screen or in the same space, existing simultaneously. Augmented Reality provides information that is normally different from what humans can perceive. It not only displays information from the real world but also simultaneously displays virtual information, with the two types of information complementing and overlaying each other.

[0062] 2. Eye point position: The driver's point of view. The eye point can be the midpoint between the driver's left and right eye points, or the intersection of the driver's center line and the line connecting the centers of the driver's left and right eye ellipses.

[0063] In existing technologies, during the installation of AR-HUDs on vehicles, issues with installation accuracy often lead to deviations between the installation position and the preset theoretical position. This results in errors between the optical-mechanical parameters of the installed AR-HUD and the theoretical optical-mechanical parameters. Consequently, when the AR-HUD images based on the theoretical optical-mechanical parameters, a positional deviation will appear between the actual object and its corresponding virtual image, making it impossible to display correctly and resulting in poor AR effects.

[0064] To address this issue, this application provides an augmented reality head-up display imaging method, apparatus, device, and storage medium. The method involves first acquiring information about the target object, then determining the viewing cone for imaging based on current optomechanical parameters. These parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of a calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal to the current imaging plane. Since the current optomechanical parameters used for imaging are determined based on the image of the calibration object and the theoretical optomechanical parameters—that is, re-determined and calibrated optomechanical parameters based on the image of the calibration object and the theoretical optomechanical parameters—the current optomechanical parameters used for imaging are relatively accurate. Therefore, when imaging based on more accurate optomechanical parameters, the positional deviation between the target object (such as a real object or person) and the virtual image can be reduced, improving the degree of fit between the target object and the virtual image and enhancing the AR experience.

[0065] Next, examples of application scenarios involved in the embodiments of this application will be provided.

[0066] The augmented reality head-up display imaging method provided in this application embodiment can be applied to at least the following application scenarios, which will be described below in conjunction with the accompanying drawings.

[0067] For example, Figure 1 This is a schematic diagram illustrating an application scenario of the augmented reality head-up display imaging method provided in the embodiments of this application, such as... Figure 1 As shown, the diagram relates to a head-up display (HUD) 1 and a windshield 2. The HUD 1 may include a projection device, a reflector, and a projection mirror 3 (such as an aspherical mirror). In one embodiment, the HUD 1 may be installed inside the driver's cabin of a vehicle, with the windshield 2 positioned opposite the driver's seat. Figure 1 Image 4 in the diagram is an actual AR display effect. The head-up display device 1 generates a beam of light for creating a virtual image, and the projection lens 3 reflects this beam onto the windshield 2. After being reflected by the windshield 2, the beam enters the driver's eyes (i.e., the eye point). The driver can see the displayed virtual image on the windshield 2.

[0068] Optionally, the head-up display device 1 can be an AR-HUD, and the virtual image projected onto the windshield 2 by the AR-HUD can be located directly in front of the driver's field of vision. Compared with traditional HUDs, AR-HUDs, in addition to projecting important information displayed on the instrument panel during vehicle operation (such as vehicle speed, navigation information, traffic signs, etc.), also have augmented reality capabilities, and the virtual image can change its position and size according to the constantly changing real-world scenery such as vehicles in front, pedestrians, and intersections. Optionally, the AR-HUD can display an AR layer and an indicator / signage layer on the windshield 2.

[0069] The augmented reality head-up display imaging method provided in this application embodiment can be used in a head-up display device 1. After the head-up display device 1 is installed in a vehicle or other equipment, the optical-mechanical parameters of the head-up display device are calibrated (also called corrected). After calibration, the optical-mechanical parameters of the head-up display device can be made more accurate. Thus, during use, the head-up display device can reduce the positional deviation between the target object and the virtual image, improve the degree of fit between the target object and the virtual image, and improve the AR experience.

[0070] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0071] Figure 2 This is a flowchart illustrating an augmented reality head-up display imaging method provided in this application embodiment. The executing entity in this embodiment can be a head-up display device, such as... Figure 2 As shown, the method in this embodiment may include:

[0072] S101. Obtain information about the target object.

[0073] Optionally, the target object information may include the target object's location information and attribute information. Alternatively, the target object information may include the target object's attribute information. Specifically, the target object refers to the actual object or person to be imaged, such as a vehicle in front, a pedestrian, an intersection, a street lamp, etc. The target object's location information refers to its three-dimensional coordinates in the vehicle coordinate system, and the target object's attribute information refers to the type of the target object, such as whether the target object is a pedestrian, road sign, intersection, navigation arrow indicating the direction of travel, or vehicle in front, etc.

[0074] S102. Determine the viewing cone for imaging based on the current optomechanical parameters. The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of the calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the current imaging plane center point coordinates, the current imaging plane size, and the current imaging plane normal.

[0075] S103. Based on the information of the target object, generate a virtual image of the target object using a view frustum.

[0076] Specifically, after obtaining the information of the target object, a virtual image of the target object can be generated based on the current optical engine parameters and the information of the target object.

[0077] The current optomechanical parameters include the current eye point coordinates, current virtual image resolution, current imaging plane center point coordinates, current imaging plane size, and current imaging plane normal. Specifically, based on the current optomechanical parameters and the target object information, a virtual image of the target object is generated. This can be achieved by first determining the viewing frustum used for imaging using the current eye point coordinates, current virtual image resolution, current imaging plane center point coordinates, current imaging plane size, and current imaging plane normal. Then, based on the target object information, the virtual image of the target object is generated using the viewing frustum.

[0078] The image cone used for imaging is a regular square pyramid. Figure 3 This is a schematic diagram of a viewing cone, such as... Figure 3 As shown, Figure 3 The near plane is the imaging plane, and its dimensions are the same as those of the imaging plane. The coordinates of the center point of the near plane are the same as those of the center point of the imaging plane, and the normal to the near plane is the same as the normal to the imaging plane. The far plane is the bottom surface of the frustum, and the spatial region between the near and far planes is the imaging region. When the target object is located in the imaging region, a virtual image of the target object can be generated on the near plane.

[0079] S104. Display the virtual image of the target object in the virtual image display area.

[0080] Optionally, the method of this embodiment may further include a process of determining the current optomechanical parameters. Before S101, the method of this embodiment may further include:

[0081] S105. Obtain the theoretical optomechanical parameters of the head-up display device.

[0082] Specifically, after a head-up display (HUD) is installed in a vehicle, its optomechanical parameters generally need to be calibrated. This involves calibrating the theoretical optomechanical parameters of the HUD to eliminate errors and improve its accuracy.

[0083] Specifically, theoretical optomechanical parameters can be obtained through a head-up display (HUD). These parameters can include at least the theoretical eye-point coordinates, theoretical virtual image distance (VID), and theoretical virtual image resolution. Optionally, they may also include the theoretical field of view (FOV), theoretical look-down angle (LDA), and theoretical heading angle. These parameters can be obtained from the optomechanical manufacturer or through manual measurement, such as using a robotic arm to take pictures or manually measuring a target. If obtained manually, the parameters can be input into the HUD. The method for obtaining the theoretical optomechanical parameters from the HUD is not limited in this embodiment. The theoretical virtual image distance is the distance from the theoretical eye-point to the virtual image.

[0084] The following is combined with Figure 4 This section details some parameters from the theoretical optomechanical parameters of the head-up display device. Figure 4 A schematic diagram of the theoretical optomechanical parameters of a head-up display device provided in an embodiment of this application is shown below. Figure 4 As shown, the theoretical optomechanical parameters in this embodiment may include theoretical eye-point coordinates, theoretical virtual image distance, theoretical virtual image resolution, theoretical field of view, theoretical downward viewing angle, and theoretical heading angle. The theoretical eye-point coordinates are... Figure 4 The coordinates (x, y, z) at the "Eye" point shown, the virtual image, and the center point of the virtual image are as follows: Figure 4 As shown, the theoretical virtual image distance is Figure 4 The VID shown here, theoretically, has the following perspective: Figure 4 The LDA shown in the figure.

[0085] In practical applications, the theoretical optomechanical parameters used for imaging also include the coordinates of the center point of the theoretical imaging plane, the dimensions of the theoretical imaging plane, and the normal to the theoretical imaging plane. These parameters are calculated based on the obtained theoretical eye-point coordinates, theoretical virtual image distance, theoretical downward viewing angle, and theoretical heading angle. Specifically, they can be calculated using the following formula:

[0086] The x-axis coordinate of the center point of the theoretical imaging plane is: x-axis coordinate in the theoretical eye point coordinates - VID.

[0087] The y-coordinate of the center point of the theoretical imaging plane is:

[0088] The z-axis coordinate of the theoretical imaging plane center point is: the z-axis coordinate in the theoretical eyepoint coordinate system.

[0089] Where VID is the theoretical virtual image distance, Yaw is the theoretical heading angle, and LDA is the theoretical downward viewpoint.

[0090] That is, the coordinates of the center point of the theoretical imaging plane are obtained as: (x-axis coordinate in the theoretical eyepoint coordinates - VID, The z-axis coordinate in the theoretical eye point coordinate system ).

[0091] The dimensions of the theoretical imaging plane can be calculated using the following formula, based on the theoretical eyepoint coordinates, the coordinates of the center point of the theoretical imaging plane, and the theoretical field of view:

[0092] The theoretical width of the imaging plane is: The theoretical height of the imaging plane is:

[0093] Where P is the theoretical imaging plane distance, which is determined based on the coordinates of the center point of the theoretical imaging plane and the coordinates of the theoretical eye point. Specifically, it can be the distance between the center point of the theoretical imaging plane and the theoretical eye point. For example, if the theoretical eye point is Eye and the center point of the theoretical imaging plane is Target, then the theoretical imaging plane distance is... FOVH is the lateral component of the theoretical field of view, and FOVV is the longitudinal component of the theoretical field of view.

[0094] Once the coordinates of the center point of the theoretical imaging plane and the dimensions of the theoretical imaging plane are obtained, the normal to the theoretical imaging plane can be obtained based on these coordinates.

[0095] In the existing technology, during the installation of AR-HUD on a vehicle, the installation position of AR-HUD often deviates from the preset theoretical position due to installation accuracy issues. This results in errors between the optomechanical parameters of the installed AR-HUD and the aforementioned theoretical optomechanical parameters. Therefore, it is necessary to calibrate the optomechanical parameters, i.e., re-determine them.

[0096] S106. Display the calibration object in the virtual image display area.

[0097] The calibration object can be a preset graphic; optionally, it can be a raster image or a checkerboard pattern, and includes multiple calibration points. If the calibration object is a raster image, then the multiple calibration points in the calibration object are each point in the raster image. If the calibration object is a checkerboard pattern, then the multiple calibration points in the calibration object are the vertices of each cell in the checkerboard pattern.

[0098] Specifically, the head-up display device may have pre-stored calibration objects. In one implementable manner, before S106, it may further include: receiving an optomechanical parameter calibration command, and upon receiving the command, displaying the calibration objects in the virtual image display area. For example, when a user turns on a calibration switch on the head-up display device, the head-up display device can receive the optomechanical parameter calibration command, that is, display the calibration objects in the virtual image display area.

[0099] S107. Take a picture of the calibration object with a camera device to obtain a target image.

[0100] Specifically, the method of this embodiment can be implemented in conjunction with a camera device. For example, an existing camera device on the vehicle can be used, or a camera device can be installed on the vehicle during calibration. Specifically, the camera device can be installed at the driver's actual eye level. The camera device can be connected to the head-up display device via wired or wireless communication.

[0101] S108. Determine the current optomechanical parameters based on the target image and theoretical optomechanical parameters.

[0102] Optionally, the calibration object includes multiple calibration points. The current optomechanical parameters are determined based on the target image and theoretical optomechanical parameters. Specifically, this can be:

[0103] Obtain the pixel coordinates of multiple calibration points in the target image, and determine the current optomechanical parameters based on the pixel coordinates of the multiple calibration points in the target image and the theoretical optomechanical parameters.

[0104] In one feasible implementation, after the head-up display device displays the calibration object in the virtual image display area, it sends a request to the camera transposition device to acquire the pixel coordinates of the calibration points. Upon receiving the request, the camera device captures an image of the calibration object and acquires the pixel coordinates of multiple calibration points in the image, then sends these pixel coordinates to the head-up display device. Correspondingly, acquiring the pixel coordinates of multiple calibration points in the target image can be achieved by receiving the pixel coordinates of the multiple calibration points sent by the camera device.

[0105] In another feasible approach, after the head-up display device displays the calibration object in the virtual image display area, a camera transposition is manually controlled to capture an image of the calibration object. After capturing the image, the camera transposition can obtain the pixel coordinates of multiple calibration points in the image of the calibration object and send these pixel coordinates to the head-up display device. Correspondingly, obtaining the pixel coordinates of multiple calibration points in the target image can be achieved by receiving the pixel coordinates of multiple calibration points sent by the camera device.

[0106] Optionally, the imaging device can be a camera or a webcam. Before the imaging device captures an image of the calibration object, its intrinsic and extrinsic parameters can be calibrated. For example, if the imaging device is a camera, the camera's intrinsic parameters may include focal length, optical axis center, and resolution, while its extrinsic parameters may include the camera's position, field of view (FOV), lower angle of view (LDA), and heading angle. By calibrating the imaging device's intrinsic and extrinsic parameters, the accuracy of the imaging device can be improved.

[0107] Specifically, in this embodiment, the theoretical imaging plane center point coordinates, theoretical imaging plane size, and theoretical imaging plane normal in the theoretical optomechanical parameters of the head-up display device are calibrated. The eye point coordinates and virtual image resolution can be directly obtained from the theoretical optomechanical parameters in S104.

[0108] In one feasible approach, the current optomechanical parameters are determined based on the pixel coordinates of multiple calibration points in the target image and the theoretical optomechanical parameters. Specifically, this may include:

[0109] S1081. Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, determine the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the plane where the target image is located.

[0110] S1082. Determine the normal of the plane containing the target image as the normal of the current imaging plane, determine the theoretical eye point coordinates as the current eye point coordinates, and determine the theoretical virtual image resolution as the current virtual image resolution.

[0111] Optionally, based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the plane containing the target image are determined. Specifically, this can be:

[0112] Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, the coordinates of the center calibration point in a Cartesian coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane are determined. The target plane is the plane whose distance from the imaging device is equal to the theoretical virtual image distance, and the target calibration points are the calibration points other than the center calibration point. Based on the coordinates of the center calibration point in a Cartesian coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane, the normal to the plane containing the target image and the size of the target image are determined. The coordinates of the center calibration point in a Cartesian coordinate system are used as the coordinates of the center point of the current imaging plane, and the size of the target image is used as the size of the current imaging plane.

[0113] Optionally, based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, determine the coordinates of the center calibration point in a Cartesian coordinate system, as well as the physical coordinates of the target calibration point projected onto the target plane. Specifically, this may include:

[0114] S10811. Using the theoretical virtual image distance as the center calibration point, the Z-axis coordinates in the spatial rectangular coordinate system are obtained. Based on the coordinate transformation formula and the pixel coordinates of the center calibration point, the X-axis and Y-axis coordinates of the center calibration point in the spatial rectangular coordinate system are obtained.

[0115] Specifically, using the theoretical virtual image distance as the center calibration point, the Z-axis coordinates in the spatial rectangular coordinate system are obtained. Based on the conversion formula between the target image coordinate system and the spatial rectangular coordinate system and the pixel coordinates of the center calibration point, the X-axis and Y-axis coordinates of the center calibration point in the spatial rectangular coordinate system are obtained.

[0116] S10812. For each target calibration point, the theoretical virtual image distance is used as the Z-axis coordinate of the target calibration point in the physical coordinates of the projection point on the target plane. Based on the coordinate transformation formula and the pixel coordinates of the target calibration point, the X-axis coordinates and Y-axis coordinates of the target calibration point in the physical coordinates of the projection point on the target plane are obtained.

[0117] Specifically, for each target calibration point, the theoretical virtual image distance is used as the Z-axis coordinate of the target calibration point in the physical coordinates of the projection point on the target plane. Based on the conversion formula between the coordinate system of the target image and the spatial rectangular coordinate system and the pixel coordinates of the target calibration point, the X-axis coordinates and Y-axis coordinates of the target calibration point in the physical coordinates of the projection point on the target plane are obtained.

[0118] Specifically, based on the coordinates of the center calibration point in a spatial rectangular coordinate system and the physical coordinates of the target calibration point projected onto the target plane, the normal to the plane containing the target image and the dimensions of the target image are determined, which may include:

[0119] S1. Select the first target calibration point and the second target calibration point sequentially from the target calibration points. Calculate the coordinates of the first target calibration point and the second target calibration point in the spatial rectangular coordinate system in the following manner, until the coordinates of each target calibration point in the spatial rectangular coordinate system are calculated. The first target calibration point, the second target calibration point, and the center calibration point are on a straight line.

[0120] S2. Establish a spatial rectangular coordinate system with the center calibration point as the origin, the line connecting the coordinate point of the camera device in the spatial rectangular coordinate system and the center calibration point as the Y-axis, and the line connecting the physical coordinates of the first target calibration point and the second target calibration point as the X-axis.

[0121] S3. In the spatial rectangular coordinate system, based on the coordinates of the camera device in the spatial rectangular coordinate system, the physical coordinates of the projection point of the first target calibration point on the target plane, and the physical coordinates of the projection point of the second target calibration point on the target plane, the coordinates of the first target calibration point in the spatial rectangular coordinate system and the coordinates of the second target calibration point in the spatial rectangular coordinate system are calculated.

[0122] S4. Based on the coordinates of the center calibration point and the target calibration point in the spatial rectangular coordinate system, determine the plane equation of the plane containing the target image and the dimensions of the target image. Based on the plane equation of the plane containing the target image, obtain the normal to the plane containing the target image.

[0123] In this embodiment, the theoretical eye point coordinates and theoretical virtual image resolution in the theoretical optomechanical parameters continue to be used as the eye point coordinates and virtual image resolution in the current optomechanical parameters. The coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the current imaging plane are three calibrated parameters, which are relatively accurate parameters.

[0124] The augmented reality head-up display imaging method provided in this embodiment first acquires information about the target object, then determines the viewing cone used for imaging based on the current optomechanical parameters. These current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of a calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the coordinates of the current imaging plane center point, the size of the current imaging plane, and the normal to the current imaging plane. Since the current optomechanical parameters used for imaging are determined based on the image of the calibration object and the theoretical optomechanical parameters—that is, the re-determined and calibrated optomechanical parameters—the current optomechanical parameters used for imaging are relatively accurate. Therefore, when imaging based on more accurate optomechanical parameters, the positional deviation between the target object (such as a real object or person) and the virtual image can be reduced, improving the degree of fit between the target object and the virtual image and enhancing the AR experience.

[0125] The following describes the detailed process of the augmented reality head-up display imaging method provided in this application, with reference to a specific embodiment.

[0126] Figure 5 This is a flowchart illustrating an augmented reality head-up display imaging method provided in this application embodiment. The executing entity in this embodiment can be a head-up display device, such as... Figure 5 As shown, the method in this embodiment may include:

[0127] S201. Obtain the theoretical optomechanical parameters of the head-up display device.

[0128] Specifically, theoretical optomechanical parameters can be obtained through a head-up display (HUD). These parameters can include at least theoretical eye-point coordinates, theoretical virtual image distance, and theoretical virtual image resolution. Optionally, they may also include theoretical field of view (FOV), theoretical downward angle of view (LDA), and theoretical heading angle. These parameters can be obtained from the optomechanical manufacturer or through manual measurement, such as using a robotic arm to take photographs or manually measuring a target. If obtained manually, the parameters can be input into the HUD. The methods for obtaining the theoretical optomechanical parameters from the HUD are not limited in this embodiment.

[0129] In practical applications, the theoretical optomechanical parameters used for imaging also include the coordinates of the center point of the theoretical imaging plane, the dimensions of the theoretical imaging plane, and the normal to the theoretical imaging plane. These parameters are calculated based on the obtained theoretical eye-point coordinates, theoretical virtual image distance, theoretical downward viewing angle, and theoretical heading angle. Specifically, they can be calculated using the following formula:

[0130] The x-axis coordinate of the center point of the theoretical imaging plane is: x-axis coordinate in the theoretical eye point coordinates - VID.

[0131] The y-coordinate of the center point of the theoretical imaging plane is:

[0132] The z-axis coordinate of the theoretical imaging plane center point is: the z-axis coordinate in the theoretical eyepoint coordinate system.

[0133] Where VID is the theoretical virtual image distance, Yaw is the theoretical heading angle, and LDA is the theoretical downward viewpoint.

[0134] That is, the coordinates of the center point of the theoretical imaging plane are obtained as: (x-axis coordinate in the theoretical eyepoint coordinates - VID, The z-axis coordinate in the theoretical eye point coordinate system ).

[0135] The dimensions of the theoretical imaging plane can be calculated using the following formula, based on the theoretical eyepoint coordinates, the coordinates of the center point of the theoretical imaging plane, and the theoretical field of view:

[0136] The theoretical width of the imaging plane is: The theoretical height of the imaging plane is:

[0137] Where P is the theoretical imaging plane distance, which is determined based on the coordinates of the center point of the theoretical imaging plane and the coordinates of the theoretical eye point. Specifically, it can be the distance between the center point of the theoretical imaging plane and the theoretical eye point. For example, if the theoretical eye point is Eye and the center point of the theoretical imaging plane is Target, then the theoretical imaging plane distance is... FOVH is the lateral component of the theoretical field of view, and FOVV is the longitudinal component of the theoretical field of view.

[0138] Once the coordinates of the center point of the theoretical imaging plane and the dimensions of the theoretical imaging plane are obtained, the normal to the theoretical imaging plane can be obtained based on these coordinates.

[0139] In the existing technology, during the installation of AR-HUD on a vehicle, the installation position of AR-HUD often deviates from the preset theoretical position due to installation accuracy issues. This results in errors between the optomechanical parameters of the installed AR-HUD and the aforementioned theoretical optomechanical parameters. Therefore, it is necessary to calibrate the optomechanical parameters, i.e., re-determine them.

[0140] S202. Display the calibration object in the virtual image display area.

[0141] The calibration object can be a preset graphic; optionally, it can be a raster image or a checkerboard pattern, and the image to be calibrated includes multiple calibration points. If the calibration object is a raster image, then the multiple calibration points in the calibration object are each point in the raster image. If the calibration object is a checkerboard pattern, then the multiple calibration points in the calibration object are the vertices of each cell in the checkerboard pattern.

[0142] In this embodiment, a 3*3 dot matrix diagram of the calibration object is used as an example for explanation. This 3*3 dot matrix diagram has 9 calibration points.

[0143] Specifically, the head-up display device can pre-store calibration objects. In one possible implementation, S202 can be: receiving an optomechanical parameter calibration command and displaying the calibration objects in the virtual image display area. For example, when a user turns on a calibration switch on the head-up display device, the head-up display device can receive the optomechanical parameter calibration command, that is, display the calibration objects in the virtual image display area.

[0144] S203. Obtain the pixel coordinates of multiple calibration points in the target image, which is acquired by a camera device.

[0145] Specifically, a target image is obtained by photographing the calibration object using a camera device. In this embodiment, a 3x3 dot matrix image of the calibration object is used as an example for explanation. Figure 6 This is a schematic diagram of a target image corresponding to a 3*3 dot matrix image provided in an embodiment of this application. The coordinate system of this target image is a pixel coordinate system, also known as a photo coordinate system, with the upper left corner of the photo as the origin, the horizontal axis to the right as the u-axis, and the vertical axis downwards as the v-axis, with pixels as the unit. Figure 6 As shown, the virtual image has 9 calibration points, and the pixel coordinates of each calibration point are (u, v).

[0146] S204. Using the theoretical virtual image distance as the center calibration point, the Z-axis coordinates in the spatial rectangular coordinate system are obtained. Based on the conversion formula between the target image coordinate system and the spatial rectangular coordinate system and the pixel coordinates of the center calibration point, the X-axis and Y-axis coordinates of the center calibration point in the spatial rectangular coordinate system are obtained.

[0147] Specifically, the above process involves four coordinate systems: pixel coordinate system, imaging plane coordinate system, camera device coordinate system, and spatial rectangular coordinate system.

[0148] Figure 7 This is a schematic diagram of the imaging plane coordinate system, as shown below. Figure 7 As shown, the imaging plane is the plane where the photosensitive components of the camera device are located.

[0149] Figure 8 This is a schematic diagram of the coordinate system of the camera device, such as... Figure 8As shown, the camera device coordinate system has the projection center as the origin, the direction along the optical axis towards the imaging direction as the positive z direction, the rightward direction as the positive x direction, and the downward direction as the positive y direction. c The z-axis coordinate of the camera device's coordinate system.

[0150] The transformation formula between the coordinate system of the target image and the spatial rectangular coordinate system is shown in the following formula:

[0151]

[0152] Wherein, the pixel coordinates of a calibration point are (u, v), f is the focal length of the camera device, which can be obtained from calibration. R is the rotation matrix of the camera device, and T is the translation matrix of the camera device, which can be obtained from calibration. c X is the z-axis coordinate of the camera device's coordinate system. w Y w Z w These are coordinates in a spatial rectangular coordinate system.

[0153] Based on the above formula, the following derivation formula can be obtained:

[0154]

[0155]

[0156] Mat2 = R -1 T

[0157] Z w =z c Mat1(2,0) Mat2(2,0)

[0158]

[0159] Figure 9 This is a schematic diagram of the target image corresponding to a 3x3 dot matrix image of the calibration object, such as... Figure 9 As shown, the calibration points in the target image are: P1, P2, P3, P4, P5, P6, P7, P8, and P9. Among them, P5 is the center calibration point, and its Z-axis coordinate in the spatial rectangular coordinate system is determined by the theoretical virtual image distance (i.e., the theoretical virtual image distance VID obtained in S201). w Substituting the pixel coordinates (u, v) of the center calibration point P5 and the theoretical virtual image distance VID into the above derivation formula, we can obtain the coordinates (X, v) of the center calibration point P5. w Y w Z w ).

[0160] S205. For each target calibration point, the theoretical virtual image distance is used as the Z-axis coordinate of the target calibration point in the physical coordinates of the projection point on the target plane. Based on the conversion formula between the coordinate system of the target image and the spatial rectangular coordinate system and the pixel coordinates of the target calibration point, the X-axis coordinates and Y-axis coordinates of the target calibration point in the physical coordinates of the projection point on the target plane are obtained.

[0161] Specifically, the target plane is a plane whose distance from the camera device is equal to the virtual image distance, and the target calibration points are the calibration points other than the center calibration point, namely P1, P2, P3, P4, P6, P7, P8, and P9. For each target calibration point, the theoretical virtual image distance (i.e., the theoretical virtual image distance VID obtained in S201) is used as the Z-axis coordinate (i.e., Z...) of the target calibration point under the physical coordinates of the projection point on the target plane. w By substituting the pixel coordinates (u, v) and theoretical virtual image distance VID of each target calibration point into the above derivation formula, we can obtain the coordinates of each target calibration point under the physical coordinates of the target plane projection point, that is, P1', P2', P3', P4', P5, P6', P7', P8' and P9'.

[0162] S206. Based on the coordinates of the center calibration point in the spatial rectangular coordinate system and the physical coordinates of the target calibration point projected onto the target plane, calculate the coordinates of each target calibration point in the spatial rectangular coordinate system.

[0163] Specifically, this means calculating the coordinates of P1, P2, P3, P4, P5, P6, P7, P8, and P9 in a Cartesian coordinate system based on P1', P2', P3', P4, P6, P7, P8, and P9'.

[0164] Specifically, S206 can be:

[0165] S2061. Select the first target calibration point and the second target calibration point sequentially from the target calibration points. Calculate the coordinates of the first target calibration point and the second target calibration point in the spatial rectangular coordinate system in the following manner, until the coordinates of each target calibration point in the spatial rectangular coordinate system are calculated. The first target calibration point, the second target calibration point, and the center calibration point are on a straight line.

[0166] S2062. Establish a spatial rectangular coordinate system with the center calibration point as the origin, the line connecting the coordinate point of the camera device in the spatial rectangular coordinate system and the center calibration point as the Y-axis, and the line connecting the physical coordinates of the first target calibration point and the second target calibration point as the X-axis.

[0167] Specifically, taking P2' and P8' as examples of the first and second target calibration points respectively, a spatial rectangular coordinate system is established with the center calibration point P5 as the origin. Point C is assumed to be the coordinate point of the camera device in the spatial rectangular coordinate system. The line connecting point C and P5, P5C, is the Y-axis, and the line connecting P2' and P8', P2'P8', is the X-axis. P2', P8', and P5 are collinear.

[0168] S2063. In a spatial rectangular coordinate system, based on the coordinates of the camera device in the spatial rectangular coordinate system, the physical coordinates of the projection point of the first target calibration point on the target plane, and the physical coordinates of the projection point of the second target calibration point on the target plane, the coordinates of the first target calibration point in the spatial rectangular coordinate system and the coordinates of the second target calibration point in the spatial rectangular coordinate system are calculated.

[0169] Specifically, based on the coordinates of point C, P2', and P8' of the camera device in the spatial rectangular coordinate system, the coordinates of the first target calibration point and the second target calibration point in the spatial rectangular coordinate system are calculated, that is, the coordinates of P2 and P8 are calculated.

[0170] Figure 10 This is a schematic diagram of a spatial rectangular coordinate system established using P2' and P8' as examples, provided in an embodiment of this application. Figure 10 As shown, since P2P5 = P2P8, then P2.x = -P8.x, P2.y = -P8.y. Let the equation of line P2'C be y = a1x + b1, where C and P2' are known, so a1 and b1 can be solved. Let the equation of P8'C be y = a2x + b2, where C and P8' are known, so a2 and b2 can be solved, and thus, the coordinates of P2 and P8 can be obtained. The coordinates of other target calibration points can be obtained sequentially using the same method.

[0171] S207. Based on the coordinates of the center calibration point in the spatial rectangular coordinate system and the coordinates of the target calibration point in the spatial rectangular coordinate system, determine the plane equation of the plane where the target image is located and the size of the target image. Based on the plane equation of the plane where the target image is located, obtain the normal of the plane where the target image is located.

[0172] Specifically, the coordinates of the center calibration point and the target calibration point in the spatial rectangular coordinate system are the coordinates of P1, P2, P3, P4, P6, P7, P8, and P9. After obtaining the coordinates of P1, P2, P3, P4, P6, P7, P8, and P9 in the spatial rectangular coordinate system, three points that are not on a straight line are randomly selected to form a plane, which is the plane containing the target image. The equation of the plane containing the target image is as follows:

[0173]

[0174] Where (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3) are the coordinates of the three extracted points.

[0175] The normal to the plane containing the target image can be obtained from the plane equation. The dimensions of the target image, i.e., the length and height of the target image, can be calculated from P1, P2, P3, P4, P6, P7, P8 and P9.

[0176] S208. Determine the coordinates of the center calibration point in the spatial rectangular coordinate system as the coordinates of the center point of the current imaging plane, determine the size of the target image as the size of the current imaging plane, determine the normal of the plane where the target image is located as the normal of the current imaging plane, determine the theoretical eye point coordinates as the current eye point coordinates, and determine the theoretical virtual image resolution as the current virtual image resolution to obtain the current optomechanical parameters.

[0177] Therefore, the current optomechanical parameters include: current eye point coordinates, current virtual image resolution, current imaging plane center point coordinates, current imaging plane size, and current imaging plane normal.

[0178] S209. Obtain information about the target object.

[0179] S210. Determine the viewing cone for imaging based on the current optomechanical parameters, and generate a virtual image of the target object using the viewing cone based on the information of the target object.

[0180] S211. Display the virtual image of the target object in the virtual image display area.

[0181] The method provided in this embodiment enables automatic calibration of the optical-mechanical parameters of the head-up display device, so that the optical-mechanical parameters used for imaging are more accurate. Therefore, when imaging based on more accurate optical-mechanical parameters, the positional deviation between the target object (such as a real object or person) and the virtual image can be reduced, the degree of fit between the target object and the virtual image can be improved, and the AR experience effect can be enhanced.

[0182] Figure 11 This application provides an AR display schematic diagram after calibrating the optomechanical parameters of a head-up display device, as shown in the embodiment of this application. Figure 11 As shown, the current eye point position is Figure 11 In the AR projection, the position of the "eye" is almost identical to the position of the target object in both the virtual image and the virtual image visible to the human eye. The positional deviation between the target object and the virtual image is almost zero. The virtual image and the target object are displayed in close alignment, and the projection point of the target object coincides with the theoretically calculated projection point, thus improving the AR experience.

[0183] Figure 12This is a schematic diagram of an AR display after calibrating the optomechanical parameters of a head-up display device, as provided in an embodiment of this application. Figure 12 As shown, the theoretical eye point position is Figure 12 The location of the "eye" in the AR projection and the virtual image visible to the human eye are not in the same position as the target object. The virtual image and the target object are not displayed in a way that fits together. There is a positional deviation between the target object and the virtual image. The projection point of the target object does not coincide with the theoretically calculated projection point. The AR experience is poor.

[0184] As can be seen from the above comparison, the augmented reality head-up display imaging method provided in this embodiment can reduce the positional deviation between the target object (such as a real object or person) and the virtual image, improve the degree of fit between the target object and the virtual image, and enhance the AR experience.

[0185] The following are embodiments of the apparatus described in this application, which can be used to execute the method embodiments described above. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described above.

[0186] Figure 13 This is a schematic diagram of the structure of an augmented reality head-up display imaging device provided in an embodiment of this application, as shown below. Figure 13 As shown, the device in this embodiment may include: an acquisition module 11, a determination module 12, a generation module 13, and a display module 14.

[0187] The acquisition module 11 is used to acquire information about the target object.

[0188] The determination module 12 is used to determine the viewing cone for imaging based on the current optomechanical parameters. The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of the calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the current imaging plane center point coordinates, the current imaging plane size, and the current imaging plane normal.

[0189] The generation module 13 is used to generate a virtual image of the target object using a view frustum based on the information of the target object.

[0190] Display module 14 is used to display the virtual image of the target object in the virtual image display area.

[0191] The apparatus provided in this application embodiment can execute the above method embodiment. Its specific implementation principle and technical effect can be found in the above method embodiment, and will not be repeated here.

[0192] Optionally, the acquisition module 11 is also used to: acquire the theoretical optomechanical parameters of the head-up display device.

[0193] Display module 14 is also used to display a calibration object in the virtual image display area.

[0194] The acquisition module 11 is also used to: capture images of the calibration object using a camera device to obtain a target image.

[0195] The determination module 12 is also used to: determine the current optomechanical parameters based on the target image and theoretical optomechanical parameters.

[0196] Optionally, the calibration object includes multiple calibration points, and the determination module 12 is used for:

[0197] Obtain the pixel coordinates of multiple calibration points in the target image;

[0198] The current optomechanical parameters are determined based on the pixel coordinates of multiple calibration points in the target image and the theoretical optomechanical parameters.

[0199] Optionally, the acquisition module 11 is also used to: receive an optomechanical parameter calibration command before the display module 14 displays the calibration object in the virtual image display area; the acquisition module 11: receives the pixel coordinates of multiple calibration points sent by the camera device.

[0200] Optionally, the theoretical optomechanical parameters include the theoretical virtual image distance, theoretical eye point coordinates, and theoretical virtual image resolution. The determining module 12 is used for:

[0201] Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, determine the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the plane where the target image is located;

[0202] The normal to the plane containing the target image is determined as the normal to the current imaging plane, the theoretical eye point coordinates are determined as the current eye point coordinates, and the theoretical virtual image resolution is determined as the current virtual image resolution.

[0203] Optionally, module 12 is specifically used for:

[0204] Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, determine the coordinates of the center calibration point of the multiple calibration points in the spatial rectangular coordinate system, as well as the physical coordinates of the projection point of the target calibration point on the target plane. The target plane is the plane whose distance from the camera device is the theoretical virtual image distance, and the target calibration points are the other calibration points besides the center calibration point.

[0205] Based on the coordinates of the center calibration point in the spatial rectangular coordinate system and the physical coordinates of the target calibration point projected onto the target plane, determine the normal to the plane containing the target image and the size of the target image.

[0206] The coordinates of the center calibration point in the spatial rectangular coordinate system are determined as the coordinates of the center point of the current imaging plane, and the size of the target image is determined as the size of the current imaging plane.

[0207] Optionally, module 12 is specifically used for:

[0208] Using the theoretical virtual image distance as the center calibration point, the Z-axis coordinates in the spatial rectangular coordinate system are obtained. Based on the coordinate transformation formula and the pixel coordinates of the center calibration point, the X-axis and Y-axis coordinates of the center calibration point in the spatial rectangular coordinate system are obtained.

[0209] For each target calibration point, the theoretical virtual image distance is used as the Z-axis coordinate of the target calibration point in the physical coordinates of the projection point on the target plane. Based on the coordinate transformation formula and the pixel coordinates of the target calibration point, the X-axis and Y-axis coordinates of the target calibration point in the physical coordinates of the projection point on the target plane are obtained.

[0210] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0211] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0212] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0213] Figure 14 This is a schematic diagram of the structure of a head-up display device provided in an embodiment of this application, as shown below. Figure 14 As shown, the head-up display device in this embodiment may include a processor 21 and a memory 22.

[0214] The memory 22 is used to store the executable instructions of the processor 21.

[0215] The processor 21 is configured to execute the augmented reality head-up display imaging method in the above method embodiments by executing executable instructions.

[0216] Alternatively, the memory 22 can be either standalone or integrated with the processor 21.

[0217] When the memory 22 is a device independent of the processor 21, the head-up display device of this embodiment may further include:

[0218] Bus 23 is used to connect memory 22 and processor 21.

[0219] Optionally, the head-up display device of this embodiment may further include a communication interface 24, which can be connected to the processor 21 via a bus 23.

[0220] This application also provides a computer-readable storage medium storing computer-executable instructions that, when run on a computer, cause the computer to perform the augmented reality head-up display imaging method as described above.

[0221] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the augmented reality head-up display imaging method as described in the above embodiments.

[0222] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0223] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An augmented reality head-up display imaging method, characterized in that, include: Obtain information about the target object; The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of the calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the current imaging plane center point coordinates, the current imaging plane size, and the current imaging plane normal. Based on the information of the target object, a virtual image of the target object is generated using the view frustum; Display a virtual image of the target object in the virtual image display area; The method further includes: Obtain the theoretical optomechanical parameters of the head-up display device; The calibration object is displayed in the virtual image display area; The target image is obtained by capturing the calibration object using a camera device; The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters.

2. The method according to claim 1, characterized in that, The calibration object includes multiple calibration points, and the step of determining the current optomechanical parameters based on the target image and the theoretical optomechanical parameters includes: Obtain the pixel coordinates of multiple calibration points in the target image; The current optomechanical parameters are determined based on the pixel coordinates of multiple calibration points in the target image and the theoretical optomechanical parameters.

3. The method according to claim 1, characterized in that, Before displaying the calibration object in the virtual image display area, the method further includes: Receive optical-mechanical parameter calibration commands; The step of obtaining the pixel coordinates of multiple calibration points in the target image includes: Receive the pixel coordinates of the plurality of calibration points sent by the camera device.

4. The method according to claim 2, characterized in that, The theoretical optical-mechanical parameters include theoretical virtual image distance, theoretical eye point coordinates, and theoretical virtual image resolution. Determining the current optical-mechanical parameters based on the pixel coordinates of multiple calibration points in the target image and the theoretical optical-mechanical parameters includes: Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, determine the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the plane where the target image is located; The normal to the plane containing the target image is determined as the normal to the current imaging plane, the theoretical eye point coordinates are determined as the current eye point coordinates, and the theoretical virtual image resolution is determined as the current virtual image resolution.

5. The method according to claim 4, characterized in that, The step of determining the coordinates of the center point of the current imaging plane, the size of the current imaging plane, and the normal of the plane containing the target image based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance includes: Based on the pixel coordinates of multiple calibration points in the target image and the theoretical virtual image distance, the coordinates of the center calibration point of the multiple calibration points in the spatial rectangular coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane are determined. The target plane is the plane whose distance from the camera device is the theoretical virtual image distance, and the target calibration point is the other calibration point besides the center calibration point. Based on the coordinates of the center calibration point in the spatial rectangular coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane, the normal of the plane where the target image is located and the size of the target image are determined. The coordinates of the center calibration point in the spatial rectangular coordinate system are determined as the coordinates of the center point of the current imaging plane, and the size of the target image is determined as the size of the current imaging plane.

6. The method according to claim 5, characterized in that, The step of determining the coordinates of the center calibration point of the plurality of calibration points in a spatial rectangular coordinate system and the physical coordinates of the projection point of the target calibration point on the target plane based on the pixel coordinates of the plurality of calibration points in the target image and the theoretical virtual image distance includes: Using the theoretical virtual image distance as the Z-axis coordinate of the center calibration point in a spatial rectangular coordinate system, and based on the coordinate transformation formula and the pixel coordinates of the center calibration point, the X-axis and Y-axis coordinates of the center calibration point in a spatial rectangular coordinate system are obtained. For each target calibration point, the theoretical virtual image distance is used as the Z-axis coordinate of the target calibration point under the physical coordinates of the projection point on the target plane. Based on the coordinate transformation formula and the pixel coordinates of the target calibration point, the X-axis coordinates and Y-axis coordinates of the target calibration point under the physical coordinates of the projection point on the target plane are obtained.

7. An augmented reality head-up display imaging device, characterized in that, include: The acquisition module is used to obtain information about the target object. The determination module is used to determine the viewing cone for imaging based on the current optomechanical parameters. The current optomechanical parameters are determined based on the target image and the theoretical optomechanical parameters of the head-up display device. The target image is the image of the calibration object. The current optomechanical parameters include the current eye point coordinates, the current virtual image resolution, the current imaging plane center point coordinates, the current imaging plane size, and the current imaging plane normal. A generation module is used to generate a virtual image of the target object using the view frustum based on the information of the target object; The display module is used to display the virtual image of the target object in the virtual image display area; The acquisition module is also used to: acquire the theoretical optomechanical parameters of the head-up display device; The display module is also used to: display the calibration object in the virtual image display area; The acquisition module is further configured to: capture the calibration object using a camera device to obtain the target image; The determining module is further configured to: determine the current optomechanical parameters based on the target image and the theoretical optomechanical parameters.

8. A head-up display device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the augmented reality head-up display imaging method according to any one of claims 1-6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the augmented reality head-up display imaging method according to any one of claims 1-6.

Images