AR-HUD driver eye box calibration method, system, device and medium

Abstract

An AR-HUD driver eyebox calibration method, system, device, and medium relate to the field of augmented reality. The method includes the following steps: establishing a communication connection with coordinate measuring glasses worn by the driver in response to an eyebox calibration start signal; obtaining the current eyebox position coordinates of the coordinate measuring glasses via the communication connection; determining eyebox coordinate information based on the eyebox position coordinates; obtaining the driver's eye image captured by a first camera provided on the coordinate measuring glasses via the communication connection; determining first driver gaze information based on the driver's eye image; and completing the driver's eyebox calibration based on the eyebox coordinate information and the first driver gaze information. By adopting the technical solution provided in this application, the simplicity of the current eyebox calibration method is improved, and the driver's eyebox calibration can be completed using only a pair of coordinate measuring glasses, which makes the current eyebox calibration method less dependent on various calibration devices as much as possible.

Description

Technical Field

[0001] The present application relates to the field of augmented reality, and in particular to an AR-HUD driver eye box calibration method, system, device and medium. Background Art

[0002] AR-HUD (Augmented Reality Head-Up Display) refers to augmented reality head-up display technology. Unlike traditional head-up displays, which project key information (such as vehicle speed and navigation instructions) above the driver's line of sight, AR-HUD incorporates augmented reality technology. By projecting virtual images or information over the real world, AR-HUD provides drivers with more real-time information and interactive features.

[0003] Before the AR-HUD system can operate, eyebox calibration is required. Eyebox calibration is used to determine the driver's eye position and head posture, ensuring that the HUD projection is displayed correctly. Currently, traditional eyebox calibration methods are typically based on various sensors or cameras installed on the vehicle. However, to ensure the accuracy of eyebox calibration, a large number of sensors and cameras are usually required to obtain more accurate information on the driver's eye position and head posture. Current eyebox calibration methods are relatively simple and require high standards for various calibration equipment. Summary of the Invention

[0004] In order to improve the simplicity of the current eye box calibration method and make the current eye box calibration method free from dependence on various calibration devices as much as possible, the present application provides an AR-HUD driver eye box calibration method, system, device and medium.

[0005] In a first aspect, the present application provides an AR-HUD driver eye box calibration method, the method comprising the following steps:

[0006] establishing a communication connection with the coordinate measuring glasses worn by the driver in response to an eye box calibration start signal;

[0007] obtaining the current glasses position coordinates of the coordinate measuring glasses through the communication connection;

[0008] Determining eye box coordinate information according to the glasses position coordinates;

[0009] acquiring, through the communication connection, an image of the driver's eyes captured by a first camera provided on the coordinate measuring glasses;

[0010] determining first driver gaze information based on the driver eye image;

[0011] The driver's eye box calibration is completed according to the eye box coordinate information and the first driver's gaze information.

[0012] By adopting the above technical solution, the coordinates of the driver's glasses are determined through a communication connection established with the coordinate measuring glasses, thereby determining the driver's current head posture. The driver's eyes are captured by a first camera provided on the coordinate measuring glasses to determine the driver's current gaze. The eye box is ultimately calibrated based on the driver's current head posture and gaze. This is accomplished solely through the coordinate measuring glasses, without relying on other equipment. This effectively improves the simplicity of the eye box calibration method, allowing drivers to complete eye box calibration more quickly and conveniently.

[0013] Optionally, after completing the driver's eye box calibration according to the eye box coordinate information and the first driver's gaze information, the method further includes:

[0014] determining a driver's gaze point according to the first driver's gaze information, and calculating gaze point coordinate information of the driver's gaze point;

[0015] Determining projection coordinate information according to the eye box coordinate information and the gaze point coordinate information, wherein the projection coordinate information is used to describe an imaging position of the HUD projection on the virtual image plane;

[0016] The HUD projection device is calibrated according to the projection coordinate information.

[0017] By adopting the above technical solution, after determining the position of the driver's eye box, the imaging position of the HUD projection on the virtual image plane is determined according to the driver's gaze point and the driver's eye box position, thereby ensuring that the HUD projection can be displayed clearly and completely to the driver.

[0018] Optionally, after determining the driver's gaze point according to the first driver's gaze information and calculating the gaze point coordinate information of the driver's gaze point, the method further includes:

[0019] Acquiring a driver's gaze image captured by a second camera provided on the coordinate measuring glasses;

[0020] determining second driver gaze information according to the driver gaze picture;

[0021] The driver's gaze point is calibrated according to the second driver's gaze information.

[0022] By adopting the above technical solution, after determining the driver's gaze point based on the driver's eye image captured by the first camera, the driver's gaze point is calibrated with reference to the driver's gaze image captured by the forward-facing second camera, thereby ensuring the accuracy of the determined driver's gaze point.

[0023] Optionally, determining the projection coordinate information according to the eye box coordinate information and the gaze point coordinate information specifically includes:

[0024] Establish a reference coordinate system;

[0025] Transforming the eye box coordinate information into the reference coordinate system to obtain a first calibration coordinate;

[0026] Changing the gaze point coordinate information to the reference coordinate system to obtain a second calibration coordinate;

[0027] Obtaining a virtual image plane equation of the virtual image plane;

[0028] The first calibration coordinates, the second calibration coordinates, and the virtual image plane equation are processed according to a preset imaging position calculation formula to calculate and obtain the projection coordinate information.

[0029] By adopting the above technical solution, the eye box coordinate information and the gaze point coordinate information are converted into the same reference coordinate system, so as to calculate the coordinates of the intersection of the line connecting the two points of the eye box coordinate information and the gaze point coordinate information in the reference coordinate system and the equation of the virtual image plane, thereby completing the determination of the projection coordinate information.

[0030] Optionally, the reference coordinate system takes the HUD projection device as the coordinate origin.

[0031] Optionally, the imaging position calculation formula is specifically:

[0032]

[0033] Among them, the first calibration coordinate is A(x1, y1, z1), the second calibration coordinate is B(x2, y2, z2), and P(x, y, z) is the projection coordinate information. is the direction vector of the line connecting the first calibration coordinate and the second calibration coordinate, and f(x, y, z) is the equation of the virtual image plane.

[0034] Optionally, determining the driver's gaze point according to the first driver's gaze information specifically includes:

[0035] The first driver gaze information includes the driver's left eye pupil information and the driver's right eye pupil information, and the driver's left eye pupil center is located according to the driver's left eye pupil information;

[0036] locating the center of the driver's right eye pupil according to the driver's right eye pupil information;

[0037] The driver's gaze point is estimated according to the driver's left eye pupil center and the driver's right eye pupil center.

[0038] By adopting the above technical solution, the driver's gaze point is determined by the position of the pupil centers of both eyes. By detecting the pupil centers of both eyes of the driver, a rough estimation of the driver's gaze point can be indirectly achieved.

[0039] In a second aspect of the present application, an AR-HUD driver eye box calibration system is provided, the system comprising the following modules:

[0040] a communication connection establishing module, configured to establish a communication connection with the coordinate measuring glasses worn by the driver in response to an eye box calibration start signal;

[0041] a glasses position coordinate acquisition module, configured to acquire the glasses position coordinates of the current coordinate measurement glasses through the communication connection; an eye box coordinate information determination module, configured to determine the eye box coordinate information based on the glasses position coordinates;

[0042] an eye image acquisition module, configured to acquire, through the communication connection, an eye image of the driver captured by a first camera provided on the coordinate measuring glasses;

[0043] a gaze information acquisition module, configured to determine first driver gaze information based on the driver's eye image;

[0044] An eye box calibration module is used to complete the driver's eye box calibration according to the eye box coordinate information and the first driver gaze information.

[0045] In a third aspect of the present application, an electronic device is provided;

[0046] The electronic device includes a processor, a memory, a user interface and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs an AR-HUD driver eye box calibration method.

[0047] In a fourth aspect of the present application, a computer-readable storage medium is provided;

[0048] The computer-readable storage medium stores instructions, and when the instructions are executed, an AR-HUD driver eye box calibration method is performed.

[0049] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0050] 1. The system establishes a communication connection with the coordinate measuring glasses worn by the driver to determine the glasses' coordinate position, thereby determining the driver's current head position. A first camera on the coordinate measuring glasses captures the driver's eyes to determine the driver's current gaze. Eye box calibration is ultimately completed based on the driver's current head position and gaze. This is accomplished solely through the coordinate measuring glasses, without relying on other equipment. This significantly simplifies the eye box calibration method, allowing drivers to complete it more quickly and conveniently.

[0051] 2. After determining the driver's eye box position, determine the imaging position of the HUD projection on the virtual image plane based on the driver's gaze point and the driver's eye box position, so as to ensure that the HUD projection can be displayed clearly and completely to the driver. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a scene diagram of an AR-HUD driver eye box calibration method provided in an embodiment of the present application.

[0053] Figure 2 This is a flow chart of an AR-HUD driver eye box calibration method provided in an embodiment of the present application.

[0054] Figure 3 This is a flow chart of HUD device calibration in an AR-HUD driver eye box calibration method provided in an embodiment of the present application.

[0055] Figure 4 This is a structural diagram of an AR-HUD driver eye box calibration system disclosed in an embodiment of the present application.

[0056] Figure 5 This is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.

[0057] Explanation of the accompanying drawings: 401, communication connection establishment module; 402, glasses position coordinate acquisition module; 403, eye box coordinate information determination module; 404, eye image acquisition module; 405, gaze information acquisition module; 406, eye box calibration module; 500, electronic device; 501, processor; 502, communication bus; 503, user interface; 504, network interface; 505, memory. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0059] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0060] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0061] Reference Figure 1 , an AR-HUD driver eye box calibration method provided in an embodiment of the present application can be used in vehicles equipped with AR-HUD. Before driving the vehicle, the driver wears the coordinate measuring glasses to the eye position, adjusts the vehicle seat to achieve a comfortable driving angle, and starts eye box calibration and calibration of the HUD projection device. The HUD projection device can communicate with the coordinate measuring glasses to obtain various types of information measured or obtained by the coordinate measuring glasses. The HUD projection device is also associated with the vehicle's driving system to generate information to be displayed as needed, such as vehicle speed, navigation instructions, warning information, etc. This information can come from sensors in the vehicle's driving system, navigation system or other sources.

[0062] The HUD projection device is usually installed at the front end of the vehicle's cab, and reflects the information to be displayed into the driver's field of view through optical elements. The HUD projection device projects the information to be displayed onto a virtual image plane, so that the driver can determine that the information to be displayed appears in front of the road section from his own perspective and overlaps with the driving screen. The virtual image plane is the front glass of the vehicle.

[0063] The coordinate measuring glasses are equipped with a first camera and a second camera. The first camera faces the driver's eyes to capture the driver's eye image, while the second camera faces the driver's gaze directly in the direction of the driver's gaze to capture the driver's gaze. The coordinate measuring glasses are also equipped with various sensors to measure their own position information.

[0064] Reference Figure 2, this application provides an AR-HUD driver eye box calibration method, which specifically includes the following steps:

[0065] S1: Establishing a communication connection with the coordinate measuring glasses worn by the driver in response to an eye box calibration start signal.

[0066] Specifically, when a driver enters a vehicle and plans to perform eyebox calibration, they place the coordinate measuring glasses at eye level, adjust the vehicle seat to a comfortable driving angle, and input an eyebox calibration start command to the vehicle system. Upon receiving the eyebox calibration start command, an eyebox calibration start message is generated. In response to the eyebox calibration start message, the coordinate measuring glasses activate and establish a communication connection with the vehicle system. This connection can be via Wi-Fi, Bluetooth, or a mobile network connection.

[0067] S2: Obtain the glasses position coordinates of the current coordinate measurement glasses through the communication connection.

[0068] Specifically, after the coordinate measuring glasses start working, the various sensors installed inside them begin to measure the current position information of the coordinate measuring glasses. The installed sensors specifically include accelerometers, gyroscopes, and magnetometers. The accelerometer can measure the acceleration of the glasses on the three spatial axes. By integrating the acceleration, the speed and displacement of the glasses can be estimated. The gyroscope can measure the angular velocity of the glasses rotating around the three spatial axes. By integrating the angular velocity, the rotation angle of the glasses can be estimated. The magnetometer can sense the earth's magnetic field and provide information indicating the direction of the glasses relative to the earth's magnetic field. In other embodiments provided in this application, the position information of the coordinate measuring glasses can also be obtained through other sensors such as optical sensors and depth sensors.

[0069] After acquiring data from various sensors, the position and orientation of the coordinate measuring glasses are determined by fusing data from the accelerometer, gyroscope, and magnetometer. The specific data fusion method can be a Kalman filter or a complementary filter, which is a well-known technique and will not be described in detail here. The position and orientation information of the coordinate measuring glasses is then normalized to determine the current position coordinates of the glasses.

[0070] S3: Determine eye box coordinate information according to the glasses position coordinates.

[0071] Specifically, the eye box refers to the area of ​​the display where the user can see augmented reality content. It represents the spatial range within which a user can see a clear and stable augmented reality image when wearing a head-mounted display. In this application, since the eye box is calibrated using coordinate measurement glasses, which are worn directly near the driver's eyes, the glasses' position coordinates are directly used as the driver's eye box coordinate information.

[0072] Eye box coordinates are generally expressed using the pixel coordinates of the upper left and lower right corners of the eye box, or using the pixel coordinates of the upper left corner and the width and height of the eye box. For example, eye box coordinates can be expressed as follows:

[0073] Use the pixel coordinates of the upper left corner and the lower right corner: (x1, y1, x2, y2), (x1, y1) represents the pixel coordinates of the upper left corner of the eye box, and (x2, y2) represents the pixel coordinates of the lower right corner of the eye box.

[0074] Or, use the pixel coordinates of the upper left corner, width, and height: (x, y, width, height), where (x, y) represents the pixel coordinates of the upper left corner of the eye box, width represents the width of the eye box, and height represents the height of the eye box.

[0075] S4: Acquire the driver's eye image captured by the first camera provided on the coordinate measuring glasses through the communication connection.

[0076] Specifically, the coordinate measuring glasses are equipped with a first camera that is positioned directly above the driver's eyes, capturing a full view of the driver's eyes. When the coordinate measuring glasses begin operating, the first camera simultaneously activates and begins capturing the driver's eye view. This captured view is then transmitted in real time to the vehicle system or backend server via a communication link.

[0077] S5: Determine first driver gaze information according to the driver's eye image.

[0078] Specifically, after obtaining the driver's eye image, the driver's eye image is processed. First, the specific eye position is located in the driver's eye image through image recognition technology, and the specific eye image is independently segmented in the driver's eye image. The image recognition technology used can be a neural network-based face detection algorithm or a ROI region of interest positioning algorithm. Then, pupil detection is performed on the specific eye image. Through methods such as threshold segmentation, edge detection, and template matching, the position and shape of the pupil are extracted from the eye image, thereby obtaining the first driver's gaze information.

[0079] S6: Complete the driver's eye box calibration according to the eye box coordinate information and the first driver's gaze information.

[0080] Specifically, the eye box represents the spatial range in which the user can see clear and stable augmented reality images when wearing a head-mounted display. Based on the eye box coordinate information measured by the coordinate measurement glasses, the driver's eye position and posture can be known. Based on the driver's eye image taken by the first camera set on the coordinate measurement glasses, the driver's gaze situation can be known, thereby completing the eye box calibration.

[0081] Reference Figure 3 After the driver's eye box is calibrated, the HUD projection device can be calibrated according to the determined eye box coordinate information so that the image projected by the HUD projection device can be better recognized by the driver. The specific method for calibrating the HUD projection device is as follows:

[0082] S7: Determine the driver's gaze point according to the first driver's gaze information, and calculate gaze point coordinate information of the driver's gaze point.

[0083] Specifically, the first driver's gaze information specifically includes the driver's left eye pupil information and the driver's right eye pupil information. The driver's left eye pupil center is located according to the driver's left eye pupil information. At the same time, the driver's right eye pupil center is located according to the driver's right eye pupil information. The driver's gaze point is the intersection of the extended line of the vertical connection between the driver's left eye pupil center and the driver's right eye pupil center. Therefore, after determining the driver's left eye pupil center and the driver's right eye pupil center, a vertical extension line is drawn at the driver's left eye pupil center and the driver's right eye pupil center to preliminarily determine the driver's gaze point.

[0084] After the preliminary determination of the driver's gaze point is completed, the driver's gaze image captured by the second camera set on the coordinate measuring glasses is obtained. The direction of the second camera is completely opposite to that of the first camera, and the driver's gaze image of the driver facing forward can be obtained. The second driver's gaze information is determined based on the driver's gaze image, and then the driver's gaze point is calibrated based on the second driver's gaze information.

[0085] S8: Determine projection coordinate information according to the eye box coordinate information and the gaze point coordinate information.

[0086] Specifically, through the above steps, the eye box coordinate information and the gaze point coordinate information have been determined, and a reference coordinate system has been established. It should be noted that the reference coordinate system uses the HUD projection device as the coordinate origin, and the acquired eye box coordinate information and gaze point coordinate information are transformed into the reference coordinate system through a preset transformation matrix. After the eye box coordinate information is transformed, the first calibration coordinate is obtained, and after the gaze point coordinate information is transformed, the second calibration information is obtained.

[0087] Obtain the virtual image plane equation of the virtual image plane in the reference coordinate system. It should be noted that the positions of the HUD projection device and the virtual image plane (usually the front glass of the vehicle) in the vehicle are fixed. Therefore, the virtual image plane equation of the virtual image plane in the reference coordinate system with the HUD projection device as the coordinate origin is also fixed. The virtual image plane equation is a surface equation, and its specific form is determined according to the specific parameters of the front glass of the vehicle.

[0088] When the HUD projection device is operating, the projected image is reflected from the virtual image plane into the driver's eyes. Simultaneously, the driver continuously receives image information at the point of gaze. When the projected image and the light received by the driver at the point of gaze are on the same optical path, the driver perceives the projected image as coinciding with the road surface. To ensure that the projected image and the light received by the driver at the point of gaze are on the same optical path, the projection point of the HUD projection device must be determined.

[0089] The first calibration coordinate, the second calibration coordinate and the virtual image plane equation are processed by the imaging position calculation formula to calculate the projection coordinate information, which is used to describe the position of the projected image on the virtual image plane.

[0090] The specific formula for calculating the imaging position is:

[0091] Specifically, Among them, the first calibration coordinate is A(x1,y1,z1), the second calibration coordinate is B(x2,y2,z2), and P(x,y,z) is the projection coordinate information. is the direction vector of the line connecting the first calibration coordinate and the second calibration coordinate, and f(x, y, z) is the equation of the virtual image plane.

[0092] S9: Calibrate the HUD projection device according to the projection coordinate information.

[0093] Specifically, the HUD projection device includes a mechanical device for adjusting the HUD projection camera. After obtaining the projection coordinate information, the relevant mechanical device is automatically controlled to ensure that the projected image can be accurately projected to the projection coordinates, thereby completing the calibration of the HUD projection device.

[0094] Reference Figure 4 , this application also provides an AR-HUD driver eye box calibration system, which specifically includes the following modules:

[0095] a communication connection establishing module 401 for establishing a communication connection with the coordinate measuring glasses worn by the driver in response to an eye box calibration start signal;

[0096] The glasses position coordinates acquisition module 402 is used to acquire the glasses position coordinates of the current coordinate measurement glasses through a communication connection;

[0097] The eye box coordinate information determining module 403 is used to determine the eye box coordinate information according to the glasses position coordinates;

[0098] The eye image acquisition module 404 is used to acquire the driver's eye image captured by the first camera provided on the coordinate measuring glasses through a communication connection;

[0099] A gaze information acquisition module 405 is configured to determine first driver gaze information based on the driver's eye image;

[0100] The eye box calibration module 406 is configured to complete the driver's eye box calibration according to the eye box coordinate information and the first driver's gaze information.

[0101] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0102] This application also discloses an electronic device 500. Figure 5 , Figure 5 1 is a schematic diagram of the structure of an electronic device 500 disclosed in an embodiment of the present application. The electronic device 500 may include: at least one processor 501 , at least one network interface 504 , a user interface 503 , a memory 505 , and at least one communication bus 502 .

[0103] The communication bus 502 is used to implement the connection and communication between these components.

[0104] The user interface 503 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.

[0105] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0106] The processor 501 may include one or more processing cores. The processor 501 utilizes various interfaces and lines to connect various parts of the entire server, and executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, as well as calling data stored in the memory 505. Optionally, the processor 501 may be implemented in the form of at least one hardware component selected from digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 501 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 501 and may be implemented separately on a single chip.

[0107] Among them, the memory 505 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 505 may also be optionally at least one storage device located away from the aforementioned processor 501. Reference Figure 5 , the memory 505 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of an AR-HUD driver eye box calibration method.

[0108] exist Figure 5In the electronic device 500 shown, the user interface 503 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 501 can be used to call an application program stored in the memory 505 for an AR-HUD driver eye box calibration method. When executed by one or more processors 501, the electronic device 500 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited to the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0109] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0111] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory 505. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory 505 and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory 505 includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0114] The foregoing is merely an exemplary embodiment of the present disclosure and is not intended to limit the scope of the present disclosure. In other words, any equivalent variations and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the disclosure and the practical implications thereof.

[0115] This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not described herein. The description and examples are to be considered as exemplary only, and the scope and spirit of the present disclosure are to be defined by the claims.

Claims

1. A method for calibrating the driver's eye box of an AR-HUD, characterized in that: The method comprises the following steps: establishing a communication connection with the coordinate measuring glasses worn by the driver in response to an eye box calibration start signal; The current glasses position coordinates of the coordinate measuring glasses are obtained through the communication connection. After the coordinate measuring glasses start working, the sensors arranged inside the coordinate measuring glasses measure the current position information of the coordinate measuring glasses. The arranged sensors specifically include an accelerometer, a gyroscope and a magnetometer. The acceleration of the coordinate measuring glasses on three spatial axes is measured by the accelerometer, and the speed and displacement of the coordinate measuring glasses are obtained by integrating the acceleration; the angular velocity of the coordinate measuring glasses rotating around the three spatial axes is measured by the gyroscope, and the rotation angle of the coordinate measuring glasses is obtained by integrating the angular velocity; the magnetic field of the earth is sensed by the magnetometer, and information indicating the direction of the coordinate measuring glasses relative to the earth's magnetic field is provided; the position and direction information of the coordinate measuring glasses is obtained by fusing the data of the accelerometer, the gyroscope and the magnetometer, and the current position coordinates of the coordinate measuring glasses are determined based on the position and direction information; Determining eye box coordinate information according to the glasses position coordinates; acquiring, through the communication connection, an image of the driver's eyes captured by a first camera provided on the coordinate measuring glasses; determining first driver gaze information based on the driver eye image; Complete the driver's eye box calibration according to the eye box coordinate information and the first driver gaze information, After the driver's eye box calibration is completed according to the eye box coordinate information and the first driver's gaze information, the method further includes: determining a driver's gaze point according to the first driver's gaze information, and calculating gaze point coordinate information of the driver's gaze point; Determining projection coordinate information according to the eye box coordinate information and the gaze point coordinate information, wherein the projection coordinate information is used to describe an imaging position of the HUD projection on the virtual image plane; Calibrate the HUD projection device according to the projection coordinate information, Wherein, determining the projection coordinate information according to the eye box coordinate information and the gaze point coordinate information specifically includes: Establish a reference coordinate system; Transforming the eye box coordinate information into the reference coordinate system to obtain a first calibration coordinate; Changing the gaze point coordinate information to the reference coordinate system to obtain a second calibration coordinate; Obtaining a virtual image plane equation of the virtual image plane; The first calibration coordinates, the second calibration coordinates, and the virtual image plane equation are processed according to a preset imaging position calculation formula to calculate and obtain the projection coordinate information.

2. The AR-HUD driver eye box calibration method according to claim 1, characterized in that: After determining the driver's gaze point according to the first driver's gaze information and calculating the gaze point coordinate information of the driver's gaze point, the method further includes: Acquiring a driver's gaze image captured by a second camera provided on the coordinate measuring glasses; determining second driver gaze information according to the driver gaze picture; The driver's gaze point is calibrated according to the second driver's gaze information.

3. The AR-HUD driver eye box calibration method according to claim 1, characterized in that: The reference coordinate system takes the HUD projection device as the coordinate origin.

4. The AR-HUD driver eye box calibration method according to claim 1, characterized in that: The imaging position calculation formula is specifically: ； in, The first calibration coordinate is, The second calibration coordinate is, is the projection coordinate information, is the direction vector of the line connecting the first calibration coordinate and the second calibration coordinate, is the equation of the virtual image plane.

5. The AR-HUD driver eye box calibration method according to claim 2, characterized in that: Determining the driver's gaze point according to the first driver's gaze information specifically includes: The first driver gaze information includes the driver's left eye pupil information and the driver's right eye pupil information, and the driver's left eye pupil center is located according to the driver's left eye pupil information; locating the center of the driver's right eye pupil according to the driver's right eye pupil information; The driver's gaze point is estimated according to the driver's left eye pupil center and the driver's right eye pupil center.

6. An AR-HUD driver eye box calibration system, characterized in that: The system comprises: A communication connection establishing module (401) is used to establish a communication connection with the coordinate measuring glasses worn by the driver in response to an eye box calibration start signal; A glasses position coordinate acquisition module (402) is used to acquire the current glasses position coordinates of the coordinate measuring glasses through the communication connection. After the coordinate measuring glasses start working, the sensors arranged inside the coordinate measuring glasses measure the current position information of the coordinate measuring glasses. The arranged sensors specifically include an accelerometer, a gyroscope and a magnetometer. The acceleration of the coordinate measuring glasses on three spatial axes is measured by the accelerometer, and the speed and displacement of the coordinate measuring glasses are obtained by integrating the acceleration. The angular velocity of the coordinate measuring glasses rotating around the three spatial axes is measured by the gyroscope, and the rotation angle of the coordinate measuring glasses is obtained by integrating the angular velocity. The magnetic field of the earth is sensed by the magnetometer, and information indicating the direction of the coordinate measuring glasses relative to the earth's magnetic field is provided. The position and direction information of the coordinate measuring glasses are obtained by fusing the data of the accelerometer, the gyroscope and the magnetometer, and the current position coordinates of the coordinate measuring glasses are determined based on the position and direction information. An eye box coordinate information determination module (403), configured to determine eye box coordinate information based on the glasses position coordinates; An eye image acquisition module (404) is used to acquire, through the communication connection, an eye image of the driver captured by a first camera provided on the coordinate measuring glasses; A gaze information acquisition module (405) is used to determine first driver gaze information based on the driver eye image; An eye box calibration module (406) is used to complete the driver's eye box calibration according to the eye box coordinate information and the first driver gaze information, The eye box calibration module (406) is further configured to determine the driver's gaze point based on the first driver's gaze information, and calculate the gaze point coordinate information of the driver's gaze point; Determining projection coordinate information according to the eye box coordinate information and the gaze point coordinate information, wherein the projection coordinate information is used to describe an imaging position of the HUD projection on the virtual image plane; Calibrate the HUD projection device according to the projection coordinate information, The eye box calibration module (406) is further configured to transform the eye box coordinate information into a reference coordinate system to obtain a first calibration coordinate; Changing the gaze point coordinate information to the reference coordinate system to obtain a second calibration coordinate; Obtaining a virtual image plane equation of the virtual image plane; The first calibration coordinates, the second calibration coordinates, and the virtual image plane equation are processed according to a preset imaging position calculation formula to calculate and obtain the projection coordinate information.

7. An electronic device, characterized in that: The electronic device (500) comprises a processor (501), a memory (505), a user interface (503) and a network interface (504), wherein the memory (505) is used to store instructions, the user interface (503) and the network interface (504) are used to communicate with other devices, and the processor (501) is used to execute the instructions stored in the memory (505) so that the electronic device (500) executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method steps according to any one of claims 1 to 5 are performed.

Images