Head-up display system control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a head-up display system control method and device, equipment and a medium. The method comprises the following steps: acquiring image information including eyes of a user, and acquiring an eye region image based on the image information; acquiring an actual projection position of the head-up display system, and establishing a sight line dot matrix corresponding to the eye region image based on the actual projection position; determining pupil position information of the user based on the sight line dot matrix, and obtaining a mapping relation between the pupil position information and the actual projection position; and if it is monitored that the pupil position information changes, determining a target projection position of the head-up display system based on the changed pupil position information and the mapping relationship, so as to control a reflection angle of a reflector of the head-up display system based on the target projection position. According to the embodiment of the invention, the effectiveness and accuracy of information display can be continuously maintained by dynamically adjusting the projection position, the method is suitable for various use scenes and user states, the user experience is improved, and the driving safety is improved.

Description

Technical Field

[0001] The present application relates to the technical field of virtual image generation, and in particular to a head-up display system control method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of technology, the way to display information on the windshield mainly adopts a head up display (HUD). The existing augmented reality head up display (ARHUD) assists the driver in driving the vehicle by displaying information on the head up display.

[0003] In current driving scenarios, head-up display (HUD) technology has been widely used in the automotive industry, aiming to improve driving safety and convenience by projecting key driving information onto the windshield, reducing the number of times the driver looks down at the dashboard. The display position of the HUD system is usually fixed, which can meet the basic needs of most drivers. However, under different heights, sitting postures and driving environments, the fixed display position may cause some drivers to be unable to clearly see the displayed information, thus affecting driving safety and convenience. Summary of the invention

[0004] To solve the above technical problems, embodiments of the present application provide a head-up display system control method and device, electronic device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of an embodiment of the present application, a head-up display system control method is provided, including: acquiring image information including eyes of a user, so as to acquire the eye area image based on the image information; acquiring the actual projection position of the head-up display system, so as to establish a sight line dot matrix corresponding to the eye area image based on the actual projection position; determining pupil position information of the user based on the sight line dot matrix, and acquiring a mapping relationship between the pupil position information and the actual projection position; if a change in the pupil position information is monitored, determining a target projection position of the head-up display system based on the changed pupil position information and the mapping relationship, so as to control the reflection angle of a reflector of the head-up display system based on the target projection position.

[0006] According to one aspect of an embodiment of the present application, establishing a sight line dot matrix corresponding to the eye area image based on the actual projection position includes: establishing a first dot matrix coordinate system in the eye area image based on the actual projection position; extracting a target peri-ocular area from the eye area image based on the first dot matrix coordinate system; establishing a second dot matrix coordinate system in the target peri-ocular area based on the actual projection position, and adjusting the second dot matrix coordinate system based on the target dot matrix density to obtain a corresponding sight line dot matrix.

[0007] According to one aspect of an embodiment of the present application, extracting a target periocular area from the eye area image based on the first dot matrix coordinate system includes: acquiring a target human eye model, and determining a candidate periocular area from the eye area image based on the target human eye model; extracting a plurality of periocular coordinate data from the candidate periocular area based on the first dot matrix coordinate system; and determining a target periocular area based on the plurality of periocular coordinate data.

[0008] According to one aspect of an embodiment of the present application, the method also includes: extracting pupil features in the target periorbital area; acquiring multiple periorbital pupil coordinate data corresponding to the pupil features in the second dot matrix coordinate system; and determining pupil position information of the pupil in the second dot matrix coordinate system based on the multiple periorbital pupil coordinate data.

[0009] According to one aspect of an embodiment of the present application, the method also includes: filtering the acquired multiple coordinate data to obtain filtered coordinate data; calculating the difference between the multiple coordinate data and the filtered coordinate data; replacing the coordinate data whose difference between the multiple coordinate data and the filtered coordinate data reaches a preset difference threshold with the filtered coordinate data.

[0010] According to one aspect of an embodiment of the present application, before adjusting the second dot matrix coordinate system based on the target dot matrix density to obtain the corresponding line of sight dot matrix, the method also includes: obtaining operating condition information and driver information of the vehicle, the driver information including driver identity information and personalized information corresponding to the driver identity information; determining the target dot matrix density based on the operating condition information and the personalized information, so as to determine the second dot matrix coordinate system through the target dot matrix density.

[0011] According to one aspect of an embodiment of the present application, the method also includes: obtaining a calibration sample set, the calibration sample set including multiple pupil position information and target projection positions corresponding to each of the multiple pupil position information; establishing a mapping relationship between the pupil position information and the target projection position based on the multiple pupil position information and the multiple target projection positions; obtaining operation information of the user, the operation information including operation information of the user for the head-up display system; correcting the mapping relationship based on the operation information to obtain a mapping relationship between the pupil position information and the actual projection position.

[0012] According to one aspect of an embodiment of the present application, a head-up display system control device is provided, including: an acquisition module, acquiring image information including a user's eyes, so as to acquire the eye area image based on the image information; an establishment module, acquiring the actual projection position of the head-up display system, so as to establish a sight line dot matrix corresponding to the eye area image based on the actual projection position; a determination module, determining the pupil position information of the user based on the sight line dot matrix, and acquiring a mapping relationship between the pupil position information and the actual projection position; a control module, if it is monitored that the pupil position information changes, determines the target projection position of the head-up display system based on the changed pupil position information and the mapping relationship, so as to control the reflection angle of the reflector of the head-up display system based on the target projection position.

[0013] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the head-up display system control method as described above.

[0014] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the head-up display system control method as described above.

[0015] According to one aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the steps in the head-up display system control method described above are implemented.

[0016] In the technical solution provided in the embodiments of the present application, by capturing and analyzing the user's eye area image, especially the pupil position information, the projection position of the head-up display system can be accurately adjusted to make the information display more in line with the user's visual habits and comfort, provide a personalized driving or information display experience, and monitor the changes in pupil position information in real time to ensure that the display content of the head-up display system is always within the user's optimal visual field, reduce distraction or information omission caused by visual deviation, and thus improve driving or operating safety. On the other hand, since the user's eye position may change due to head movement, sitting posture changes, or physiological differences between different users, by dynamically adjusting the projection position, the effectiveness and accuracy of the information display can be continuously maintained, adapting to various usage scenarios and user states, enhancing the user experience, and improving driving safety.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment for controlling a head-up display system during vehicle driving, shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a flow chart of a head-up display system control method shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a flow chart of a head-up display system control method shown in another exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a human eye image in a first dot matrix coordinate system shown by an exemplary embodiment;

[0023] Figure 5 is a schematic diagram of a human eye image in a second dot matrix coordinate system shown by an exemplary embodiment;

[0024] Figure 6 is a flow chart of a head-up display system control method shown in another exemplary embodiment of the present application;

[0025] Figure 7is a flow chart of a head-up display system control method shown in another exemplary embodiment of the present application;

[0026] Figure 8 is a flow chart of a head-up display system control method shown in another exemplary embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of data updating after data filtering shown in an exemplary embodiment;

[0028] Fig.10 is a flow chart of a head-up display system control method shown in another exemplary embodiment of the present application;

[0029] Fig.11 is a flow chart of a head-up display system control method shown in another exemplary embodiment of the present application;

[0030] Fig.12 is a schematic diagram of a simplified flow chart for controlling a head-up display system in an exemplary application scenario;

[0031] Fig.13 is a block diagram of a head-up display system control device shown in an exemplary embodiment of the present application;

[0032] Fig.14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0033] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0036] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0037] HUD (Head-Up Display) technology is actually a display technology. The main function of HUD technology is to project key driving information (such as vehicle speed, navigation instructions, warning information, etc.) directly in front of the driver's line of sight, usually on the windshield or a transparent display screen. In this way, the driver can obtain this information without looking down at the dashboard or central control screen, thereby greatly reducing the potential risks caused by line of sight diversion and improving driving safety.

[0038] HUD technology projects images or data onto a transparent interface in front of the driver or pilot's line of sight by optical means, such as the front windshield of a car or the cockpit glass of an airplane. In this way, the driver or pilot can directly see the required information while maintaining the normal line of sight, without having to look down at the instrument or display. Among them, the core components of HUD include: an image generator is responsible for forming the image or data to be projected, which is usually achieved through a projection source such as a liquid crystal or DLP (digital light processing) display; an optical lens system is used to enlarge and distance the image formed by the image generator to ensure that the image can be clearly presented in front of the driver. The optical lens system usually includes a series of lenses and reflectors, which work together to adjust the focal length, direction and size of the image; a reflection unit is used to reflect the image processed by the optical lens system into the driver's line of sight. In the automotive HUD system, this is usually achieved through the windshield. In order to ensure that the image can be clearly displayed on the windshield, the reflection unit usually needs to be precisely designed and adjusted according to the characteristics of the windshield; a projection unit is responsible for projecting the image formed by the image generator onto the optical lens system, which is usually achieved through a projector and other devices.

[0039] Figure 1 FIG. 1 is a schematic diagram of an implementation environment for controlling a head-up display system during vehicle driving, shown in an exemplary embodiment of the present application. Figure 1As shown, during the driving process of the vehicle, image information including the user's eyes is obtained through the image acquisition device corresponding to the smart terminal 110, and the user's eye area image is obtained based on the image information. Then, the server side 120 obtains the actual projection position of the head-up display system of the vehicle, and obtains the eye area image on the smart terminal 110, and then establishes a line of sight dot matrix corresponding to the eye area image according to the actual projection position. Then, the server side 120 can determine the user's pupil position information based on the line of sight dot matrix, and obtain the mapping relationship between the pupil position information and the actual projection position. The server side 120 continuously monitors the user's pupil position. If the pupil position information is detected to have changed, the target projection position of the head-up display system is determined based on the changed pupil position information and the mapping relationship, so as to control the reflection angle of the reflector of the head-up display system based on the target projection position, thereby achieving the tracking of the user's pupil position and the updating of the projection position of the head-up display system.

[0040] in, Figure 1 The smart terminal 110 shown can be any terminal device that supports image acquisition and processing, such as a smart phone, a car computer, a tablet computer, a laptop computer, or a wearable device, but is not limited thereto. Figure 1 The server end 120 shown is a server, for example, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited here. The smart terminal 110 can communicate with the server end 120 through wireless networks such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), and 5G (fifth generation mobile information technology), which are not limited here.

[0041] In current driving scenarios, head-up display (HUD) technology has been widely used in the automotive industry, aiming to improve driving safety and convenience by projecting key driving information onto the windshield, reducing the number of times the driver looks down at the dashboard. The display position of the HUD system is usually fixed, which can meet the basic needs of most drivers. However, under different heights, sitting postures and driving environments, the fixed display position may cause some drivers to be unable to clearly see the displayed information, thus affecting driving safety and convenience.

[0042] The problems pointed out above are generally applicable in general travel scenarios. It can be seen that the projection position of the existing head-up display system is relatively fixed and cannot adapt to the changes of the user's line of sight, and the user experience is not high. In order to solve these problems, the embodiments of the present application respectively propose a head-up display system control method, a head-up display system control device, an electronic device, a computer-readable storage medium and a computer program product, which will be described in detail below.

[0043] See also Figure 2 , Figure 2 is a flowchart of a head-up display system control method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is specifically executed by the server 120 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0044] like Figure 2 As shown, in an exemplary embodiment, the head-up display system control method includes at least steps S210 to S240, which are described in detail as follows:

[0045] Step S210, acquiring image information including the user's eyes, so as to acquire an eye area image based on the image information.

[0046] In the cabin, high-resolution cameras are usually selected, or the user's eye images can be directly obtained through the image acquisition device (camera) of the HUD system. These cameras need to have good optical performance and image processing capabilities to ensure that clear and accurate images of the eye area can be captured. The installation position of the camera needs to be carefully selected to ensure that the user's eye area can be captured while avoiding obstruction or interference from other objects in the cabin. Afterwards, the eye area can be extracted from the captured image through image processing algorithms, and further processing and analysis can be performed. These algorithms usually include steps such as image preprocessing, feature extraction, and target detection to achieve accurate recognition of the eye area.

[0047] Among them, the user's eye images are captured using cameras installed in the cabin. These images need to contain enough information for subsequent processing and analysis. The captured images are preprocessed, including steps such as denoising, contrast enhancement, and brightness adjustment to improve the quality and clarity of the images. Image processing algorithms are applied to detect the eye area in the image. This usually involves using specific feature extraction algorithms such as edge detection, circle fitting, etc. to identify the shape and position of the eyes. After the eye area is detected, it is extracted from the original image using image processing technology. This ensures that subsequent processing and analysis can focus on the eye area, improving accuracy and efficiency.

[0048] Step S220, obtaining the actual projection position of the head-up display system to establish a sight line dot matrix corresponding to the eye area image based on the actual projection position.

[0049] First, make sure the HUD system is properly installed and calibrated, which typically includes adjusting the height, brightness, and contrast of the HUD, as well as calibrating it horizontally and vertically to ensure that the projected information corresponds to the actual road conditions. These adjustments can be made through the vehicle's settings menu or physical buttons. During the adjustment process, pay attention to the projection position of the HUD to ensure that it is parallel to the driver's line of sight and located in the middle and lower part of the windshield. In order to more accurately determine the actual projection position of the HUD, test patterns or images can be used. These patterns or images can contain easily identifiable marking points or lines to determine the boundaries of the projection area on the windshield.

[0050] By observing the projection positions of these marking points or lines on the windshield, the actual projection position of the HUD can be accurately determined. After obtaining the actual projection position of the HUD, the eye area needs to be determined. This can usually be done by capturing the driver's facial image through a camera and using an image processing algorithm to identify the position and shape of the eye area.

[0051] The eye region can be defined as a rectangular area around a given eye (or a vertical strip for a one-dimensional shutter), with the eye spacing being the horizontal direction. Based on the position and shape of the eye region, a gaze dot matrix can be established. The gaze dot matrix is ​​a dot matrix consisting of a series of points representing the gaze direction, which can be arranged and adjusted according to the position and shape of the eye region.

[0052] The establishment of the sight line matrix can determine the position and size of the shutter pupil based on the binocular parallax 3D display method. By drawing a line from the right edge of any image display pixel to the left edge of the eye area, and then drawing a line from the left edge of the same image display pixel to the right edge of the same eye area, the intersection of the two lines defines an imaging pinhole plane. Then, according to the lateral distance between the intersection points on the imaging pinhole plane, the shutter screen position is fine-tuned forward and backward to determine the shutter pupil position and size.

[0053] In addition, the establishment of the sight dot matrix may need to take into account multiple factors, such as individual differences of drivers, seat position, tilt angle of the windshield, etc. Therefore, in practical applications, the sight dot matrix may need to be dynamically adjusted and calibrated. After the sight dot matrix is ​​established, it needs to be verified and optimized. This can be done by having the driver watch the test pattern or image on the HUD and observe whether the sight dot matrix can accurately reflect the driver's sight direction. If deviations or errors are found in the sight dot matrix, it can be optimized by adjusting the camera position, image processing algorithm or the arrangement of the sight dot matrix.

[0054] Step S230, determining the user's pupil position information based on the sight line matrix, and obtaining a mapping relationship between the pupil position information and the actual projection position.

[0055] Exemplarily, the gaze tracking technology can utilize the pupil-corneal reflection method, which relies on the Purkinje spots on the outer surface of the cornea of ​​the eyeball (produced by the reflection of light entering the pupil on the outer surface of the cornea) and the position of the center of the pupil. By locating the positions of the pupil and Purkinje spots on the eye image in real time, the corneal reflection vector can be calculated, and then the user's gaze direction can be estimated using a geometric model. In the established gaze dot matrix, each point represents a possible gaze direction. When the user looks at a certain position on the HUD, the user's current gaze direction can be determined by detecting the positions of the pupil and Purkinje spots, and their changes relative to the gaze dot matrix.

[0056] Furthermore, a corresponding point can be found in the line of sight dot matrix based on the line of sight direction. This point represents the user's current pupil position information (relative position relative to the HUD projection position). In order to establish a mapping relationship between the pupil position information and the actual projection position, a calibration process is usually required. During the calibration process, the user will be asked to look at specific points on the HUD (these points are usually pre-set projection points with known coordinates). At the same time, the system will record the pupil position information (relative to the position of the line of sight dot matrix) when the user looks at these points.

[0057] Step S240: If it is detected that the pupil position information changes, the target projection position of the head-up display system is determined based on the changed pupil position information and the mapping relationship, so as to control the reflection angle of the reflector of the head-up display system based on the target projection position.

[0058] Exemplarily, the head-up display system first monitors the user's pupil position information. This is usually achieved through a built-in camera or sensor that can capture the position of the user's eyes, especially the position of the pupil, in real time. When the user moves his head, eyes or body, the position information of the pupil will change. This change may be small, but it is enough to affect the projection effect of the head-up display system. Once the change in pupil position information is detected, the head-up display system will use this information, combined with the previously established mapping relationship (i.e., the correspondence between the pupil position and the projection position), to determine the new target projection position. This mapping relationship may be established through a pre-calibration process to ensure that the projected content can be accurately presented at the focus of the user's line of sight. After determining the target projection position, the head-up display system will adjust the reflection angle of its internal reflector. The reflector is a key component in the head-up display system, which is responsible for reflecting the projection light source into the user's line of sight. By adjusting the angle of the reflector, the direction of the projection light can be changed to ensure that the projected content can appear accurately at the target projection position. The whole process is dynamic, that is, the head-up display system will continuously monitor the pupil position information and adjust the projection position and the angle of the reflector in real time as needed. In this way, no matter how the user moves, the head-up display system can ensure that the projected content always remains in the user's line of sight, providing a clear and comfortable viewing experience.

[0059] In some embodiments of the present application, by capturing and analyzing the user's eye area image, especially the pupil position information, the projection position of the head-up display system can be accurately adjusted to make the information display more in line with the user's visual habits and comfort, provide a personalized driving or information display experience, and monitor the changes in pupil position information in real time to ensure that the display content of the head-up display system is always within the user's optimal visual field, reduce distraction or information omission caused by visual deviation, and thus improve driving or operating safety. On the other hand, since the user's eye position may change due to head movement, sitting posture changes, or physiological differences between different users, by dynamically adjusting the projection position, the effectiveness and accuracy of the information display can be continuously maintained, adapting to various usage scenarios and user states, enhancing the user experience, and improving driving safety.

[0060] Based on the above examples, please refer to Figure 3 In one of the exemplary embodiments provided in the present application, the specific implementation process of establishing the sight line dot matrix corresponding to the eye area image based on the actual projection position may also include steps S310 to S330, which are described in detail as follows:

[0061] Step S310: establishing a first dot matrix coordinate system in the eye area image based on the actual projection position.

[0062] For example, the first dot matrix coordinate system can be established in the eye area image according to the actual projection position of the head-up display system at the current moment, which can be referred to in Figure 4 The first dot matrix coordinate system is used to obtain the relative spatial position of the user's eyes with respect to the head-up display system, that is, the position information corresponding to the user's eye area can be expressed by coordinates in the first dot matrix coordinate system.

[0063] Optionally, in a head-up display system (HUD), the actual projection position refers to the specific position of the image or information formed by the projection light in front of the user's eyes. This position can be monitored by the sensor or camera of the HUD system and compared with the user's pupil position to ensure that the projected content is at the focus of the user's line of sight. The eye area image is usually captured by the built-in camera of the HUD system. This image contains the user's eyes and the area around them, and is the basis for the subsequent establishment of a dot matrix coordinate system. After determining the actual projection position and obtaining the eye area image, the next step is to establish a dot matrix coordinate system on this image, that is, the first dot matrix coordinate system. This coordinate system is used to describe and locate each point on the image, especially the points related to the actual projection position.

[0064] The dot matrix coordinate system usually consists of a set of mutually perpendicular lines (i.e., coordinate axes), and the points on the lines (i.e., coordinate points) are used to represent specific locations on the image. The first dot matrix coordinate system may be established based on specific features of the eye area (such as the center of the pupil, the corner of the eye, etc.) to ensure the accuracy and stability of the coordinate system. Once the first dot matrix coordinate system is established, it can be used to accurately describe and locate the position of the projected content on the eye area image. This helps the HUD system to more accurately adjust the projection position and the angle of the reflector to ensure that the projected content always remains in the focus of the user's line of sight.

[0065] In addition, in some feasible embodiments, the first dot matrix coordinate system can also be used for other image processing tasks, such as image enhancement, feature extraction, image recognition, etc. These tasks help the HUD system provide a clearer and more comfortable viewing experience and enhance the overall performance and reliability of the system.

[0066] Step S320: extracting the target eye area from the eye area image based on the first dot matrix coordinate system.

[0067] Step S330, establishing a second dot matrix coordinate system in the target periorbital area based on the actual projection position, and adjusting the second dot matrix coordinate system based on the target dot matrix density to obtain a corresponding sight line dot matrix.

[0068] like Figure 4 As shown, multiple coordinate points near the user's eyes can be determined through the first dot matrix coordinate system, and then the user's target peri-ocular area can be extracted based on the multiple coordinate points. For example, by introducing a "dot matrix" on the captured image, a horizontal and vertical coordinate that is roughly equivalent to the height of the human eye, that is, relative to the actual width of about one centimeter, is established for the first level of rough matching of human eye tracking, and the approximate location of the human eye is output. The matching method will introduce the Euclidean distance algorithm, which is a commonly used method for measuring the direct distance between two points in space. It has the advantages of high computational efficiency and strong stability. It can quickly calculate the point in the dot matrix closest to the human eye position with low performance consumption, and then extract the user's target peri-ocular area based on the point closest to the human eye position.

[0069] Exemplarily, these points form a grid or framework for locating and extracting specific image areas. Here, the "eye area image" is an image containing the eye and its surrounding area. By applying the first dot matrix coordinate system, the part of interest, namely the "target eye area", can be accurately located and extracted. This area usually includes the eye and the surrounding skin, eyelashes, etc. Next, based on the actual projection position (which may be the center of the eyeball, the position of the pupil, or other reference points), a new "second dot matrix coordinate system" is established in the extracted target eye area.

[0070] like Figure 5 As shown in the figure, this new second dot matrix coordinate system is used to further refine the analysis of the periocular area, especially to capture the gaze direction, and then the second dot matrix coordinate system is adjusted according to the target dot matrix density. The dot matrix density in the target dot matrix density refers to the number of dots per unit area. By adjusting this density, the characteristics of the periocular area can be captured and analyzed more finely or roughly. Finally, the adjusted second dot matrix coordinate system is used to generate the "sight matrix", which is a dot matrix representation that can reflect the gaze direction.

[0071] Optionally, a predefined first dot matrix coordinate system is used to accurately extract the target periocular region from the eye region image. This region contains the key features of the eye and its surroundings. Then, in this extracted target periocular region, a new second dot matrix coordinate system is established based on the actual projection position (such as the center of the pupil). This new coordinate system is used to further analyze the periocular region, especially to capture the direction of sight. In order to obtain more accurate sight information, the second dot matrix coordinate system is adjusted according to the target dot matrix density. The adjusted dot matrix coordinate system can more accurately reflect the characteristics of the periocular region, thereby generating a sight dot matrix that can represent the direction of sight.

[0072] In some embodiments of the present application, based on the first dot matrix coordinate system, the target eye area can be more effectively extracted from the eye area image. This extraction method not only improves the recognition rate of the eye area, but also reduces unnecessary image processing time, optimizes the eye area extraction and flexibly adjusts the sight line dot matrix, and can more accurately respond to the user's sight line changes, providing HUD display content that better meets the user's needs, which not only improves the user's driving safety, but also enhances the user's overall satisfaction and comfort.

[0073] Based on the above examples, please refer to Figure 6 In one of the exemplary embodiments provided in the present application, the specific implementation process of extracting the target eye area from the eye area image based on the first dot matrix coordinate system may also include steps S610 to S630, which are described in detail as follows:

[0074] Step S610, obtaining a target human eye model, and determining a candidate eye periorbital region from the eye region image based on the target human eye model;

[0075] Step S620, extracting a plurality of periorbital coordinate data from the candidate periorbital region based on the first dot matrix coordinate system;

[0076] Step S630: determining a target periorbital area based on the plurality of periorbital coordinate data.

[0077] Exemplarily, positioning can be performed by detecting shape features of the eyes, for example, by using multiple standard eye models to extract potential eye areas from the eye area image, and calculating the geometric features of these eye areas. Thereafter, the extracted geometric features are matched with a predefined target human eye model to obtain a candidate periocular area, and then multiple periocular coordinate data are extracted from the candidate periocular area based on the first dot matrix coordinate system. Then, the target periocular area of ​​the user is determined based on the extracted multiple periocular coordinate data, and the eye area image is cut out of the target periocular area.

[0078] Optionally, a target human eye model. This model can be a general model built based on average facial features or a model customized for a specific individual. It contains the structure and shape information of the eye and its surrounding area. With the target human eye model, it can be compared or matched with the eye area image. Through this process, the area in the image that may contain the eye, namely the "candidate periocular area", can be roughly determined. This area may contain some additional background or noise, but it is close enough to the real periocular area. Next, a predefined "first dot matrix coordinate system" is applied to the candidate periocular area. This coordinate system consists of a series of uniformly distributed points that cover the entire candidate periocular area. By sampling these points, multiple periocular coordinate data can be extracted. These data represent the pixel values ​​or feature information at different positions in the candidate periocular area. With these periocular coordinate data, various image processing and analysis techniques (such as edge detection, morphological operations, machine learning algorithms, etc.) can be used to further process and filter these data. Finally, a more accurate "target periocular area" that is closer to the real periocular area can be determined from these data. This area usually includes the eye itself and the surrounding skin area immediately surrounding the eye.

[0079] In some embodiments of the present application, based on the target human eye model, candidate periocular areas can be quickly screened out from the eye area image. This screening method reduces unnecessary image processing steps and improves the overall processing efficiency. The first dot matrix coordinate system is used to extract multiple periocular coordinate data from the candidate periocular areas to ensure the accuracy of the coordinate data.

[0080] Based on the above examples, please refer to Figure 7 In one of the exemplary embodiments provided in the present application, the specific implementation process of the head-up display system control method may further include steps S710 to S730, which are described in detail as follows:

[0081] Step S710, extracting pupil features in the target eye area;

[0082] Step S720, obtaining a plurality of pupil periphery coordinate data corresponding to pupil features in the second dot matrix coordinate system;

[0083] Step S730: determining pupil position information of the pupil in the second dot matrix coordinate system based on the plurality of pupil periphery coordinate data.

[0084] Exemplarily, the second dot matrix coordinate system is used to capture the user's pupil, that is, the changes in the user's line of sight. Therefore, after obtaining the target periocular area from the first dot matrix coordinate system, a coordinate system can be established in the target periocular area and set to a more refined line of sight dot matrix coordinate system. The orientation of the pupil is the direct determinant of the line of sight direction. The position details of the pupil will be magnified in this frame, and the subtle movements of the pupil can be captured with higher accuracy. The pupil position is matched with the dot matrix coordinates through the Euclidean distance algorithm, thereby outputting higher-precision line of sight position information.

[0085] Optionally, the pupil features must first be extracted from the target periocular area. These features may include the pupil's outline, color, brightness, etc., depending on the image processing technology and algorithm used. In the determined second dot matrix coordinate system, multiple pupil periorbital coordinate data corresponding to the pupil features need to be found. These coordinate data describe the key points or boundary positions around the pupil. Based on the multiple pupil periorbital coordinate data obtained in the previous step, the specific position information of the pupil in the second dot matrix coordinate system can be further calculated or inferred. This may involve techniques such as statistical analysis of coordinate data, geometric calculations, or pattern recognition.

[0086] In the target periocular area, a more refined sight line dot matrix coordinate system (i.e., the second dot matrix coordinate system) is established. This coordinate system is designed to improve the accuracy of pupil position detection and capture the subtle movement of the pupil through denser coordinate points. In this refined sight line dot matrix coordinate system, the orientation of the pupil becomes the direct determinant of the sight line direction. In order to capture the subtle movement of the pupil, the pupil position details are magnified, which may involve image scaling, interpolation and other technologies. Mathematical tools such as the Euclidean distance algorithm are used to match the position of the pupil in the sight line dot matrix coordinate system with the preset dot matrix coordinates. This step is designed to accurately determine the current position of the pupil and provide basic data for subsequent sight line tracking.

[0087] In some embodiments of the present application, by extracting pupil features in the area around the eye, the pupil position can be located more accurately, which can improve the accuracy of pupil recognition, reduce the possibility of misrecognition, and improve the effectiveness of tracking the user's line of sight.

[0088] Based on the above examples, please refer to Figure 8 In one of the exemplary embodiments provided in the present application, the specific implementation process of the head-up display system control method may further include steps S810 to S830, which are described in detail as follows:

[0089] Step S810, filtering the acquired multiple coordinate data to obtain filtered coordinate data;

[0090] Step S820, calculating the difference between the plurality of coordinate data and the filtered coordinate data;

[0091] Step S830: Replace the coordinate data among the plurality of coordinate data whose difference with the filtered coordinate data reaches a preset difference threshold with the filtered coordinate data.

[0092] For example, in the process of determining the shortest distance point in the above embodiment, it is inevitable to encounter some situations where the actual measurement is close to the distance of multiple coordinate points, and slight changes will cause the position of the collection point to change, and ultimately the output HUD display position will frequently jitter within a small range. In order to avoid this situation, the multiple coordinate data collected can be filtered to obtain filtered data. After that, the difference between the multiple coordinate data and the filtered coordinate data can be calculated respectively, and the coordinate data whose difference between the multiple coordinate data and the filtered coordinate data reaches a preset difference threshold through comparison is replaced with the filtered coordinate data to achieve a smooth transition in the multiple coordinate data and avoid the occurrence of display position jitter and the like.

[0093] Optionally, select a suitable filtering algorithm to process the coordinate data. The choice of filtering algorithm depends on the characteristics of the data and the desired results. Common filtering methods include moving average filtering, Gaussian filtering, Kalman filtering, etc. Apply the selected filtering algorithm to obtain filtered coordinate data. These filtered data are usually smoother than the original data, reducing noise and mutations. For each original coordinate data point, calculate the difference between it and the corresponding filtered coordinate data point. This usually involves calculating the difference between each coordinate component (such as x and y). Summarize these differences so that they can be used in subsequent steps to determine which original data points need to be replaced. Set a preset difference threshold. This threshold determines which original data points are considered to be significantly different from the filtered data points and need to be replaced. Traverse all original coordinate data points and check whether the difference between them and the filtered data points exceeds the preset threshold. For those original data points whose difference exceeds the threshold, replace them with the corresponding filtered data points. This ensures that significant noise or mutations in the data are smoothed out while retaining most of the original features of the data.

[0094] Optionally, in some feasible embodiments, such as Fig. 9As shown, a moving average filtering algorithm can be used to filter the multiple collected coordinate data to obtain filtered coordinate data, and the first data obtained from the multiple coordinate data can be removed based on the filtered coordinate data, and then the filtered coordinate data can be added to the multiple coordinate data. Alternatively, the average value of all data in the multiple coordinate data can be used as the actual calculated value and then participate in the calculation of the shortest distance point in the above embodiment. In this way, smoothing processing can minimize the impact of jitter on the final output value.

[0095] In this embodiment, the filtering process can remove noise and abnormal values ​​in the original coordinate data, thereby obtaining smoother and more accurate coordinate data. This helps to improve the accuracy and reliability of subsequent processing. By calculating the difference between the original coordinate data and the filtered coordinate data and replacing the coordinate data whose difference exceeds a preset threshold, the fluctuation and instability factors in the data can be further reduced, which helps to ensure the stability and consistency of the data.

[0096] Based on the above examples, please refer to Fig.10 In one of the exemplary embodiments provided in the present application, before adjusting the second dot matrix coordinate system based on the target dot matrix density to obtain the corresponding sight line dot matrix, the specific implementation process of the head-up display system control method may further include step S1010 and step S1020, which are described in detail as follows:

[0097] Step S1010, obtaining vehicle operating condition information and driver information, where the driver information includes driver identity information and personalized information corresponding to the driver identity information;

[0098] Step S1020, determining a target lattice density based on the working condition information and the personalized information, so as to determine a second lattice coordinate system through the target lattice density.

[0099] Exemplary, driver identity information: including basic information such as the driver's name, gender, age, etc., personalized information: this may involve the driver's driving habits (such as preferred speed, acceleration, braking habits, etc.), sight height, speed of reading information, etc. This information helps the system better understand the driver's needs, thereby providing more personalized services. The driver's personalized needs may change as the driving time increases. For example, as the degree of driving fatigue increases, the driver may need a more concise and clear information display, so the accuracy requirement for sight tracking is lower, then the target dot matrix density can be set to a lower accuracy, such as driving at a low speed in a congested section of the city, or the driver mainly relies on other sensors and display screens to obtain information, then the HUD system can set the target dot matrix density to a lower level. A lower dot matrix density means that the display of information may be more concise and reduce details, but it can also meet basic driving needs. For drivers with less driving experience, they may rely more on the HUD system to assist driving, so they need more accurate information display. In this case, it is reasonable to set the target dot matrix density higher, because this can provide more detailed and intuitive information to help novice drivers adapt to and understand the vehicle status more quickly. That is, the HUD system should be able to dynamically adjust according to the driver's real-time status and needs to provide more personalized services.

[0100] In some embodiments of the present application, by obtaining the driver's identity information and its corresponding personalized information (such as driving habits, preference settings, etc.), the driver's actual needs can be matched more accurately. This personalized matching helps to improve the driving experience and make the driving assistance system more in line with the driver's usage habits. By obtaining the vehicle's operating condition information (such as vehicle speed, engine status, steering angle, etc.), the vehicle's driving status can be monitored in real time, and the parameters of the driving assistance system can be adjusted based on this information. This real-time adjustment helps to ensure that the driving assistance system can provide the best effect under different working conditions, thereby enhancing driving safety.

[0101] Based on the above examples, please refer to Fig.11 In one of the exemplary embodiments provided in the present application, the specific implementation process of the head-up display system control method may further include steps S1110 to S1140, which are described in detail as follows:

[0102] Step S1110, obtaining a calibration sample set, where the calibration sample set includes a plurality of pupil position information and target projection positions corresponding to the plurality of pupil position information;

[0103] Step S1120, establishing a mapping relationship between pupil position information and target projection positions based on the multiple pupil position information and the multiple target projection positions;

[0104] Step S1130, obtaining user operation information, where the operation information includes user operation information on the head-up display system;

[0105] Step S1140: correct the mapping relationship based on the operation information to obtain the mapping relationship between the pupil position information and the actual projection position.

[0106] Exemplarily, a camera or other sensor in the HUD system is used to capture the pupil position information of the user in different head postures. This usually includes the X and Y coordinates of the pupil in the screen or image coordinate system. For each pupil position information, a desired projection position, i.e., the target projection position, is set manually or automatically. This can be a specific point or area on the HUD image. Using the data in the calibration sample set, a machine learning model (such as a regression model, a neural network, etc.) is trained to learn the mapping relationship from pupil position information to the target projection position, and cross-validation or other validation methods can be used to evaluate the accuracy of the model to ensure that the mapping relationship can also perform well on unseen data.

[0107] Through the sensors and interfaces of the HUD system, the user's operations on the HUD, such as head movement, eye gaze, etc., are continuously monitored. The user's operation information is recorded, including the operation type, timestamp, and related pupil position information. Based on the user's operation information, the user's satisfaction and feedback on the HUD projection position are analyzed. If the user's feedback indicates that the current mapping relationship is not accurate enough, the incremental learning or online learning function of the machine learning model is used to update the mapping relationship. This can be achieved by introducing new training data (i.e., the pupil position information in the user's operation information and the actual projection position). Verify the corrected mapping relationship: After correcting the mapping relationship, use the verification method again to ensure the accuracy and consistency of the mapping relationship.

[0108] Optionally, in some feasible embodiments, due to different heights, driving habits and driving postures of drivers, the human eye position may appear in different positions. The initial position of the human eye has little effect on the perception of the HUD imaging effect. The "human eye dot matrix" can be divided into several points with smaller precision for calibration, for example, calibration is performed every 5 cm. The calibration work requires the calibration tester to actually sit in the vehicle, change different postures to make the eyes in different positions, simulate various driving postures that the driver may have during driving, and use professional calibration tools to calibrate all adjustable hardware parameters of the HUD as variable parameters of the tool, so that the various angle parameters of the HUD can be changed in real time through the calibration tool.

[0109] In addition, the calibration tester needs to simulate the various postures of the driver so that the eyes are at each point to be calibrated in the "dot matrix" in turn, and then use the calibration tool to make the HUD achieve the best display effect, and record the projection angle information calibrated at this time. The calibration work of the "sight dot matrix" needs to be fully covered, and even the coordinate density of the "sight dot matrix" needs to be increased when the accuracy is not up to standard. For example, the HUD changes the display position by changing the projection angle. In the calibration work, the HUD screen is adjusted to the most suitable viewing position under the combination of these two dot matrices, and the projection angle information at this time is recorded as (-30, 120), and the two dot matrix combinations are bound to the projection angle. When reflected in the actual working conditions, when the driver's sight position meets this situation, the HUD projection will be adjusted to the (-30, 120) angle position, so that the same optimal display effect as during calibration can be achieved.

[0110] In some achievable embodiments of the present application, a calibration sample set is obtained and a mapping relationship between pupil position information and target projection position is established. By correcting the mapping relationship, the present embodiment can be personalized according to the user's preferences and driving habits, thereby improving the user experience.

[0111] Fig.12 1 is a simplified flow chart of controlling a head-up display system in an exemplary application scenario. In the application scenario shown in 12, image information including the user's eyes is obtained to obtain an eye area image based on the image information; the actual projection position of the head-up display system is obtained to establish a sight line dot matrix corresponding to the eye area image based on the actual projection position; the user's pupil position information is determined based on the sight line dot matrix, and the mapping relationship between the pupil position information and the actual projection position is obtained; a first dot matrix coordinate system is established in the eye area image based on the actual projection position; a target eye area is extracted from the eye area image based on the first dot matrix coordinate system; a second dot matrix coordinate system is established in the target eye area based on the actual projection position, and the second dot matrix coordinate system is adjusted based on the target dot matrix density to obtain the corresponding sight line dot matrix. If a change in pupil position information is detected, the target projection position of the head-up display system is determined based on the changed pupil position information and the mapping relationship, so as to control the reflection angle of the reflector of the head-up display system based on the target projection position. For the detailed implementation process, please refer to the records in the aforementioned embodiments, which will not be repeated here.

[0112] Fig.13 is a block diagram of a head-up display system control device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown in the figure is specifically configured in the intelligent terminal 210. The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.

[0113] like Fig.13 As shown, the exemplary head-up display system control device includes: an acquisition module 1310, which acquires image information including the user's eyes, so as to acquire an eye area image based on the image information; an establishment module 1320, which acquires the actual projection position of the head-up display system, so as to establish a sight line dot matrix corresponding to the eye area image based on the actual projection position; a determination module 1330, which determines the user's pupil position information based on the sight line dot matrix, and acquires a mapping relationship between the pupil position information and the actual projection position; a control module 1340, which determines the target projection position of the head-up display system based on the changed pupil position information and the mapping relationship if a change in the pupil position information is monitored, so as to control the reflection angle of the reflector of the head-up display system based on the target projection position.

[0114] According to one aspect of an embodiment of the present application, the above-mentioned establishment module 1320 also includes establishing a first dot matrix coordinate system in the eye area image based on the actual projection position; extracting the target periocular area from the eye area image based on the first dot matrix coordinate system; establishing a second dot matrix coordinate system in the target periocular area based on the actual projection position, and adjusting the second dot matrix coordinate system based on the target dot matrix density to obtain a corresponding line of sight dot matrix.

[0115] According to one aspect of an embodiment of the present application, the above-mentioned establishment module 1320 also includes obtaining a target human eye model, and determining a candidate periocular area from the eye area image based on the target human eye model; extracting multiple periocular coordinate data from the candidate periocular area based on a first dot matrix coordinate system; and determining the target periocular area based on the multiple periocular coordinate data.

[0116] According to one aspect of an embodiment of the present application, the above-mentioned establishment module 1320 also includes extracting pupil features in the target periorbital area; obtaining multiple periorbital coordinate data corresponding to the pupil features in the second dot matrix coordinate system; and determining the pupil position information of the pupil in the second dot matrix coordinate system based on the multiple periorbital coordinate data.

[0117] According to one aspect of an embodiment of the present application, the head-up display system control device further includes: a filtering module, which performs filtering processing on the acquired multiple coordinate data to obtain filtered coordinate data; calculates the difference between the multiple coordinate data and the filtered coordinate data; and replaces the coordinate data whose difference between the multiple coordinate data and the filtered coordinate data reaches a preset difference threshold with the filtered coordinate data.

[0118] According to one aspect of an embodiment of the present application, the above-mentioned establishment module 1320 also includes obtaining the vehicle's operating condition information and driver information, the driver information including driver identity information and personalized information corresponding to the driver identity information; determining the target dot matrix density based on the operating condition information and the personalized information, so as to determine the second dot matrix coordinate system through the target dot matrix density.

[0119] According to one aspect of an embodiment of the present application, the above-mentioned head-up display system control device also includes: a calibration module, obtaining a calibration sample set, the calibration sample set including multiple pupil position information and target projection positions corresponding to each of the multiple pupil position information; establishing a mapping relationship between the pupil position information and the target projection position based on the multiple pupil position information and the multiple target projection positions; obtaining user operation information, the operation information including the user's operation information for the head-up display system; correcting the mapping relationship based on the operation information to obtain a mapping relationship between the pupil position information and the actual projection position.

[0120] The head-up display system control device provided in the above embodiment and the head-up display system control method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the head-up display system control device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0121] An embodiment of the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the electronic device implements the head-up display system control method provided in the above-mentioned embodiments.

[0122] Fig.14 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. Fig.14 The computer system 1400 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0123] like Fig.14As shown, the computer system 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1402 or the program loaded from the storage part 1408 to the random access memory (RAM) 1403, such as executing the method in the above embodiment. In the RAM 1403, various programs and data required for system operation are also stored. The CPU 1401, the ROM 1402 and the RAM 1403 are connected to each other through the bus 1404. The input / output (I / O) interface 1405 is also connected to the bus 1404.

[0124] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0125] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1409, and / or installed from a removable medium 1411. When the computer program is executed by a central processing unit (CPU) 1401, various functions defined in the system of the present application are executed.

[0126] The computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0127] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0129] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the head-up display system control method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0130] Another aspect of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the head-up display system control method provided in each of the above embodiments.

[0131] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A head-up display system control method, characterized in that: include: Acquire image information including eyes of the user, so as to acquire the eye region image based on the image information; Acquire an actual projection position of a head-up display system, so as to establish a sight line dot matrix corresponding to the eye region image based on the actual projection position; Determining pupil position information of the user based on the sight line dot matrix, and acquiring a mapping relationship between the pupil position information and the actual projection position; If a change in the pupil position information is detected, a target projection position of the head-up display system is determined based on the changed pupil position information and the mapping relationship, so as to control a reflection angle of a reflector of the head-up display system based on the target projection position.

2. The method according to claim 1, characterized in that The establishing of a sight line dot matrix corresponding to the eye region image based on the actual projection position comprises: Establishing a first dot matrix coordinate system in the eye area image based on the actual projection position; Extracting a target eye periorbital region from the eye region image based on the first dot matrix coordinate system; A second dot matrix coordinate system is established in the target periorbital area based on the actual projection position, and the second dot matrix coordinate system is adjusted based on the target dot matrix density to obtain a corresponding sight line dot matrix.

3. The method according to claim 2, characterized in that The step of extracting a target eye periorbital region from the eye region image based on the first dot matrix coordinate system includes: Acquire a target human eye model, and determine a candidate eye periorbital region from the eye region image based on the target human eye model; Extracting a plurality of periorbital coordinate data from the candidate periorbital region based on the first dot matrix coordinate system; A target periocular area is determined based on the plurality of periocular coordinate data.

4. The method according to claim 2, characterized in that The method further comprises: Extracting pupil features in the target eye periorbital region; Acquire a plurality of pupil periphery coordinate data corresponding to the pupil feature in the second dot matrix coordinate system; Pupil position information of the pupil in the second dot matrix coordinate system is determined based on the plurality of pupil circumference coordinate data.

5. The method according to any one of claims 3 and 4, characterized in that The method further comprises: Performing filtering processing on the acquired multiple coordinate data to obtain filtered coordinate data; Calculating the difference between the plurality of coordinate data and the filtered coordinate data; The coordinate data among the plurality of coordinate data whose difference with the filtered coordinate data reaches a preset difference threshold or more is replaced with the filtered coordinate data.

6. The method according to claim 2, characterized in that Before adjusting the second dot matrix coordinate system based on the target dot matrix density to obtain the corresponding sight line dot matrix, the method further includes: Acquiring vehicle operating condition information and driver information, wherein the driver information includes driver identity information and personalized information corresponding to the driver identity information; The target dot matrix density is determined based on the operating condition information and the personalized information, so as to determine the second dot matrix coordinate system through the target dot matrix density.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a calibration sample set, wherein the calibration sample set includes a plurality of pupil position information and target projection positions corresponding to each of the plurality of pupil position information; Establishing a mapping relationship between the pupil position information and the target projection positions based on the multiple pupil position information and the multiple target projection positions; Acquiring operation information of the user, wherein the operation information includes operation information of the user on the head-up display system; The mapping relationship is corrected based on the operation information to obtain a mapping relationship between the pupil position information and the actual projection position.

8. A head-up display system control device, characterized in that: The device comprises: An acquisition module, which acquires image information including the user's eyes, so as to acquire the eye area image based on the image information; An establishment module is provided to obtain an actual projection position of a head-up display system, so as to establish a sight line dot matrix corresponding to the eye area image based on the actual projection position; A determination module, which determines the pupil position information of the user based on the sight line dot matrix, and obtains a mapping relationship between the pupil position information and the actual projection position; The control module determines a target projection position of the head-up display system based on the changed pupil position information and the mapping relationship if it is detected that the pupil position information changes, so as to control a reflection angle of a reflector of the head-up display system based on the target projection position.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the head-up display system control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the head-up display system control method according to any one of claims 1 to 7.

Images