Head-up display system control method, device, vehicle, medium and program

Abstract

The present application relates to the technical field of display control, and discloses a control method and device of a head-up display system, a vehicle, a medium and a program, which collects user sound data by using sound collecting devices located at different positions, and then determines sound source coordinates and a sound source orientation by sound source positioning. Next, the distance between the user's eyes and the sound production site is determined, and the sound source coordinates and the sound source orientation are corrected according to the distance and the voiceprint characteristics. The method has less environmental interference, can obtain more accurate user eye coordinates and eye view angles, is more efficient than tracking the user's eyeball focal point by using a camera, and has lower requirements for system hardware configuration. Finally, the light machine projection direction is adjusted according to the user eye coordinates and the eye view angles, so as to realize eye following of the projection content, without the need for the user to manually adjust the projection direction, thereby improving the user's use experience.

Description

Technical Field

[0001] The present invention relates to the field of display control technology, and in particular to a control method, device, vehicle, medium and program for a head-up display system. Background Art

[0002] The augmented reality head-up display (AR-HUD) system projects important driving information, such as speed and navigation, onto the windshield in front of the driver, allowing the driver to see this information without looking down or turning their head. AR-HUD is a new type of automotive HUD that combines augmented reality technology with head-up display (HUD). It boasts a wider field of view and a longer imaging distance, and can directly overlay the display onto the real road surface, resulting in an outstanding overall display effect.

[0003] Currently, the existing technology requires the driver to manually adjust the projection position of the optical machine, and manual operation is used to ensure the driver's viewing angle height. However, if the driver operates the optical machine while driving, it will distract the driver's attention. During the operation, the driver's eye focus will not be consistent with the driving focus. After the manual operation is completed, there will be a deviation between the projection angle and the ideal angle. Every time the driver changes, the projection angle of the optical machine must be readjusted, which is a poor user experience. In addition, the driver's eyes can be monitored by a camera, and the projection position can be adjusted by identifying the eye focus. This solution requires the cockpit camera to track the driver's gaze and use the chip computing power of the vehicle computer to perform a large amount of calculations. The hardware cost is high, and it will occupy a large amount of the vehicle computer's computing power for a long time to track the eyeballs. It is not suitable for vehicles with lower hardware configurations. Summary of the Invention

[0004] In view of this, the present invention provides a control method, device, vehicle, medium and program for a head-up display system to solve the problems that manual adjustment of the projection angle may easily lead to a poor user experience, and the use of a camera to adjust the projection angle following the user's eye focus requires high vehicle computer computing power configuration.

[0005] In a first aspect, the present invention provides a method for controlling an augmented reality head-up display system, the system comprising an optical engine and a plurality of non-collinear sound receiving devices, the method comprising:

[0006] Obtain user voice data collected by each audio receiving device;

[0007] Determine the direction and coordinates of the sound source based on the user's voice data, and extract the voiceprint features in the user's voice data;

[0008] Determine the distance between the user's eyes and the vocalization site, and perform collaborative positioning based on the distance between the user's eyes and the vocalization site, voiceprint characteristics, sound source direction, and sound source coordinates to obtain the user's eye viewing angle and eye coordinates;

[0009] The target projection focus of the optical machine is determined based on the user's eye perspective and eye coordinates, and the optical machine is controlled to project the screen at the target projection focus.

[0010] Beneficial effects: The control method of the augmented reality head-up display system provided by the present invention utilizes sound receiving devices located at different positions to collect user sound data, and then determines the sound source coordinates and sound source direction through sound source positioning. Next, the distance between the user's eyes and the sound-producing part is determined, and the sound source coordinates and sound source direction are corrected according to the distance and voiceprint characteristics. It is less affected by environmental interference and can obtain more accurate user eye coordinates and eye viewing angles. Compared with using a camera to track the focus of the user's eyeballs, it is more efficient and has lower requirements for system hardware configuration. Finally, the projection direction of the optical machine is adjusted according to the user's eye coordinates and eye viewing angle to achieve eye tracking of the projection content. The user does not need to manually adjust the projection direction, which improves the user experience.

[0011] In some optional implementations, collaborative positioning is performed based on the distance between the user's eyes and the sound source, voiceprint characteristics, sound source direction, and sound source coordinates to obtain the user's eye perspective and eye coordinates, including:

[0012] Determine the user's current vocalization location based on voiceprint features;

[0013] Obtaining a calibrated distance between the user's eyes and the current vocalization site based on the distance between the user's eyes and the vocalization site, wherein the vocalization site includes the nose and throat;

[0014] The sound source direction and coordinates are corrected based on the calibration distance to obtain the user's eye perspective and eye coordinates.

[0015] Beneficial effects: The present invention identifies the voiceprint features to determine whether the user's current vocalization site is the nose or the throat, and then obtains the distance between the nose and the throat and the user's eyes respectively, thereby obtaining the calibrated distance between the current vocalization site and the eyes. Based on the calibrated distance, the sound source direction and the sound source coordinates are translated to obtain the user's eye perspective and eye coordinates, thereby determining the user's perspective focus. There is no need to continuously track the eye focus through a camera, and the hardware configuration requirements are relatively low.

[0016] In some optional implementations, determining the sound source direction and the sound source coordinates based on the user sound data includes:

[0017] Perform sound source localization based on the user's voice data, determine the sound source arrival angle of each sound receiving device, and obtain the sound source direction corresponding to the user's voice location;

[0018] According to the user's voice data, the volume data of the user's voice collected by each sound receiving device is obtained;

[0019] Obtaining a volume ratio between two of the plurality of sound receiving devices based on the volume data, and determining a preset relative distance between two of the plurality of sound receiving devices;

[0020] The sound source coordinates corresponding to the user's vocal position are obtained based on the sound source arrival angle, the preset relative distances between multiple sound receiving devices, and the volume ratio.

[0021] Beneficial Effects: The present invention uses sound source localization to determine the angle of arrival of each sound source on each audio device, thereby obtaining the sound source orientation at the user's vocal location, facilitating the determination of the user's viewing angle. Next, the user's voice volume data is used to analyze the volume ratios between each audio device and calculate the distance between the audio device and the sound source. This allows the user's eye coordinates to be determined based on the sound source coordinates. Based on the user's eye coordinates and viewing angle, eye tracking of the projected content is achieved.

[0022] In some optional embodiments, obtaining the sound source coordinates corresponding to the user's sound emitting position based on the sound source arrival angle, the preset relative distances between the multiple sound receiving devices, and the volume ratio includes:

[0023] Determining a sound source distance ratio between each of the plurality of sound receiving devices based on a volume ratio between each of the plurality of sound receiving devices;

[0024] Calculate the sound source distance between each sound receiving device and the user's sound source location based on the sound source arrival angle, the preset relative distances between multiple sound receiving devices, and the sound source distance ratio;

[0025] According to the sound source distance corresponding to each sound receiving device, the sound source coordinates corresponding to the user's vocalization part are obtained.

[0026] Beneficial effects: The present invention utilizes the volume ratio between each pair of sound receiving devices to measure the sound source distance ratio between each pair of sound receiving devices, and calculates the sound source distance between the sound receiving device and the sound source based on the sound source arrival angle and the preset relative distance between multiple sound receiving devices, thereby determining the sound source coordinates. The calculation method is simple and has low requirements on computing resources and hardware configuration.

[0027] In some optional embodiments, the sound source distance between each sound receiving device and the user's sound source position is calculated based on the sound source arrival angle, the preset relative distances between the multiple sound receiving devices, and the sound source distance ratio, including:

[0028] For each pair of sound receiving devices, a sound source distance association relationship is obtained based on a first sound source arrival angle of the first sound receiving device, a second sound source arrival angle of the second sound receiving device, and a preset relative distance between the first sound receiving device and the second sound receiving device; wherein the sound source distance association relationship is used to represent a numerical association between the first sound source distance of the first sound receiving device and the second sound source distance of the second sound receiving device;

[0029] The proportional relationship and numerical correlation of the sound source distance between each two sound receiving devices are solved to obtain the sound source distance between each sound receiving device and the user's sound source position.

[0030] Beneficial effect: The present invention performs spatial position analysis on the preset relative distance between each two sound receiving devices and the sound source arrival angle, determines the numerical correlation between the sound source distances of each two sound receiving devices, and then uses the proportional relationship of the sound source distances between each two sound receiving devices to solve the specific size of the sound source distance. The above process is repeated to calculate the sound source distance between each sound receiving device and the user's sound source part. The calculation method is simple and efficient, which can greatly improve the calculation efficiency.

[0031] In some optional implementations, determining the target projection focus of the optical engine based on the user's eye perspective and eye coordinates includes:

[0032] According to the user's eye perspective and eye coordinates, the visual focus coordinates of the user's line of sight at the preset projection position are obtained;

[0033] Based on the visual focus coordinates, determine the target projection focus of the optical machine.

[0034] Beneficial effects: The present invention calculates the visual focus coordinates of the user's line of sight at a preset projection position through the user's eye perspective and eye coordinates, thereby adjusting the projection focus of the optical machine and displaying the projection content within the user's visual range. It can display information to the user, assist driving and assist decision-making, etc., greatly improving the user experience.

[0035] In some optional implementations, before controlling the optical engine to project the screen at the target projection focus, the method further includes:

[0036] In response to a screen projection focus correction instruction issued by a user, obtaining a screen projection focus correction parameter;

[0037] Adjust the coordinates of the target projection focus according to the projection focus correction parameters.

[0038] Beneficial effect: The present invention responds to the user's projection focus correction instructions and adjusts the coordinates of the projection focus, so that the projection position of the projection content is more reasonable, the clarity is higher, and the color is clearer, thereby improving the projection display effect and user experience.

[0039] In some optional implementations, controlling the optical engine to project the screen at a target projection focus includes:

[0040] Determine the projection content of the optical machine; wherein the projection content includes at least one of device operating status information, video information, and audio information;

[0041] Control the optical machine to project the screen content to the target screen focus.

[0042] Beneficial effects: The present invention projects the device operating status information on the screen, making it easier for users to understand the device operating status and assisting users in making decisions and controlling the device; by projecting video information and audio information on the screen, it provides users with entertainment when the device is idle, thereby improving the user experience.

[0043] In some optional implementations, determining the projection content of the optical machine includes:

[0044] Obtain the vehicle's speed, navigation information, and road obstacle information to obtain the projection content of the optical machine.

[0045] Beneficial effects: In vehicle driving scenarios, the present invention can project navigation information such as the vehicle's driving speed, navigation route, direction, and information on road obstacles in front of the vehicle onto the screen, thereby improving the driver's driving safety and experience through assisted driving.

[0046] In some optional implementations, after controlling the optical engine to project the screen at the target projection focus, the method further includes:

[0047] In response to a screen projection content adjustment instruction issued by a user, obtaining a screen projection content adjustment parameter;

[0048] Adjust the parameters according to the projection content and adjust at least one of the device operating status information, video information and audio information of the optical screen projection.

[0049] Beneficial effect: After receiving the projection content adjustment instruction issued by the user, the present invention adjusts the projection content of the optical machine. For example, it can switch the display of device operation status information, video information and audio information to meet the user's projection needs in various usage scenarios.

[0050] In some optional implementations, determining the distance between the user's eyes and the sound-producing location includes:

[0051] Obtaining a user's facial image, extracting facial features from the user's facial image, and determining the distance between the user's eyes and the vocalization site based on the facial features;

[0052] Alternatively, the distance between the user's eyes and the vocalization site is obtained based on a preset facial feature distance.

[0053] Beneficial Effects: This invention extracts facial features from a user's facial image to determine the distance between the user's eyes and the source of sound, thereby correcting the sound source location and obtaining more accurate eye viewing angles and eye coordinates. Alternatively, the distance between facial features can be preset, and the sound source location can be corrected based on the preset distance to obtain the corrected eye viewing angles and eye coordinates, thereby improving computational efficiency.

[0054] In some optional embodiments, the method further comprises:

[0055] Storing the target projection focus corresponding to the voiceprint feature;

[0056] After the optical machine is restarted, when it is detected that the newly acquired voiceprint feature is consistent with the voiceprint feature, the optical machine is controlled to project the screen according to the target projection focus corresponding to the voiceprint feature.

[0057] Beneficial effect: By recording the user's voiceprint characteristics, when the optical machine is restarted, the user who spoke is identified by the voiceprint characteristics, and then the optical machine is controlled to project the screen according to the stored target projection focus. This eliminates the need to recalculate the user's visual focus, saves the time for confirming the projection focus, and provides a better user experience.

[0058] In a second aspect, the present invention provides a control device for an augmented reality head-up display system, the system comprising an optical engine and a plurality of non-collinear sound receiving devices, the device comprising:

[0059] An acquisition module is used to obtain user voice data collected by each sound receiving device;

[0060] A first processing module is used to determine the direction and coordinates of the sound source based on the user voice data, and to extract the voiceprint features in the user voice data;

[0061] The second processing module is used to determine the distance between the user's eyes and the sound source, and perform collaborative positioning based on the distance between the user's eyes and the sound source, the voiceprint characteristics, the direction of the sound source, and the coordinates of the sound source to obtain the user's eye angle and eye coordinates;

[0062] The third processing module is used to determine the target projection focus of the optical machine based on the user's eye perspective and eye coordinates, and control the optical machine to project the screen with the target projection focus.

[0063] In a third aspect, the present invention provides a vehicle comprising: an augmented reality head-up display system and a controller, the system comprising an optical engine and a plurality of non-collinear sound receiving devices;

[0064] The controller includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the control method of the augmented reality head-up display system of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the control method of an augmented reality head-up display system according to the first aspect or any corresponding embodiment thereof.

[0066] In a fifth aspect, the present invention provides a computer program product comprising computer instructions, the computer instructions being used to enable a computer to execute the control method of an augmented reality head-up display system according to the first aspect or any corresponding embodiment thereof.

[0067] The beneficial effects of the present invention are:

[0068] Compared to existing technologies that use cameras to track the user's facial image in real time and process the user's facial image through algorithms to determine the eye position and visual focus, the present invention uses sound source localization and distance correction to obtain the user's eye perspective and eye coordinates, eliminating the need for real-time tracking of the user's facial image. This algorithm consumes less energy, is more efficient, and requires fewer hardware resources. Furthermore, even if the user's head turns left or right but the perspective remains unchanged, the projection screen position can be maintained within the user's perspective, eliminating frequent adjustments to the projection focus due to head rotation, avoiding lens disturbances and thus improving the user experience.

[0069] In addition, compared with the solution of using a camera to track the eyes, the present invention not only improves the algorithm processing efficiency, but also uses the sound source direction and sound source coordinates calculated by sound source positioning to have a high reference value for adjusting the projection focus. It does not use higher computing power resources to process invalid data and data with low reference value, saving computing resources, and can ensure that the projection focus is within the user's field of view. Compared with the solution of camera facial recognition, the projection accuracy is higher.

[0070] The control method of the augmented reality head-up display system provided by the present invention utilizes sound receiving devices located at different positions to collect user sound data, and then determines the sound source coordinates and sound source direction through sound source positioning. Next, the distance between the user's eyes and the sound-producing part is determined, and the sound source coordinates and sound source direction are corrected according to the distance and voiceprint characteristics. It is less affected by environmental interference and can obtain more accurate user eye coordinates and eye viewing angles. Compared with using a camera to track the focus of the user's eyeballs, it is more efficient and has lower requirements for system hardware configuration. Finally, the projection direction of the optical machine is adjusted according to the user's eye coordinates and eye viewing angle to achieve eye tracking of the projection content. The user does not need to manually adjust the projection direction, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 is a flow chart of a control method for an augmented reality head-up display system according to an embodiment of the present invention;

[0073] Figure 2 is a schematic diagram of a vehicle coordinate system according to an embodiment of the present invention;

[0074] Figure 3 is a schematic structural diagram of a vehicle according to an embodiment of the present invention;

[0075] Figure 4 is a schematic diagram of a spherical coordinate system according to an embodiment of the present invention;

[0076] Figure 5 is a schematic diagram of a screen projection display effect according to an embodiment of the present invention;

[0077] Figure 6 is a flow chart of another method for controlling an augmented reality head-up display system according to an embodiment of the present invention;

[0078] Figure 7 is a structural block diagram of a control device for an augmented reality head-up display system according to an embodiment of the present invention;

[0079] Figure 8 4 is a schematic diagram of the hardware structure of a vehicle controller according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0081] Screen projection refers to the use of a set of light rays to project the shape of an object onto a flat surface. The AR-HUD system can project important driving information, such as speed and navigation, onto the windshield in front of the driver, allowing the driver to see it without looking down or turning their head. AR-HUD is a new type of automotive HUD that combines augmented reality technology with a heads-up display (HUD). It boasts a wider field of view and a longer imaging distance, and can directly overlay the display onto the real road surface, resulting in outstanding overall display quality. It is a key component of intelligent driving and intelligent cockpit solutions.

[0082] A major feature of AR-HUD display is its integration of virtual and real elements. In actual driving scenarios, AR-HUD displays can provide intuitive and accurate route, direction, and lane-level navigation guidance, as well as real-time warnings of pedestrians, vehicles, traffic lights, obstacles, and more. When displaying AR effects, various data and guidance effects need to be projected onto the screen, and finally displayed on the windshield. The position of the information on the windshield is determined by the optical engine.

[0083] The existing technical solution is that the driver manually adjusts the projection position of the optical machine, and ensures the driver's viewing angle height through manual operation. This solution can adjust the projection angle of the optical machine, but requires manual operation. During driving, the driver's attention is on observing the road. If the optical machine is operated, it will distract the driver's attention. During the operation, the driver's eye focus is inconsistent with the eye focus when driving. After the manual operation is completed, there will be an error between the projection angle and the ideal angle. Every time the driver is changed, the projection angle of the optical machine must be readjusted, which results in a poor user experience.

[0084] In addition, there is a solution that uses multiple cameras to monitor the driver's eyes and adjusts the position of the projection screen by identifying the focus of the eyeballs. This solution requires a camera in the cockpit to track the driver's gaze. After tracking, a large amount of calculations must be performed using the chip computing power of the car computer. This solution has a high hardware cost and will occupy a large amount of computing power of the car computer for a long time to track the eyeballs. It is not suitable for car computers with low hardware configuration.

[0085] Therefore, an embodiment of the present invention provides a control method for an augmented reality head-up display system, which calculates the sound source coordinates and sound source direction through the sound emitted by the user, calculates the user's eye coordinates, that is, the viewing angle height, using the user's facial features and the sound source coordinates, and calculates the user's eye viewing angle direction through the sound source direction, and then adjusts the optical projection direction of the AR-HUD system according to the viewing angle height and viewing angle direction to achieve eye tracking of the projection content. There is no need for the user to manually adjust the projection direction, and the hardware configuration requirements of the system are relatively low, which is conducive to reducing costs and improving user experience.

[0086] According to an embodiment of the present invention, an embodiment of a control method for an augmented reality head-up display system is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0087] In this embodiment, a control method for an augmented reality head-up display system is provided, which can be used for devices equipped with an AR-HUD system, such as vehicles, projectors, etc. Figure 1 is a flow chart of a control method of an augmented reality head-up display system according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0088] Step S101: Acquire user voice data collected by each sound receiving device.

[0089] Specifically, the embodiment of the present invention is described by taking a vehicle as an example. Figure 2 As shown, first, a local coordinate system is established with the vehicle itself as a reference. The origin of the coordinate system is set at the center of the vehicle's main driver's seat, with the left and right direction of the driver as the x-axis (right is positive, left is negative), the front and back direction of the driver as the y-axis (front is positive, back is negative), and the z-axis perpendicular to the ground (high is positive, low is negative).

[0090] Specifically, if Figure 3 As shown, an embodiment of the present invention provides a vehicle having an AR-HUD system and a controller for controlling the AR-HUD system ( Figure 2 Not shown), wherein the AR-HUD system includes an optical machine for projection and multiple non-collinear sound receiving devices. The multiple sound receiving devices cannot be located on the same straight line, otherwise the three-dimensional coordinates of the sound source cannot be determined when locating the sound source. Figure 3The installation positions of three sound receivers are shown. These receivers are installed in a triangular shape. The three receivers cannot be on the same axis at the same time, that is, they cannot be on the x, y, and z axes at the same time. The position of the optical engine can be installed based on the remaining space in the vehicle. For example, the sound receiver can be a device such as a microphone array.

[0091] Step S102: determining the direction and coordinates of the sound source based on the user voice data, and extracting the voiceprint features in the user voice data.

[0092] Specifically, when the user speaks, Figure 3 The sound receiving device shown receives the user's voice data. The sound source is located according to the user's voice data, such as Figure 4 As shown, sound source localization usually uses a spherical coordinate system, and the coordinate information includes distance, azimuth, and elevation angle information.

[0093] In some optional embodiments, the relative distances between each pair of sound receiving devices can be preset. Since the speed of sound propagation in the air is fixed, the proportional relationship between the sound source and the sound source distances between different sound receiving devices is also fixed. In addition, the sound source distance between the sound source and the sound receiving device, the sound source arrival angle when the sound reaches the sound receiving device, and the preset relative distances between each pair of sound receiving devices have a certain numerical correlation. Therefore, based on these correlations and proportional relationships, the sound source distance from the sound source to the sound receiving device can be calculated. A sphere can be drawn with the sound receiving device as the center and its corresponding sound source distance as the radius, and the sound source must be on this sphere; using the same calculation method, multiple spheres with different sound receiving devices as the center will be obtained, and the position where the multiple spheres intersect is the sound source coordinate (x, y, z). In addition, the sound source direction r can be calculated based on the controllable beamforming technology with maximum output power.

[0094] In some optional embodiments, sound source positioning can also be performed based on the relative time delay between different sound receiving devices. The user sound data obtained by a single sound receiving device is used to calculate the sound source distance and the sound source arrival angle to determine a two-dimensional positioning of the sound source. Then, through the two-dimensional positioning calculated by each microphone, the sound source coordinates and sound source direction in three-dimensional space can be calculated. The specific calculation process is detailed in the description of the relevant technology and will not be repeated here.

[0095] Step S103, determining the distance between the user's eyes and the sound source, and performing collaborative positioning based on the distance between the user's eyes and the sound source, voiceprint characteristics, sound source direction, and sound source coordinates to obtain the user's eye perspective and eye coordinates.

[0096] In some optional embodiments, a user's facial image can be acquired, facial features can be extracted from the user's facial image, and the distance between the user's eyes and the vocalization site can be determined based on the facial features. For example, a camera can be used to acquire the user's facial image. Since vocalization occurs in the throat / nose and visual observation occurs in the eyes, the distance between the throat / nose and eyes can be determined through facial feature recognition. Therefore, the sound source position can be corrected based on the facial features and voiceprint features to obtain more accurate eye perspective and eye coordinates.

[0097] In some optional implementations, the distance between the user's eyes and the vocalization site can be determined based on a preset facial feature distance. The distance between the vocalization site and the eyes varies widely from person to person. Therefore, by presetting the distance between the throat / nose and the eyes, the sound source position can be corrected based on the preset distance to obtain the corrected eye perspective and eye coordinates, thereby improving computational efficiency.

[0098] Step S104: determine the target projection focus of the optical machine based on the user's eye perspective and eye coordinates, and control the optical machine to project the screen at the target projection focus.

[0099] Specifically, taking a vehicle as an example, Figure 5 As shown, the target projection focus during optical projection can be determined based on the user's eye perspective and the visual focus of the eye coordinates at the projection position in the car. In a projection scenario within a 3D space, since the user's visual focus position changes, when the visual focus changes, the user's eye perspective and eye coordinates must be tracked in real time to project the screen to the visual focus position. This allows the user to determine the user's visual focus and project the content they want to observe to the real-time acquired position.

[0100] Specifically, the projected content of the optical machine must first be determined, and then the optical machine must be controlled to project the content to the target projection focus. The projected content includes at least one of the following: device operating status information, video information, and audio information. Projecting the device operating status information facilitates user understanding of the device's operating status, assisting users in decision-making and device control. Projecting video and audio information provides users with entertainment during their free time, thereby enhancing the user experience.

[0101] In some optional implementations, the vehicle's speed, navigation information, and road obstacle information are obtained to obtain the projection content of the optical machine. In the AR-HUD projection scenario in the field of automotive smart cockpits, due to the differences in height, driving position, etc. of different drivers, the visual focus position on the windshield is also different. The present invention can automatically locate the driver's visual focus position, so that the AR projection content display position during AR-HUD projection is consistent with the visual focus of each driver, and the vehicle's speed, navigation route, direction, traffic lights and other navigation information as well as road obstacle information in front of the vehicle are projected, thereby improving the driver's driving safety and experience through assisted driving.

[0102] In aerospace scenarios, pilots must pay attention to the aircraft's operating data and make decisions at all times. The more timely the data within the field of view, the more accurate the decisions made by the pilot. By inventing a real-time positioning of the pilot's visual range, the aircraft data can be projected into the pilot's field of view in a timely manner, improving driving agility and the real-time nature of the pilot's decision-making.

[0103] In the high-speed rail cockpit scenario, the driver must pay attention to the high-speed rail operation data in real time while driving the high-speed rail. By locating the driver's observation direction in real time through the present invention, the high-speed rail operation data can be projected into the driver's visual range in real time, which can avoid distraction of the driver.

[0104] In home indoor scenes, by locating people's visual direction in real time and projecting the video onto the wall in that direction, people can watch the video while walking around and doing other things after returning home, instead of watching the video in a fixed position (direction) like traditional screen projection, which greatly improves the user experience.

[0105] In some optional embodiments, in response to a projection content adjustment instruction issued by the user, a projection content adjustment parameter is obtained. According to the projection content adjustment parameter, at least one of the device operating status information, video information, and audio information of the optical machine projection is adjusted. For example, in a driving scenario, the previously displayed content is the vehicle's operating status information. After the vehicle stops, the user can adjust the display content through voice commands and control the optical machine to display video information, thereby meeting the user's projection needs in various usage scenarios.

[0106] Compared with the existing technology that uses a camera to track the user's facial image in real time and processes the user's facial image through an algorithm to obtain the position of the eyeball and visual focus, the present invention uses sound source positioning and distance correction to obtain the user's eye perspective and eye coordinates. There is no need to track the user's facial image in real time, the algorithm consumes less energy, has higher processing efficiency, and has lower requirements for hardware resource configuration.

[0107] In addition, the projection focus of the optical machine should be consistent with the user's eye perspective but should not be adjusted frequently. Since the pupil position changes frequently due to the left and right rotation of the user's head, but the user's perspective is usually relatively fixed, when using eye tracking to adjust the visual focus, the projection focus changes frequently, which can easily cause lens disturbance in the projection, greatly affecting the user's experience. In severe cases, it can also distract the user's attention and cause the user to make wrong driving decisions. The present invention uses sound source positioning and distance correction to obtain the user's eye coordinates and eye perspective, thereby adjusting the projection focus of the optical machine. When the user's head rotates left and right but the perspective remains unchanged, the projection position can be kept within the user's perspective, and the projection focus will not be adjusted frequently due to the user's head rotation, avoiding lens disturbance, thereby improving the user's experience.

[0108] In addition, when using a camera to track eye focus in real time, a two-dimensional eye image of the user is usually analyzed. However, due to the resolution and cost limitations of the camera, the information contained in the two-dimensional eye image is very limited, and the accuracy of the eye perspective obtained by eye image recognition is also low. The eye perspective obtained in this way does not have a high reference value for the adjustment process of the projection focus, and the computing power consumption is still very high. In actual applications, facial images are often only used to analyze the approximate coordinates of the user's eyes before projection is performed. It is difficult to ensure that the projection focus is consistent with the user's eye perspective focus, and the actual projection accuracy is not high. Compared with the solution of using a camera to track the eyes, the present invention improves the algorithm processing efficiency while using the sound source localization calculation of the sound source direction and sound source coordinates, which have a high reference value for adjusting the projection focus. It does not use high computing power resources to process invalid data and data with low reference value, saving computing resources, and can ensure that the projection focus is within the user's perspective. Compared with the solution of camera facial recognition, the projection accuracy is higher.

[0109] The control method of the augmented reality head-up display system provided in this embodiment uses sound receiving devices located at different positions to collect user sound data, and then determines the sound source coordinates and sound source direction through sound source positioning. Next, the distance between the user's eyes and the sound-producing part is determined, and the sound source coordinates and sound source direction are corrected according to the distance and voiceprint characteristics. It is less affected by environmental interference and can obtain more accurate user eye coordinates and eye viewing angles. Compared with using a camera to track the focus of the user's eyeballs, it is more efficient and has lower requirements for system hardware configuration and computing resources. Finally, the direction of the optical machine projection screen is adjusted according to the user's eye coordinates and eye viewing angle to achieve eye tracking of the projection content. The user does not need to manually adjust the projection direction, which improves the user experience.

[0110] In this embodiment, a control method for an augmented reality head-up display system is provided, which can be used for devices equipped with an AR-HUD system, such as vehicles, projectors, etc. Figure 6 is a flow chart of a control method of an augmented reality head-up display system according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0111] Step S601: Acquire the user's voice data collected by each sound receiving device. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0112] Step S602: determining the direction and coordinates of the sound source based on the user voice data, and extracting the voiceprint features in the user voice data.

[0113] Specifically, the above step S602 includes:

[0114] Step S6021: Perform sound source positioning based on the user's voice data, determine the sound source arrival angle of each sound receiving device, and obtain the sound source direction corresponding to the user's voice position.

[0115] Specifically, the direction of the sound source can be calculated using the sound receiving devices in the vehicle's interior space, utilizing controllable beamforming technology based on maximum output power. Beamforming is a relatively intuitive angle measurement method. It adjusts the receiving direction of the microphone array and, subject to certain criteria, forms a beam using the sound source signal received by the microphones. The beam then searches the entire airspace to determine the possible location of the sound source. The direction of the beam's maximum output power is the direction of the sound source, which in turn determines the angle of arrival of the sound source for each sound receiving device. Combined with the calculated angle of arrival for each sound receiving device, the direction of the sound source can be determined. The specific calculation process can be found in the description of related technologies and will not be elaborated on here.

[0116] Step S6022: According to the user voice data, obtain the volume data of the user voice collected by each sound receiving device.

[0117] Specifically, the received user voice data is converted into an analog signal, which is then converted into a digital signal via an analog-to-digital converter. The root mean square (RMS) value of the digital signal can be calculated to measure the intensity of the sound. This RMS value is then converted into a sound pressure level (in decibels) to obtain the user's voice volume data. The specific calculation process can be found in the description of related technologies and will not be repeated here.

[0118] Step S6023: Obtain the volume ratio between each of the plurality of sound receiving devices according to the volume data, and determine the preset relative distance between each of the plurality of sound receiving devices.

[0119] Specifically, based on the boost level corresponding to the volume data, the decibel ratio of the volume between two radio devices can be calculated to obtain the volume ratio relationship. The installation distance of the radio devices is fixed, so the relative distance between two radio devices can be calculated based on the installation location.

[0120] Step S6024: Obtain the coordinates of the sound source corresponding to the user's vocal position based on the sound source arrival angle, the preset relative distances between the multiple sound receiving devices, and the volume ratio.

[0121] In some optional implementations, step S6024 includes:

[0122] Step a1: determining a sound source distance ratio between each of the plurality of sound receiving devices based on a volume ratio between each of the plurality of sound receiving devices.

[0123] Specifically, since the propagation speed of sound is fixed, its attenuation speed in the same medium is also fixed, so the volume ratio between two sound receiving devices can be used as the distance ratio between their sound sources.

[0124] Step a2: Calculate the sound source distance between each sound receiving device and the user's sound source location based on the sound source arrival angle, the preset relative distances between the multiple sound receiving devices, and the sound source distance ratio.

[0125] Specifically, for each pair of sound receiving devices, a sound source distance association is determined based on the first sound source arrival angle θ1 of the first sound receiving device, the second sound source arrival angle θ2 of the second sound receiving device, and the preset relative distance d between the first and second sound receiving devices. The proportional relationship and numerical association between the sound source distances between each pair of sound receiving devices are then solved to determine the sound source distance between each sound receiving device and the user's sound source location. The sound source distance association represents the numerical association between the first sound source distance l1 of the first sound receiving device and the second sound source distance l2 of the second sound receiving device.

[0126] In some optional embodiments, the sound source distance relationship between the first sound receiving device and the second sound receiving device can be expressed as d = l1cosθ1-l2cosθ2, where the first sound source distance l1 is greater than the second sound source distance l2. Then, based on the sound source distance relationship between the first sound source distance l1 and the second sound source distance l2 and the sound source distance proportional relationship (i.e., l1=k*l2, where k is the proportionality coefficient), the specific values ​​of the first sound source distance l1 and the second sound source distance l2 can be obtained by solving an equation.

[0127] The present invention performs spatial position analysis on the preset relative distance between each two sound receiving devices and the sound source arrival angle, determines the numerical correlation between the sound source distances of each two sound receiving devices, and then uses the proportional relationship of the sound source distances between each two sound receiving devices to solve the specific size of the sound source distance. By repeating the above process, the sound source distance between each sound receiving device and the user's sound source part can be calculated. The calculation method is simple and efficient, which can greatly improve the calculation efficiency.

[0128] Step a3: Obtain the sound source coordinates corresponding to the user's vocal position according to the sound source distance corresponding to each sound receiving device.

[0129] Specifically, a sphere can be drawn with the radio device as the center and the corresponding sound source distance as the radius, and the sound source must be on this sphere; using the same calculation method, multiple spheres with different radio devices as the center will be obtained, and the position where multiple spheres intersect is the sound source coordinate (x, y, z).

[0130] The present invention uses sound source localization to determine the angle of arrival of each sound source for each receiver, thereby determining the sound source orientation at the user's voice location and facilitating the determination of the user's viewing angle. Next, the user's voice volume data is used to analyze the volume ratios between each receiver and measure the sound source distance ratios between each receiver. This facilitates the calculation of the sound source distances between the receivers and the sound source, and determines the user's eye coordinates. This calculation is simple and requires minimal computing resources and hardware configuration.

[0131] Step S6025: extract voiceprint features from the user's voice data.

[0132] Specifically, neural networks, image recognition and other algorithms can be used to extract facial features in images, and voiceprint recognition and other algorithms can be used to extract user voiceprint features. For details, please refer to the description of related technologies and will not be repeated here.

[0133] Step S603, determine the distance between the user's eyes and the sound source, perform collaborative positioning based on the distance between the user's eyes and the sound source, voiceprint characteristics, sound source direction and sound source coordinates, and obtain the user's eye perspective and eye coordinates.

[0134] Specifically, the above step S603 includes:

[0135] Step S6031: Determine the user's current vocalization location based on the voiceprint features.

[0136] Specifically, the vocal parts mainly include the nose and throat. When the human body speaks, the vocal part is in the throat, and when breathing, the vocal part is in the nose. Due to the different vocal parts, the voiceprints are also different. The voiceprint features can be used to distinguish whether the calculated sound source coordinates are in the nose or throat.

[0137] Step S6032: Obtain a calibrated distance between the user's eyes and the current sound-producing part according to the distance between the user's eyes and the sound-producing part.

[0138] Specifically, based on the distance characteristics between different facial organs, the distances between the nose and throat and the eyes are obtained respectively. Then, by determining whether the current vocal part is the nose or throat, the calibrated distance between the current vocal part and the eyes can be obtained.

[0139] Step S6033: Correct the sound source direction and sound source coordinates based on the calibration distance to obtain the user's eye perspective and eye coordinates.

[0140] Specifically, according to the calibrated distance between the current sound source and the eye, the sound source direction and the sound source coordinates are translated in the z-axis direction to obtain the user's eye perspective and eye coordinates.

[0141] The present invention identifies voiceprint features to determine whether the user's current vocalization site is the nose or throat, and then obtains the distance between the nose and throat and the user's eyes respectively, thereby obtaining the calibrated distance between the current vocalization site and the eyes. Based on the calibrated distance, the sound source direction and sound source coordinates are translated to obtain the user's eye perspective and eye coordinates, thereby determining the user's perspective focus. There is no need to continuously track the eye focus through a camera, which has low hardware configuration requirements and reduces costs.

[0142] Step S604: determine the target projection focus of the optical machine based on the user's eye perspective and eye coordinates, and control the optical machine to project the screen at the target projection focus.

[0143] Specifically, the above step S604 includes:

[0144] Step S6041: Obtain the visual focus coordinates of the user's line of sight at the preset projection position according to the user's eye perspective and eye coordinates, and determine the target projection focus of the optical machine based on the visual focus coordinates.

[0145] Specifically, taking the vehicle as an example, see again Figure 5 Using the user's eye perspective and eye coordinates, along with pre-acquired window structure data, we can calculate the intersection of the user's gaze and the window glass. This intersection is the visual focus of the gaze on the window glass. The projection focus of the AR-HUD optical machine is the user's visual focus coordinates.

[0146] Step S6042, control the optical machine to project the screen with the target projection focus.

[0147] Specifically, by adjusting the focus of the optical machine projection screen, the projection content can be displayed within the user's visual range, and can be adjusted accordingly according to changes in the user's viewing angle.

[0148] In an embodiment of the present invention, the visual focus coordinates of the user's line of sight at the preset projection position are calculated through the user's eye perspective and eye coordinates, so as to adjust the projection focus of the optical machine and display the projection content within the user's visual range, which can display information to the user, assist driving and assist decision-making, etc., greatly improving the user experience.

[0149] In some optional embodiments, before controlling the optical engine to project the screen at the target projection focus, a projection focus correction parameter may be obtained in response to a projection focus correction instruction issued by the user, and the coordinates of the target projection focus may be adjusted based on the projection focus correction parameter. For example, the projection focus correction parameter may include not only coordinate position correction parameters, but also color correction, clarity correction, and other parameters.

[0150] The present invention responds to the user's projection focus correction instructions and adjusts the coordinates of the projection focus, so that the projection position of the projection content is more reasonable, the clarity is higher, and the color is clearer, thereby improving the projection display effect and user experience.

[0151] In some optional implementations, the target projection focus corresponding to the voiceprint feature can also be stored. In this way, after the optical engine is restarted, when it detects that the newly acquired voiceprint feature matches the voiceprint feature, the optical engine is controlled to project the screen according to the target projection focus corresponding to the voiceprint feature.

[0152] Specifically, taking a vehicle as an example, for the same user, their facial features and voiceprint features are fixed. If the seat position does not change, the corresponding target projection focus is also relatively fixed. Therefore, by recording the user's voiceprint features, when the optical machine restarts, the voiceprint features can be used to identify the user who spoke, and then the optical machine can be controlled to project the screen based on the stored target projection focus. This eliminates the need to recalculate the user's visual focus, saves time in confirming the projection focus, and provides a better user experience.

[0153] The control scheme of the augmented reality head-up display system of the present invention is described in detail below with reference to a specific application example.

[0154] Specifically, using a vehicle as an example, a local coordinate system is first established in 3D space, with the center of the driver's seat as the origin. This coordinate system is equipped with at least three microphones. The sound volume data collected by each microphone is used to calculate the distance to each microphone's sound source. The sound source coordinates in the local coordinate system are then calculated based on the distances between the sound source and each microphone. Simultaneously, sound source localization is performed on the user's voice data to analyze the direction of the sound source.

[0155] Secondly, the distance between the sound source and the eyeball is calibrated. When the human body speaks, the sound is produced in the throat, and when breathing, the sound is produced in the nose. Since the two sounds are produced in different places, the voiceprint is also different. The voiceprint characteristics can be used to distinguish whether the collected sound source coordinates are in the nose or throat. The precise distance of the nose and throat from the eyeball can be determined by camera recognition. Since the position of the nose and throat from the eyeball is fixed for the same user, it only needs to be recognized once, and real-time recognition is not required. Therefore, the precise eye coordinates are obtained based on the sound source coordinates and the precise distance of the nose and throat from the eyeball.

[0156] Then, since human vocalization is linear, that is, the sound produced is continuous at a certain moment, a straight line can be obtained according to the coordinates and direction of the sound source. The intersection of this straight line and the projection surface in front is the visual focus of the eyeball.

[0157] Finally, use the visual focus coordinates of the projection surface calculated in the above steps as the projection focus to adjust the projection angle of the projection machine.

[0158] For example, in a home setting, screen projection can be controlled by appliances with microphones, such as smart speakers and smart TVs. Furthermore, to establish visual positioning through sound in a home setting, the following conditions must be met: at least three sound receiving devices are required, the sound receiving devices must be fixed (i.e., the position of these microphone-equipped appliances does not change), and the range of movement of people must be within a range connected by multiple appliances (e.g., a triangle formed by three appliances).

[0159] After multiple audio devices are positioned at home, a local coordinate system is established. The coordinates and direction of the sound source are then determined using the calculation method described in the vehicle scenario. Unlike in-car scenarios, some home scenarios lack cameras for facial recognition, resulting in errors in the calculated projection focus (due to the lack of distance between the throat, mouth, and eyes). Therefore, in home scenarios, manual correction is performed by the user to address the error. After projection, the projection position is manually adjusted up and down. The difference in height between the two positions is the error. In subsequent projections, the calculated sound source position plus the manually corrected error is the visual focus position.

[0160] This application example uses a microphone as a tool for calculating the viewing angle and focus. It calculates the user's visual focus by calculating the coordinates and direction of the sound source, and can dynamically adjust the projection position to improve the projection visual effect. It uses automatic adjustment of the projection angle to reduce the spatial gap between the visual and projection focus.

[0161] This method uses existing sound collection devices (microphones) in 3D space to calculate the coordinates and direction of the sound source, then the person's line of sight and gaze direction. The intersection of the line of sight in the direction of the gaze and the projection surface is the projection position. This method combines sound localization with visual localization, calculating the real-time visual focus by acquiring the sound source position in real time. The calculation process is fast, has low hardware resource requirements (no GPU, memory, etc.), and provides accurate results with minimal environmental interference.

[0162] This embodiment also provides a control device for an augmented reality head-up display system, which is used to implement the above-mentioned embodiments and preferred implementations. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0163] This embodiment provides a control device for an augmented reality head-up display system, which includes an optical machine and multiple non-collinear sound receiving devices, such as Figure 7 As shown, the control device of the augmented reality head-up display system includes:

[0164] An acquisition module 701 is used to acquire user voice data collected by each sound receiving device;

[0165] A first processing module 702 is configured to determine a sound source direction and a sound source coordinate based on the user voice data, and extract a voiceprint feature from the user voice data;

[0166] The second processing module 703 is used to determine the distance between the user's eyes and the sound source, and perform collaborative positioning based on the distance between the user's eyes and the sound source, the voiceprint characteristics, the direction of the sound source, and the coordinates of the sound source to obtain the user's eye angle and eye coordinates;

[0167] The third processing module 704 is used to determine the target projection focus of the optical machine based on the user's eye perspective and eye coordinates, and control the optical machine to project the screen at the target projection focus.

[0168] In some optional implementations, the first processing module 702 is further configured to:

[0169] Perform sound source localization based on the user's voice data, determine the sound source arrival angle of each sound receiving device, and obtain the sound source direction corresponding to the user's voice location;

[0170] According to the user's voice data, the volume data of the user's voice collected by each sound receiving device is obtained;

[0171] Obtaining a volume ratio between two of the plurality of sound receiving devices based on the volume data, and determining a preset relative distance between two of the plurality of sound receiving devices;

[0172] The sound source coordinates corresponding to the user's vocal position are obtained based on the sound source arrival angle, the preset relative distances between multiple sound receiving devices, and the volume ratio.

[0173] In some optional implementations, the first processing module 702 is further configured to:

[0174] Determining a sound source distance ratio between each of the plurality of sound receiving devices based on a volume ratio between each of the plurality of sound receiving devices;

[0175] Calculate the sound source distance between each sound receiving device and the user's sound source location based on the sound source arrival angle, the preset relative distances between multiple sound receiving devices, and the sound source distance ratio;

[0176] According to the sound source distance corresponding to each sound receiving device, the sound source coordinates corresponding to the user's vocalization part are obtained.

[0177] In some optional implementations, the first processing module 702 is further configured to:

[0178] For each pair of sound receiving devices, a sound source distance association relationship is obtained based on a first sound source arrival angle of the first sound receiving device, a second sound source arrival angle of the second sound receiving device, and a preset relative distance between the first sound receiving device and the second sound receiving device; wherein the sound source distance association relationship is used to represent a numerical association between the first sound source distance of the first sound receiving device and the second sound source distance of the second sound receiving device;

[0179] The proportional relationship and numerical correlation of the sound source distance between each two sound receiving devices are solved to obtain the sound source distance between each sound receiving device and the user's sound source position.

[0180] In some optional implementations, the second processing module 703 is further configured to:

[0181] Obtaining a user's facial image, extracting facial features from the user's facial image, and determining the distance between the user's eyes and the vocalization site based on the facial features;

[0182] Alternatively, the distance between the user's eyes and the vocalization site is obtained based on a preset facial feature distance.

[0183] In some optional implementations, the second processing module 703 is further configured to:

[0184] Determine the user's current vocalization location based on voiceprint features;

[0185] Obtaining a calibrated distance between the user's eyes and the current vocalization site based on the distance between the user's eyes and the vocalization site, wherein the vocalization site includes the nose and throat;

[0186] The sound source direction and coordinates are corrected based on the calibration distance to obtain the user's eye perspective and eye coordinates.

[0187] In some optional implementations, the third processing module 704 is further configured to:

[0188] According to the user's eye perspective and eye coordinates, the visual focus coordinates of the user's line of sight at the preset projection position are obtained;

[0189] Based on the visual focus coordinates, determine the target projection focus of the optical machine.

[0190] In some optional embodiments, the device is further used to:

[0191] In response to a screen projection focus correction instruction issued by a user, obtaining a screen projection focus correction parameter;

[0192] Adjust the coordinates of the target projection focus according to the projection focus correction parameters.

[0193] In some optional implementations, the third processing module 704 is further configured to:

[0194] Determine the projection content of the optical machine; wherein the projection content includes at least one of device operating status information, video information, and audio information;

[0195] Control the optical machine to project the screen content to the target screen focus.

[0196] In some optional implementations, the third processing module 704 is further configured to:

[0197] Obtain the vehicle's speed, navigation information, and road obstacle information to obtain the projection content of the optical machine.

[0198] In some optional implementations, the third processing module 704 is further configured to:

[0199] In response to a screen projection content adjustment instruction issued by a user, obtaining a screen projection content adjustment parameter;

[0200] Adjust the parameters according to the projection content and adjust at least one of the device operating status information, video information and audio information of the optical screen projection.

[0201] In some optional embodiments, the device is further used to:

[0202] Storing the target projection focus corresponding to the voiceprint feature;

[0203] After the optical machine is restarted, when it is detected that the newly acquired voiceprint feature is consistent with the voiceprint feature, the optical machine is controlled to project the screen according to the target projection focus corresponding to the voiceprint feature.

[0204] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0205] The control device of the augmented reality head-up display system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0206] The embodiment of the present invention also provides a vehicle controller having the above Figure 7 Controls for the augmented reality head-up display system are shown.

[0207] See also Figure 8 , Figure 8 : is a schematic diagram of the structure of a controller provided by an optional embodiment of the present invention, such as Figure 8 As shown, the controller includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the controller, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0208] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0209] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0210] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0211] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0212] The controller further includes a communication interface 30 for the controller to communicate with other devices or a communication network.

[0213] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0214] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0215] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A control method for an augmented reality head-up display system, characterized in that: The system includes an optical machine and a plurality of non-collinear sound receiving devices, and the method includes: Acquiring user voice data collected by each of the sound receiving devices; determining a sound source direction and a sound source coordinate based on the user voice data, and extracting a voiceprint feature from the user voice data; Determine the distance between the user's eyes and the vocalization site, and perform collaborative positioning based on the distance between the user's eyes and the vocalization site, voiceprint characteristics, sound source direction, and sound source coordinates to obtain the user's eye viewing angle and eye coordinates; The target projection focus of the optical machine is determined based on the user's eye perspective and eye coordinates, and the optical machine is controlled to project the screen at the target projection focus.

2. The method according to claim 1, characterized in that The collaborative positioning based on the distance between the user's eyes and the sound source, the voiceprint characteristics, the direction of the sound source, and the sound source coordinates to obtain the user's eye perspective and eye coordinates includes: Determine the user's current vocalization location based on voiceprint features; Obtaining a calibrated distance between the user's eyes and the current sound-producing part according to the distance between the user's eyes and the sound-producing part; wherein the sound-producing part includes the nose and throat; The sound source direction and the sound source coordinates are corrected based on the calibration distance to obtain the user's eye perspective and eye coordinates.

3. The method according to claim 1 or 2, characterized in that The determining the sound source direction and the sound source coordinates based on the user sound data includes: Performing sound source localization based on the user's voice data, determining the sound source arrival angle of each of the sound receiving devices, and obtaining the sound source direction corresponding to the user's voice position; According to the user voice data, obtaining the volume data of the user voice collected by each of the sound receiving devices; Obtaining a volume ratio between two of the plurality of sound receiving devices based on the volume data, and determining a preset relative distance between two of the plurality of sound receiving devices; The sound source coordinates corresponding to the user's vocal position are obtained according to the sound source arrival angle, the preset relative distances between the plurality of sound receiving devices, and the volume ratio.

4. The method according to claim 3, characterized in that Obtaining the sound source coordinates corresponding to the user's vocal position according to the sound source arrival angle, the preset relative distances between the plurality of sound receiving devices, and the volume ratio, includes: Determining a sound source distance ratio between each of the plurality of sound receiving devices based on a volume ratio between each of the plurality of sound receiving devices; Calculating the sound source distance between each sound receiving device and the user's sound source location based on the sound source arrival angle, the preset relative distances between the plurality of sound receiving devices, and the sound source distance ratio; According to the sound source distance corresponding to each sound receiving device, the sound source coordinates corresponding to the user's vocalization part are obtained.

5. The method according to claim 4, characterized in that The step of calculating the sound source distance between each sound receiving device and the user's sound emitting part based on the sound source arrival angle, the preset relative distances between the plurality of sound receiving devices, and the sound source distance ratio includes: For each pair of sound receiving devices, a sound source distance association relationship is obtained based on a first sound source arrival angle of the first sound receiving device, a second sound source arrival angle of the second sound receiving device, and a preset relative distance between the first sound receiving device and the second sound receiving device; wherein the sound source distance association relationship is used to represent a numerical association between the first sound source distance of the first sound receiving device and the second sound source distance of the second sound receiving device; The sound source distance proportional relationship and the numerical correlation relationship between every two sound receiving devices are solved to obtain the sound source distance between each sound receiving device and the user's sound emitting part.

6. The method according to claim 1 or 2, characterized in that The determining of the target projection focus of the optical machine based on the user's eye perspective and eye coordinates includes: According to the user's eye perspective and eye coordinates, the visual focus coordinates of the user's line of sight at the preset projection position are obtained; Based on the visual focus coordinates, the target projection focus of the optical machine is determined.

7. The method according to claim 6, characterized in that Before controlling the optical engine to project the screen at the target projection focus, the method further includes: In response to a screen projection focus correction instruction issued by a user, obtaining a screen projection focus correction parameter; According to the projection focus correction parameters, the coordinates of the target projection focus are adjusted.

8. The method according to claim 1 or 2, characterized in that The controlling the optical engine to project the screen at the target projection focus includes: Determine the projection content of the optical machine; wherein the projection content includes at least one of device operating status information, video information, and audio information; Control the optical machine to project the projection content to the target projection focus.

9. The method according to claim 8, characterized in that Determining the projection content of the optical machine includes: Obtain the vehicle's speed, navigation information, and road obstacle information to obtain the projection content of the optical machine.

10. The method according to claim 8, characterized in that After controlling the optical engine to project the screen at the target projection focus, the method further includes: In response to a screen projection content adjustment instruction issued by a user, obtaining a screen projection content adjustment parameter; According to the projection content adjustment parameters, at least one of the device operating status information, video information and audio information of the optical screen projection is adjusted.

11. The method according to claim 1 or 2, characterized in that Determining the distance between the user's eyes and the sound-producing part includes: Acquire a user's facial image, extract facial features from the user's facial image, and determine the distance between the user's eyes and a vocalization site based on the facial features; Alternatively, the distance between the user's eyes and the vocalization site is obtained based on a preset facial feature distance.

12. The method according to claim 1 or 2, characterized in that The method further comprises: Storing the target screen projection focus corresponding to the voiceprint feature; After the optical machine is restarted, when it is detected that the newly acquired voiceprint feature is consistent with the voiceprint feature, the optical machine is controlled to project the screen according to the target projection focus corresponding to the voiceprint feature.

13. A control device for an augmented reality head-up display system, characterized in that: The system includes an optical machine and a plurality of non-collinear sound receiving devices, and the device includes: An acquisition module, configured to acquire user voice data collected by each of the sound receiving devices; a first processing module, configured to determine a sound source direction and a sound source coordinate based on the user voice data, and extract a voiceprint feature from the user voice data; The second processing module is used to determine the distance between the user's eyes and the sound source, and perform collaborative positioning based on the distance between the user's eyes and the sound source, the voiceprint characteristics, the direction of the sound source, and the coordinates of the sound source to obtain the user's eye angle and eye coordinates; The third processing module is used to determine the target projection focus of the optical machine based on the user's eye perspective and eye coordinates, and control the optical machine to project the screen with the target projection focus.

14. A vehicle, characterized in that: include: An augmented reality head-up display system and controller, the system comprising an optical machine and multiple non-collinear sound receiving devices; The controller includes a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the control method of the augmented reality head-up display system according to any one of claims 1 to 12 by executing the computer instructions.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the control method of the augmented reality head-up display system according to any one of claims 1 to 12.

16. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the control method of the augmented reality head-up display system according to any one of claims 1 to 12.

Images